Compare for uniqueness between 2 very large matrices

I have two matrices of the same size - 95 x 100,000.

They are in different orders but I would like to compare if columns in one matrix are repeated elsewhere in the other matrix - or if the 2 matrices are completely unique?

[Iuliu Ardelean]

https://uk.mathworks.com/help/matlab/ref/double.ismember.html

You can use Lia = ismember(A,B,'rows'), which returns a vector of ones and zeros of length(A), representing those rows of A which are members of B.

You will need to transpose your matrices though.

[Jan]

I do not understand this sentence: "I want to know if the columns are completely unique - even if the same rows are filled in each they could have different values."

ismember(x,y,'rows') searches for equal rows in both matrices. This is exactly, what "if columns in one matrix are repeated elsewhere in the other matrix" means, isn't it?

Answer 1

[Adam Danz]

To determine if two arrays are 100% identical, use isequal or isequaln to ignore NaN values.

To determine if columns in matrix A are found in matrix B,

% create demo data
rng('default')
A = randi(3,3,20);
B = randi(4,3,20);

for i = 1:size(A,2)
    isInB(i) = ismember(A(:,i)', B','rows');
end

isInB is a 1xn logical vector for n columns of A where isInB(i) indicates whether column i of A is found in B.

To find the column numbers in B that match each column number in A

isMatchInB = false(size(A,2),size(B,2));
for i = 1:size(A,2)
    isMatchInB(i,:) = arrayfun(@(j)isequaln(A(:,i),B(:,j)),1:size(B,2));
end
isInB = any(isMatchInB,2);

isInB is a 1xn logical vector for n columns of A where isInB(i) indicates whether column i of A is found in B.

isMatchInB is an ixj logical vector where isMatchInB(i,j) indicates where colum i of A matches column j of B.

[Koren Murphy]

This appears like it may work on small data sets but i get this error:

Error using false
Requested 100274x100274 (9.4GB) array exceeds maximum array size preference. Creation of
arrays greater than this limit may take a long time and cause MATLAB to become unresponsive.

Error in uniquetest (line 7)
isMatchInB = false(size(A,2),size(B,2));

More information

[Adam Danz]

You could try storing the subscript indicies instead,

c = cell(1,size(A,2));
for i = 1:size(A,2)
    c{i} = find(arrayfun(@(j)isequaln(A(:,i),B(:,j)),1:size(B,2)));
end

[Bruno Luong]

Or using sparse matrix

isMatchInB = sparse([],[],false(0),size(A,2),size(B,2));