empty sparse matrix with size greater than (100000,100000) needs overhead space

9 views (last 30 days)
For an industry project we want to set up a lookup-table from many (15) inputs to one output. The input values can be only in a small range, depending on each other, but may stretch over a great range otherwise, with at least 100 steps needed for resolution (-> 1e30 values). So I advised for sparse matrices.
Of course I may only use 2d-matrices, but that can be solved quite easily by re-indexing.
Nevertheless, I run into "Out-of-memory errors". The reason seems to be overhead memory that is needed to build empty sparse matrices with a size greater than (100000,100000) (-> 1e10 values). This overhead memory is only a fraction of the memory the full matrix would need of course, and this fraction grows less with the size of the matrix. But the absolute value grows with the size, up to the Out-of-memory error.
Why does a sparse matrix need any extra space at all? And why does it not need this space for relatively small sizes? Is it a bug? Is there a workaround?
clear
close all
[uV sV] = memory;
basemem = uV.MemUsedMATLAB;
try
for i = 1:20
A = sparse(10^i,10^i);
[uV sV] = memory;
mem(i) = uV.MemUsedMATLAB - basemem;
end
catch
dummy = 4;
end
l = numel(mem);
ls = 1;
addrSpace = 10.^((ls:l)*2);
figure
loglog(addrSpace,mem(ls:l));
figure
semilogx(addrSpace,mem(ls:l)./addrSpace);
clear

Accepted Answer

James Tursa
James Tursa on 15 Feb 2012
Sparse matrices in MATLAB always store an index vector that is the same size as the number of columns+1. So even for an empty sparse matrix with no memory yet allocated for any elements, there will still be this column index vector created. If just that much is killing your memory, maybe you can come up with your own manual indexing scheme into the actual sparse matrix that only has a few actual columns in the sparse matrix. E.g., have the actual sparse matrix be a column vector and then manually calculate the actual column location of an element from a 2D index pair off to the side.

More Answers (5)

Mark Shore
Mark Shore on 17 Feb 2012
for i = 1:20
A = sparse(10^i,10^i);
etc.
end
It seems that you are being a little optimistic in trying to create a (1E20,1E20) matrix, sparse or not.

Jochen Schuettler
Jochen Schuettler on 15 Feb 2012
Sadly I started with your given solution, but then I get
_Sparse matrix sizes must be non-negative integers less than MAXSIZE as defined by COMPUTER. Use HELP COMPUTER for more details. _
  1 Comment
James Tursa
James Tursa on 15 Feb 2012
How about multiple sparse column vectors contained in a cell array, each one that is within the limits of MAXSIZE. Complicates your indexing scheme, but seems like it could be done.

Sign in to comment.


Sean de Wolski
Sean de Wolski on 15 Feb 2012
You could feed sparse an nzmax to help cut down on memory.

Jochen Schuettler
Jochen Schuettler on 16 Feb 2012
Obviously this does not help, as the matrix is empty already, it needs that much space just for existing (I tried).
I also tried putting a (1e14,1) sparse matrix in a cell array - that works, but gives me the "out of memory" error earlier, because it needs n*2 bytes for n such sparse matrices, instead of n+1 bytes for a (1e14,n) sparse matrix. I simply can't produce a space of 1e30 values!
Any more ideas?
  1 Comment
James Tursa
James Tursa on 16 Feb 2012
Can you show the exact code you are using to build the cell array of sparse matrices that is running out of memory?

Sign in to comment.


Vanny moor
Vanny moor on 17 Feb 2012
Is there a way to only keep part of the matrix in memory at given time and the rest buffered on disk?

Products

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!