How to solve out of memory error when constructing sparse matrix

I am construcing a stiffness matrix in finite element computation.

I have got i, j, and v vectors as input arguments for sparse.

Both i and j are 222263280-by-1 int32 vectors while v is a 222263280-by-1 double vector. The stiffness matrix is 1202454-by-1202454.

However I encountered out of memory error when I run the command S = sparse(i,j,v,m,n).

Are there any solutions or suggestions to address this problem?

Why does not sparse support tall array? I think this problem can be solved if tall array used.

Thanks.

Answer 1

[Bruno Luong]

You might try the syntax

nz = ... ; < 222263280
S = sparse(i,j,v,m,n,nz);

with nz is the estimation of non-zero elements of your matrix. If you build P1 FEM, it's the number of edges of your setup. You might use Euler formula to estimte it : https://www.mathsisfun.com/geometry/eulers-formula.html

or using

nz = size(unique([i(:) j(:)], 'row'),1)

Note that the byte size required to store a sparse matrix S is

if isa(S,'double')
    elbyte = 8;
else
    elbyte = 1;
end
Sbytesize = 8*(size(S,2)+1) + ... % Jc array
            8*nnzmax(S) + ... % Ir array
            elbyte*nnzmax(S)  % data value array

So if you deal well, your S will take less than 3.566 Gb (I assumz nz < 222263280),

>> (1202454+1)*8+222263280*8+222263280*8

ans =

   3.5658e+09

plus the space of I, J, V. It will require 7 Gb to build it. If your have other suff around 16 Gb become a little tight.

[lala]

Thanks for your answer.

Answer 2

[Matt J]

It's a 5 GB matrix. Are you sure you have enough RAM?

[lala]

I have 16 GB RAM, but that's all I have. Only matlab is running. I also have other necessary variables in RAM representing elements (8-by-374872 int32), node coordinates (3-by-400818 double), nodal values (type double), etc. If I clear all variables except i, j, and v for sparse, it can construct the sparse matrix successfully.

[Matt J]

So? Problem solved?

[lala]

No, I need help. 😭

since all i, j, and v can be maintained in RAM, why sparse failed? Does sparse need more memory than i, j, and v?

[Torsten]

If I clear all variables except i, j, and v for sparse, it can construct the sparse matrix successfully.

This should show you that it's not possible with the RAM available.

Answer 3

[John D'Errico]

Remember that i and j and v also take memory.

(222263280*4*2 + 222263280*8)/1024^3

ans = 3.3120

So those three variables alone require 3.3 GB. And then the sparse array would require 5 GB. Next, you need to factor in that MATLAB will require CONTIGUOUS memory for any arrays. And MATLAB itself, and your OS take up memory. So I can understand that you may just have room to create that array, if you clear everything else. But you still won't be able to do virtually anything with it once built.

This is not really a problem of needing support for tall arrays to build a sparse matrix. That is just a crutch that will encourage you to build larger arrays yet. If you are going to work with these problems on that computer, then you desperately need to get more RAM. RAM is cheap these days. Honestly, I would suggest 16 GB of RAM is just barely a viable amount these days if you are doing any serious work in MATLAB.

John's law: Computer problems always expand to just a little larger than the memory available on the largest computer you can find.

(So, yes, according to the law I just gave you, even if you do get more RAM, you will quickly decide to solve larger problems yet. Since who wants to solve small problems? But it gives you a few years of grace.)

[lala]

Thanks for your answer. I would consider upgrading the RAM.