Program Optimisation: Out of Memory
1 view (last 30 days)
Show older comments
I have a program that is on the limit of using all the available memory. I have done all I can to increase the memory limit but because the student edition is 32-bit the limit is 2Gb (correct me if I am wrong please). Below is >>memory when started up.
Maximum possible array: 2046 MB (2.146e+009 bytes) *
Memory available for all arrays: 3275 MB (3.434e+009 bytes) **
Memory used by MATLAB: 474 MB (4.974e+008 bytes)
Physical Memory (RAM): 3882 MB (4.071e+009 bytes)
* Limited by contiguous virtual address space available.
** Limited by virtual address space available.
Therefore I need to optimise my code to work within these small memory limits. The error comes from here:
??? Error using ==> times
Out of memory. Type HELP MEMORY for your options.
Error in ==> couple at 58
k = idx1.*100e6 + idx2.*10e6 + idx3.*10 +
idx4.*10e6 + idx5.*100e6;
The size of idx is (2.6e5, 200). Is there any way to optimise the code to allow a larger matrix to be computed. The matrix has been generated from ODE45, however, reducing the accuracy even further is undesirable.
Thank you for your assistance.
1 Comment
Oleg Komarov
on 3 Aug 2011
I usually have 1.1 GB of maximum possible array at startup (2011a Win32 Vista) and a dataset of ~450 MB.
I had to split the dataset into a structure and saved it with -struct and -v6 option.
Then I wrote a function that loops a handle through each portion of the struct.
It takes a little to understand how to operate on parts rather than on the entire dataset, but if you think it may be helpful I will share my solution.
Also, you can save the dataset in binary format and fread it with the approariate offsets (alternatively to save with -struct option).
Accepted Answer
More Answers (6)
Titus Edelhofer
on 29 Jul 2011
Hi,
hmm, somehow this looks strange: the size of idx is (2.6e5, 1)? That means, that one of your idx vectors needs about 2MByte of memory. Doesn't sound too much to me ...???
Titus
Bjorn Gustavsson
on 29 Jul 2011
Maybe you can cut up your problem and work with subsets of idx in sequence. This might not be "ideal" but if nothing else works it might get the job done...
5 Comments
Bjorn Gustavsson
on 30 Jul 2011
OK, if you can't use this subset approach in either dimension, then it looks dire. Maybe you can use memmapfile to store all your big matrices. It might be a workaround - but I haven't used it so I have no grip on if it would work or if there is other pitfalls.
Walter Roberson
on 30 Jul 2011
memmapfile requires virtual memory equivalent to the size of the data stored.
Walter Roberson
on 29 Jul 2011
The limit for the student edition is the same as the limit for the professional edition.
2 Comments
Walter Roberson
on 30 Jul 2011
The 32 bit versions (student _or_ professional) have a limit of 32 bits of address space, which is 4 Gb total including all memory required for system interfaces.
The 64 bit versions (student _or_ professional) in theory could have a limit of 64 bits of address space, but in practice current processors use no more than 48 bits of address space. I do not know how MATLAB divides up that virtual address space. I _think_ I saw a hint that _one_ of the MATLAB library calls was effectively restricted to 40 bits worth of addressing, but I could easily be wrong about that.
Anyhow, student version and professional version are exactly the same memory limits and have been since the first version that was actually called "Student Version". The original two Springer-Verlog books that included a student matlab *did* have lower memory limits, but it was called "Student Edition". (And yes, I do continue to get "Edition" vs "Version" confused myself.)
Walter Roberson
on 30 Jul 2011
In-place operations might help:
k = idx1.*100e6 + idx2.*10e6 + idx3.*10 +
idx4.*10e6 + idx5.*100e6;
could become
k = idx1;
k = k + idx5;
k = k .* 10;
k = k + idx2;
k = k + idx4;
k = k .* 10e5;
k = k + idx3;
k = k .* 10;
This would not require any intermediate vectors, just the output vector that is necessary anyhow. In theory any relatively recent MATLAB should be able to handle
A = A + B
by overwriting A as the computation goes along.
2 Comments
Walter Roberson
on 30 Jul 2011
I didn't start with k = idx1.*100e6 -- I started with k = idx1 .
None the less, if your idx1 is so big that another array the same size cannot be created, then there is really not much you can do about the situation.
You could do hacks such as saving the variables to individual .mat files, and then using system() to run a matlab routine that loaded just enough at a time to take one more step, clearing the variable as soon as it was done with, and then save()'ing the result to a .mat file. What this would get you is that in the second matlab session, the _only_ thing in your workspace would be MATLAB itself, the standard I/O libraries, and the variables you were working with: the effect would be to gain virtual address space.
If you are willing to do some mex'ing, you could probably create a pipe() over to a MATLAB session to do the calculation for you. Note that pipe() might need to be coded differently between Windows and the unix-based systems; in _theory_ from Windows XP SP2 and later the code could be compatible, but I have seen too many hints that the POSIX module built in to Windows doesn't really deliver POSIX.
Jan
on 30 Jul 2011
If the values of idx and k match into a UINT32, use this type to reduce the memory consumption to 50%:
idx1 = uint32(...);
idx2 = uint32(...); ... etc
k = idx1.* uint32(100e6) + idx2.* uint32(10e6) + idx3.*10 + tc...
This together with Walter's inplace method can reduce the needed memory efficiently.
HaveF
on 3 Aug 2011
In spite of your algorithm, you can try this:
If you use 32bit OS, and have 4GB memory, you can check the biggest contiguous memory space which related your biggest matrix size:
feature memstats
at this section:
Largest Contiguous Free Blocks:
you will get the biggest continued memory space. Then you can calculate if this is big enough for your matrix.
If not, you can enable 3GB switch in system: <http://usa.autodesk.com/adsk/servlet/ps/dl/item?siteID=123112&id=9583842&linkID=9240697>
After reboot your system, and type "feature memstats" in Matlab again. You will find your "Largest Contiguous Free Blocks" become larger.
2 Comments
Walter Roberson
on 3 Aug 2011
Note: the above applies only to MS Windows, not to any other 32 bit OS.
feature memstats is, unfortunately, only available in MS Windows.
See Also
Categories
Find more on Logical in Help Center and File Exchange
Products
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!