"Error using zeros" maximum variable size allowed by the program is exceeded
Show older comments
I was trying to generate a 2^57 x 57 zeros matrix and error occurred. I tried with sparse and the error still persist. Is there anywhere can I generate the matrix efficiently.
Please help. I am student, still exploring how to work with MATLAB. Thank you.
Accepted Answer
Steven Lord
on 23 Jun 2016
If you wanted to create that matrix as a logical array, it would require:
numElements = (2^57)*57;
exabytes = numElements/(1000^6)
exabytes =
8.21456572032379
"Therefore, one exabyte could hold a hundred thousand times the printed material, or 500 to 3000 times all content of the Library of Congress."
To process all possible subsets of a set with 57 elements (which is what I assume you're trying to do, based on the dimensions of the matrix you're trying to create) would require quite a bit of time.
numCombinations = 2^57;
processingTime = years(seconds(numCombinations))
processingTime =
4.5668e+09
You probably want to rethink your approach, unless perhaps you have a VERY large cluster of machines available.
1 Comment
Cassandra Wong
on 24 Jun 2016
More Answers (3)
James Tursa
on 23 Jun 2016
Edited: James Tursa
on 23 Jun 2016
2 votes
A matrix of size 2^57 x 57 doubles would take about 6.57e19 bytes of memory ... way way more than your computer has available. So no, there is no way to generate this matrix on your computer. Even a sparse matrix of this size would take too much memory just to store the index data.
1 Comment
Cassandra Wong
on 24 Jun 2016
Walter Roberson
on 23 Jun 2016
0 votes
A full array for that would require a virtual memory space more than 3 1/2 times larger than can be addressed with 64 bit memory. I don't think you are going to be able to create something that big. You might not be able to create anything that requires more than 2^48 bytes as all current implementations of the x64 architecture only have at most 48 address pins.
1 Comment
Cassandra Wong
on 24 Jun 2016
Najme Behdad
on 2 Jan 2017
Edited: Najme Behdad
on 2 Jan 2017
0 votes
A question comes to me, Is there a way processing for binary large scale matrix in MATLAB 2 ^ 40. Was this possibility added in MATLAB 2016? please help me
10 Comments
John D'Errico
on 2 Jan 2017
Please don't ask questions as an answer. Anyway, the answer is a flat out no. An array with any dimension of 2^40 is still wildly too large of an array to store on your computer.
Walter Roberson
on 2 Jan 2017
I have to disagree. An array with 2^40 double precision elements would occupy only 8 terabytes of memory. That is possible on a sufficiently large system. For example SGI makes servers with up to 64 terabytes of memory. If you have a large enough system, then MATLAB can create an array this large.
This is a different situation than the 2^57 by 57 that was asked about originally, as that exceeded the address space available to the x64 architecture. 2^40 does not.
Even 2^40 x 40 uint8() or logical() does not exceed 64 terabytes (it would require 40 terabytes.) 2^40 x 40 double() would exceed 64 terabytes though, so you would have to find a really big computer for that. Not impossible, though.
John D'Errico
on 2 Jan 2017
Edited: John D'Errico
on 2 Jan 2017
True. But on almost all current systems, 2^40 is far too large. And if someone is asking this question, it is most likely true that they don't understand why that is a problem, so they surely don't have experience on and access to such a large machine. This is surely true for >99% of all MATLAB users.
It is also true that most people who want to do something like this really don't want to do it anyway. They think they do, due to inexperience with computing. Much of the time there are better ways to solve a problem than just throwing huge amounts of memory at it.
Walter Roberson
on 2 Jan 2017
8 terabytes barely gets you onto the Top 500 Supercomputers list (it is more than the computer at current position 499 but much less than the one at current position 494, with memory sizes not given for 495-498). I thus have to doubt ">99% of all MATLAB users". 98% maybe ;-)
I do find, though, that people who ask these kinds of questions and who will actually have the resources tend to specifically mention having the resources, such as if they are cross-checking their planning for a large computer proposal.
Najme Behdad
on 2 Jan 2017
thanks a lot, What is the solution for the processing of this matrix? 40 is the number of features. Subset = dec2bin (0: 2 ^ 40) == '1' Do you recommend I use the function "map and reduce"?
Najme Behdad
on 2 Jan 2017
What is the maximum size of binary matrix, Processable in MATLAB 2016؟
Walter Roberson
on 3 Jan 2017
The maximum size of a matrix of logical or uint8, of size 2^N by N, that can be processed in MATLAB, is 2^42 by 42. That requires a system with 168 terabytes of memory.
"What is the solution for the processing of this matrix? 40 is the number of features."
You have not described the goal. It sounds as if you are thinking you need to do an exhaustive trial of all the possible combinations of features being present or not, but what are you calculating?
Considering your use of the word "feature", I suspect you might be trying to find the "best" set of features for some kind of classification. That is a topic known as "feature selection". A complete search is probably not going to do a significantly better job than using some approximations: you would run a significant risk of over-fitting -- the answer you came up with would likely involve small statistical accidents that made that particular choice work marginally better on your test data but worse than other possibilities on real-world data.
There are a number of different approaches to feature selection; different approaches have been studied more with respect to different classification schemes.
In some cases a good approach is to run a number of different random approaches, pick the best several, and use an ensemble classification scheme
Najme Behdad
on 3 Jan 2017
"You have not described the goal. It sounds as if you are thinking you need to do an exhaustive trial of all the possible combinations of features being present or not " Yes, that's exactly right، I want to gain all combinations of features. "but what are you calculating?"
I want to calculate the value of each combination of features. When I know the value of some combinations feature that is equal to one.
Walter Roberson
on 3 Jan 2017
Do you mean you have a vector of 40 values and you are trying to find a subset of the values that totals exactly 1.0 ? If so, then you have to be very careful about round-off in your calculations.
Finding a subset of values with a particular total is known as the Knapsack problem.
Najme Behdad
on 4 Jan 2017
thank you
Categories
Find more on Matrix Indexing in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!
