Tall three dimensional arrays

5 views (last 30 days)
Science Machine
Science Machine on 6 Sep 2022
I am determine if the following is both doable and beneficial use of tall arrays:
  1. Let's say I have a set of larger matrices A_1,...,A_99 , where dim(A)=800,000 x 600.
  2. Due to memory constraints, (196 GB), I cannot hold the full A_k in memory.
  3. I would like to take say, one entry ij, of each A_k , collect it in a vector v(1:99) for k=[1,99], then perform an operation on v - for each ij pair. Note I have 3 directions: the original of each matrix, and k as a result of collecting the matricies. And I am interested in performing an operation (fft) in the k direction, the 'tall' dimension of the tallArray.
  4. Question (a): is this use of tallArrays somehow optimal, or at least not terrible?
  5. Question (b): Save or not save each matrix to disk before calling tallArray? Since I have A_k in memory, would it be advisable to use arrayDatastore ? Or should I save each A_k into memory as a .mat file in the same directory, then create a datastore from the saved-to-disk *.mat files?

Answers (0)


Find more on Large Files and Big Data in Help Center and File Exchange




Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!