calculate the averages and standard deviations of massive matrix

Hi， everyone. I confront with a problem and hope to get help from you. Now I have a massive matrix, 1000*1000 and I would like to calculate the changes of the averages and standard deviations of the matrix as the dimension goes up. The simple but time-comsuming way is easy. First,I can calculate the matrix of 1000*1000, 500*500,250*250,...... then the averages and standard deviations of each matrix can be calculated. I wonder, however, can anyone help me with a new way to calculate the changes of averages and deviations as the dimension goes up, a method that can calculate with several code, rather than so much repeated work.

[Walter Roberson]

Are the smaller matrices always the upper-left portion of the larger matrix? Or is this to be done block by block with different block sizes?

[Genhou]

it is be done block by block with different block sizes.

[Image Analyst]

It's funny how you think an array only 3% the size of an ordinary digital camera image is "massive" but anyway,....I'm still trying to figure out if you want to use different size input arrays, or if you want just a single largest matrix and you just compute the mean and stddev in a block-by-block fashion with blockproc() with different size blocks. Or if you don't want to move in "jumps" of block sizes and just move an element at a time with conv2() and stdfilt(). Which is it?

[Genhou]

Embarransing.... the 1000*1000 is only an example, not the real size of matrix.

But, Thank you very much.

[Walter Roberson]

Are the blocks each half the size of the previous blocksize? If so then if you start with the smallest block you can easily calculate the values for the larger blocks.

[Image Analyst]

Genhou, I won't help until you answer our questions because I don't know what help to offer. There are several possible ways to answer your ambiguous question and I want to make sure I give only the one answer you want.

[Genhou]

@Image Analyst It is so nice of you. The process is introduced as follows, and take a matrix of 1000*1000 as an example. First, I will calculate the mean and std of the matrix of 1000*1000. Then, the matrix will be changed into the matrix of 500*500, and the mean and std will be calculated. The matrix of 500*500 are constructed by the mean of block of 2*2 from the matrix of 1000*1000. Then, the mean and std of matrix of 250*250 are calculated. Then, the mean and std of matrix of 200*200, 100*100 are calculated. ...... So， the mean and std of different matrix can be calculated.

I hope that the process is introduced in a clear way now and donot hesitate to ask me if you have any puzzles. Thank you once again.

[Walter Roberson]

Suppose you start with a 1000x1000 matrix and take mean and std over the entire matrix,

m1000 = mean(Array1000(:));
s1000 = std2(Array1000(:));

and then you construct a 500 x 500 matrix of 2 x 2 means:

t500 = conv2(double(Array1000), ones(2,2)/4, 'same');
Array500 = t500(1:2:end, 1:2:end);

and then you take the mean of that:

m500 = mean(Array500(:));

then mathematically this must be the same mean as m1000, to within roundoff error. (The exception would be if you rounded or truncated the fractions when you constructed the 500 x 500 version, then the mean could be slightly different due to loss of precision.)

Answer 1

[Guillaume]

Well, since the smaller matrices are generating by taking the mean of blocks of the larger matrix, the mean of the whole matrices is never going to change. I'm not sure that there is a way to calculate the standard deviation of the small matrices without generating them but if there is, I'm not convinced that it's going to be faster than generating the smaller matrix in the first place.

I'm not sure what your 'simple but time-comsuming way' is, but the following is simple and shouldn't be too time-consuming:

%reduce a matrix size by averaging block of 2x2:
m = rand(1000); %demo data
assert(all(mod(size(m), 2) == 0), 'cannot create 2x2 blocks since matrix has odd size')
smallm = cellfun(@(mm) mean(mm(:)), mat2cell(m, 2 * ones(1, size(m, 1) / 2), 2 * ones(1, size(m, 2) / 2)));
smallstd = std(smallm(:));