I assume you are trying to vectorize this for speed purposes. In your current code, the x(:,:,k) expression copies data, and the y(k,:)=temp(:)' assignment copies data. Plus, the loop itself carries some overhead. This can be eliminated with a mex routine that calls the BLAS library directly, hence my question about variable sizes and C compiler (there are FEX submissions available that can do this calculation ... you would not have to write any C code yourself). Plus, part of the vectorization strategy would depend on the variable sizes involved. You have:
x is 8 x 8 x 10
W_matrix is 8 x 8
Dct_matrix is 10 x 10
These sizes are so small that I doubt there would be any significant speed increase gained with whatever method you choose, including mex routines.
If you don't want to consider mex then I would offer the following observations:
X = zeros(K,M*N);
temp = zeros(M,N);
y = zeros(K,M*N);
Not the best way to organize your intermediate results in y. Storing results by row in y causes fragmented memory access to y at each iteration. It would be better to do something like this which will improve the memory access in y during iterations since it keeps the intermediate results contiguous in y:
y = zeros(M,N,K);
y(:,:,k) = W_matrix*x(:,:,k)*W_matrix';
Then after the loop reshape and transpose as needed for downstream processing.
Bottom line is that at the m-code level you cannot avoid those intermediate 2D slice copies since MATLAB does not support nD matrix multiply directly ... you need to use loops in some form.
FEX submissions that can do these types of nD matrix multiply calculations (some are mex and some are m-code):
(needs to be updated ... auto build does not work with later versions of MATLAB)