read and divide HDF5 data into chunks

I have 1000 + HDF5 files of 1800 by 3600 matrix. I want to divide the 1800 * 3600 matrices into 4 chunks and store with a ID into an array. I want to repeat this process for 1000 + files. Can someone help how to use H5P.set_chunk OR H5S.select_hyperslab ? I used H5S.select_hyperslab to get only one slab, how should I repeat this process ?

Answer 1

[Dinesh Iyer]

The H5P.set_chunk is used to specify the chunk dimensions of a dataset i.e. what should the size of each chunk when it is is stored in the file. The H5S.select_hyperslab is used to specify the portion of the dataset that you want to read. If you are reading data a portion of the data from a dataset, this is probably what you need to do.

When you say that you want to store each chunk with an ID into an array, do you mean you want to read it into MATLAB or do you want to store it again into another HDF5 file?

For starters, you can use the high-level h5read function to read a portion of the dataset. I am not sure how you want to divide the data into 4 chunks but I am going to assume that each chunk is 1800x900. This does not impact the code.

The code below provides an idea on how you can do this.

fileNames = dir('*.h5');
fileNames = {fileNames.name}'

numChunks = 4;
chunkSize = [1800 900];
for cnt = 1:numel(fileNames)
    fileToRead = fileNames{cnt};
    s = struct();
    for cnt = 1:numChunks
        ID = sprintf('%s_Chunk_%02d', matlab.land.makeValidname(fileToRead), cnt);
        startLoc = [1 chunkSize(2)*(cnt-1)+1];
        s.(ID) = h5read(fileToRead, '/mydataset', startLoc, chunkSize);
    end
end

I have not run the above code and so apologies for any errors but it does give an idea of how you can do this.

If you want to use the low-level functions such as H5D.read, you have to loop and update the h5_start input argument to point to the location of the dataset that you want to read.

[nlm]

Hi Thank you for your response. I ran your code, it resulted in 4 chunks only. Please find the data attached. As you can see I have many fields each of them 1700*3800, and many of which I currently empty. I have 1000's of such HDF5 files each of which will result in 10 1700*3600 matrices. To handle such large data sets I was suggested to divide the HDF5 files into chunks and run my analysis separately on each of them. I'm able to read hyper slabs of certain dimensions from 1000 files and conduct my analysis. But if I want to divide it into chunks and run my analysis parallelly, how should I do that ? I want to save it as structure or matfile.

[Dinesh Iyer]

The code that I have provided should help you get started. It results in 4 chunks because I have taken a chunk size of [1800 900]. You can modify this.

If you want to speed up the operation, you can use PARFOR loops to parallelize the files that you are processing.

[nlm]

It results in only 4 chunks from 400 files that are there. How do I get 4 chunks for each of the 400 HDF5 files and store it in structure.

I modified your code, and I get empty arrays,

fileNames = dir('*.HDF5');

fileNames = {fileNames.name}';

numChunks = 4; chunkSize = [1800 900];

for cnt1 = 1:numel(fileNames)

    fileToRead = fileNames{cnt1};
    s = struct();
    for cnt = 1:numChunks
        ID = sprintf('C_%d', cnt);
        startLoc = [1 chunkSize(2)*(cnt-1)+1]
        s(cnt1).(ID) = h5read(fileToRead, '/Grid/HQprecipitation', startLoc, chunkSize);
    end
end