How to export complex-valued matrix to HDF file

Hello,

I need to export complex (real,imaginary) data from Matlab using the HDF5 format. I am using the Armadillo C++ template library and I need to duplicate the file structure used by Armadillo.

The HDF5 documentation is not too helpful. It is way too general for me to understand how to simply export complex data. Someone else must surely have done this before.

Below is the output from h5disp so you can see the structure of the files written by Armadillo.

>> h5disp('Example-complex.h5')
HDF5 Example-complex.h5 
Group '/' 
    Dataset 'dataset' 
        Size:  4x5
        MaxSize:  4x5
        Datatype:   H5T_COMPOUND
            Member 'real':  H5T_IEEE_F64LE (double)
            Member 'imag':  H5T_IEEE_F64LE (double)
        ChunkSize:  []
        Filters:  none
        FillValue:  H5T_COMPOUND

Thanks a lot for any pointers.

[per isakson]

See Armadillo C++ library for linear algebra & scientific computing

Answer 1

[Matteo Seclì]

Answer/summary after five years just for future reference

Charles' original question asked for a way to save complex matrices from MATLAB in HDF5 in such a way that Armadillo could read them. As a matter of fact, MAT-files v7.3 are indeed HDF5-based, and it happens that the way Armadillo stores complex matrices in HDF5 is the same as MATLAB's. So, you can just save one or more variables into a .mat file, making sure you're using the right version, and then load the variables in Armadillo just as if the .mat file was a regular HDF5 file (details and demonstration below).

As for the other way around, i.e. complex matrices from Armadillo to MATLAB, there is no native way to do it despite the format being basically the same. If you try to treat a HDF5 file generated from Armadillo as a .mat file, you get an error. However, per iakson's suggestions paid out, and EasyH5 (nightly as of 15 Dec 2019) now supports loading Armadillo's HDF5 file with complex matrices without extra effort.

Demo: from MATLAB to Armadillo and viceversa

You'll need:

• MATLAB ≥ 2006b (or ≥ 2017a to save without compression) [source1, source2]
• Armadillo ≥ 3.4 (bare minimum, highly discouraged, you cannot specify custom dataset names and you have to change the demo code); recommended ≥ 8.100 (demo code works without edits), highly suggested ≥ 8.300 (for .save() capabilities not covered in the demo, but useful in production) [source]. It's better to install Armadillo with the CMake installer so to generate the runtime wrapper library and make sure that HDF5 is enabled at Armadillo's build time; in this way, one can link with just '-larmadillo' and forget about HDF5 libraries forever.
• EasyH5 Toolbox (https://github.com/fangq/easyh5), rev. ≥ 791dd7b (previous versions partially work as well; in previous versions you had to do some extra work to merge real and imaginary parts of Armadillo's complex arrays and the saveh5 function was not able to save in a format compatible with Armadillo, though you always had the .mat file option).

Let's first generate some complex data from MATLAB and let's save it in double-precision ASCII by separating real and imaginary parts, which can then be imported in Armadillo as a reference data, as HDF5 via EasyH5 and as compressed/uncompressed .mat file v7.3:

% Generate random data
myCx1000x1000Mat=randn(1000)+1i*randn(1000);
% Separate real/imag parts
myCx1000x1000Mat_Re=real(myCx1000x1000Mat);
myCx1000x1000Mat_Im=imag(myCx1000x1000Mat);
% Save as ASCII
save('myCx1000x1000Mat_Re.dat','myCx1000x1000Mat_Re','-ascii','-double');
save('myCx1000x1000Mat_Im.dat','myCx1000x1000Mat_Im','-ascii','-double');
% Save as MAT-file v7.3, both compressed and uncompressed
save('myCx1000x1000Mat_comp.mat','myCx1000x1000Mat','-v7.3');
save('myCx1000x1000Mat_nocomp.mat','myCx1000x1000Mat','-v7.3','-nocompression');
% Save as pure HDF5 via EasyH5, specifying a complex number storage format friendly to Armadillo
saveh5(myCx1000x1000Mat,'myCx1000x1000Mat.h5','ComplexFormat',{'real','imag'});

You can inspect the MAT-files and the HDF5 file via the h5disp function, e.g.:

h5disp('myCx1000x1000Mat_comp.mat')
h5disp('myCx1000x1000Mat_nocomp.mat')
h5disp('myCx1000x1000Mat.h5')

which outputs:

HDF5 myCx1000x1000Mat_comp.mat 
Group '/' 
    Dataset 'myCx1000x1000Mat' 
        Size:  1000x1000
        MaxSize:  InfxInf
        Datatype:   H5T_COMPOUND
            Member 'real':  H5T_IEEE_F64LE (double)
            Member 'imag':  H5T_IEEE_F64LE (double)
        ChunkSize:  4x1000
        Filters:  deflate(3)
        FillValue:  H5T_COMPOUND
        Attributes:
            'MATLAB_class':  'double'
HDF5 myCx1000x1000Mat_nocomp.mat 
Group '/' 
    Dataset 'myCx1000x1000Mat' 
        Size:  1000x1000
        MaxSize:  InfxInf
        Datatype:   H5T_COMPOUND
            Member 'real':  H5T_IEEE_F64LE (double)
            Member 'imag':  H5T_IEEE_F64LE (double)
        ChunkSize:  4x1000
        Filters:  deflate(0)
        FillValue:  H5T_COMPOUND
        Attributes:
            'MATLAB_class':  'double'
HDF5 myCx1000x1000Mat.h5 
Group '/' 
    Dataset 'myCx1000x1000Mat' 
        Size:  1000x1000
        MaxSize:  1000x1000
        Datatype:   H5T_COMPOUND
            Member 'real':  H5T_IEEE_F64LE (double)
            Member 'imag':  H5T_IEEE_F64LE (double)
        ChunkSize:  []
        Filters:  none
        FillValue:  H5T_COMPOUND
        

Now download the attached file testHDF5.txt, rename it as testHDF5.cpp and compile it with

g++ testHDF5.cpp -larmadillo -o testHDF5

This small C++ demo imports the data generated from MATLAB via the real/imag ASCII files and then compares the complex matrix imported in this way to:

• The same matrix exported via Armadillo to a HDF5 file
• The HDF5 file exported from MATLAB via the EasyH5 Toolbox
• The non-compressed MAT-file
• The compressed MAT-file

The output is the max difference between the ASCII-imported matrix and these matrices, in absolute value, which on my machine is:

matteo$ ./testHDF5

Max absolute difference between ASCII and load method:

armaHDF5 ---> (0,0)

HDF5 ---> (0,0)

MAT(NC) ---> (0,0)

MAT(C) ---> (0,0)

As you can see, the matrices are all equal to the reference matrix imported via ASCII files. The compressed MAT-file seems to work as well, though I would not blindly trust it; I suggest to either use the uncompressed version or the saveh5 function (unless you really have to save some space).

The C++ program also generates a 'myCx1000x1000Mat_ARMA.h5' file which contains the ASCII-imported matrix subsequently saved in HDF5 via Armadillo. We can check that the matrix saved in Armadillo is the same as the one that we have in MATLAB by first importing the matrix and then checking the max of the absolute difference:

myCx1000x1000Mat_ARMA=loadh5('myCx1000x1000Mat_ARMA.h5');
max(max(abs(myCx1000x1000Mat_ARMA.myCx1000x1000Mat-myCx1000x1000Mat)))

which returns exactly 0 on my machine, so the matrices are equal.

TL;DR

• from MATLAB to Armadillo: save as a v7.3 MAT-file via the built in save() function with the appropriate options or as a pure HDF5 file via EasyH5's saveh5() function (in this case, you must use the syntax saveh5(...,'ComplexFormat',{'real','imag'}) ); then, in both cases, import the file in Armadillo as a regular HDF5 file.
• from Armadillo to MATLAB: save as a HDF5 file from Armadillo and then import it into MATLAB by using EasyH5's loadh5() function, without the need to use additional options.

[Charles]

Thanks, I'll give this a try next time I need to do this.

Answer 2

[per isakson]

Datatype: H5T_COMPOUND &nbsp is not supported by the Matlab high-level HDF5 functions. The low-level functions, which interact directly with HDF5 library functions, are needed to write to this file. (I find these functions a bit tricky to handle.)

Documentation says: " Using the MATLAB Low-Level HDF5 Functions to Export Data &nbsp - &nbsp MATLAB provides direct access to dozens of functions in the HDF5 library with low-level functions that correspond to the functions in the HDF5 library. In this way, you can access the features of the HDF5 library from MATLAB, such as reading and writing complex data types and using the HDF5 sub-setting capabilities." &nbsp

However see

• Compound Data Type
• HDF5 data to struct reader by Pauli Virtanen
• HDF5 1.8 Matlab Examples &nbsp A large set of examples with the low-level functions. I think they are good.
• hdf-forum &nbsp I think that's the place to pose this question. There are some Matlab related questions.

It might be easier for you to use C++ and make a &nbsp mex-file (See: Build MEX-function from C/C++ or Fortran source code).

Added five years later in response to a comment

HDF5 still doesn't handle complex numbers natively. Thus, I assume we will not see it in Matlab anytime soon.

See the File Exchange submission EasyH5 by Qianqian Fang. Its description says: "The `saveh5.m` can handle almost all MATLAB data types, including [...] real and complex arrays" and "EasyH5 stores complex numerical arrays using a special compound data type in an HDF5 dataset."

And see the File Exchange submission HDF5TOOLS by John [Staff]. It contains a subfunction with a promising name

function write_complex_dataset(datasetId,data,offset,stride,count)

[Matteo Seclì]

Hello,

I'm stuck on the same issue and just found this old topic. Did you eventually managed to sort it out? If yes, via a regular MATLAB function using the low-level API or via a MEX-function approach?

I have to do I/O of very large complex matrices, so I definitely cannot write to raw_ascii with Armadillo and then import it into MATLAB.

Thank you!

[per isakson]

See the links to the File Exchange, which I've added to my answer.

[Matteo Seclì]

Thanks a lot,

the two links you've sent look very promising!

Answer 3

[Charles]

I'm sorry, I did not figure out an HDF5 way of doing this. It was 5 years ago and I used Armadillo's ASCII export format, which sounds like it won't work for you. I don't see why MathWorks wouldn't add this support eventually since it's an obvious shortcoming. Either that or you must store two real-valued matrices, which has its own issues.

Good luck!

[Matteo Seclì]

Thank you anyway! :)