MATLAB Answers

How to export complex-valued matrix to HDF file

45 views (last 30 days)
Charles
Charles on 28 Oct 2014
Commented: per isakson on 15 Dec 2019
Hello,
I need to export complex (real,imaginary) data from Matlab using the HDF5 format. I am using the Armadillo C++ template library and I need to duplicate the file structure used by Armadillo.
The HDF5 documentation is not too helpful. It is way too general for me to understand how to simply export complex data. Someone else must surely have done this before.
Below is the output from h5disp so you can see the structure of the files written by Armadillo.
>> h5disp('Example-complex.h5')
HDF5 Example-complex.h5
Group '/'
Dataset 'dataset'
Size: 4x5
MaxSize: 4x5
Datatype: H5T_COMPOUND
Member 'real': H5T_IEEE_F64LE (double)
Member 'imag': H5T_IEEE_F64LE (double)
ChunkSize: []
Filters: none
FillValue: H5T_COMPOUND
Thanks a lot for any pointers.

Accepted Answer

Matteo Seclì
Matteo Seclì on 15 Dec 2019
Answer/summary after five years just for future reference
Charles' original question asked for a way to save complex matrices from MATLAB in HDF5 in such a way that Armadillo could read them. As a matter of fact, MAT-files v7.3 are indeed HDF5-based, and it happens that the way Armadillo stores complex matrices in HDF5 is the same as MATLAB's. So, you can just save one or more variables into a .mat file, making sure you're using the right version, and then load the variables in Armadillo just as if the .mat file was a regular HDF5 file (details and demonstration below).
As for the other way around, i.e. complex matrices from Armadillo to MATLAB, there is no native way to do it despite the format being basically the same. If you try to treat a HDF5 file generated from Armadillo as a .mat file, you get an error. However, per iakson's suggestions paid out, and EasyH5 (nightly as of 15 Dec 2019) now supports loading Armadillo's HDF5 file with complex matrices without extra effort.
Demo: from MATLAB to Armadillo and viceversa
You'll need:
  • MATLAB ≥ 2006b (or ≥ 2017a to save without compression) [source1, source2]
  • Armadillo ≥ 3.4 (bare minimum, highly discouraged, you cannot specify custom dataset names and you have to change the demo code); recommended ≥ 8.100 (demo code works without edits), highly suggested ≥ 8.300 (for .save() capabilities not covered in the demo, but useful in production) [source]. It's better to install Armadillo with the CMake installer so to generate the runtime wrapper library and make sure that HDF5 is enabled at Armadillo's build time; in this way, one can link with just '-larmadillo' and forget about HDF5 libraries forever.
  • EasyH5 Toolbox (https://github.com/fangq/easyh5), rev. ≥ 791dd7b (previous versions partially work as well; in previous versions you had to do some extra work to merge real and imaginary parts of Armadillo's complex arrays and the saveh5 function was not able to save in a format compatible with Armadillo, though you always had the .mat file option).
Let's first generate some complex data from MATLAB and let's save it in double-precision ASCII by separating real and imaginary parts, which can then be imported in Armadillo as a reference data, as HDF5 via EasyH5 and as compressed/uncompressed .mat file v7.3:
% Generate random data
myCx1000x1000Mat=randn(1000)+1i*randn(1000);
% Separate real/imag parts
myCx1000x1000Mat_Re=real(myCx1000x1000Mat);
myCx1000x1000Mat_Im=imag(myCx1000x1000Mat);
% Save as ASCII
save('myCx1000x1000Mat_Re.dat','myCx1000x1000Mat_Re','-ascii','-double');
save('myCx1000x1000Mat_Im.dat','myCx1000x1000Mat_Im','-ascii','-double');
% Save as MAT-file v7.3, both compressed and uncompressed
save('myCx1000x1000Mat_comp.mat','myCx1000x1000Mat','-v7.3');
save('myCx1000x1000Mat_nocomp.mat','myCx1000x1000Mat','-v7.3','-nocompression');
% Save as pure HDF5 via EasyH5, specifying a complex number storage format friendly to Armadillo
saveh5(myCx1000x1000Mat,'myCx1000x1000Mat.h5','ComplexFormat',{'real','imag'});
You can inspect the MAT-files and the HDF5 file via the h5disp function, e.g.:
h5disp('myCx1000x1000Mat_comp.mat')
h5disp('myCx1000x1000Mat_nocomp.mat')
h5disp('myCx1000x1000Mat.h5')
which outputs:
HDF5 myCx1000x1000Mat_comp.mat
Group '/'
Dataset 'myCx1000x1000Mat'
Size: 1000x1000
MaxSize: InfxInf
Datatype: H5T_COMPOUND
Member 'real': H5T_IEEE_F64LE (double)
Member 'imag': H5T_IEEE_F64LE (double)
ChunkSize: 4x1000
Filters: deflate(3)
FillValue: H5T_COMPOUND
Attributes:
'MATLAB_class': 'double'
HDF5 myCx1000x1000Mat_nocomp.mat
Group '/'
Dataset 'myCx1000x1000Mat'
Size: 1000x1000
MaxSize: InfxInf
Datatype: H5T_COMPOUND
Member 'real': H5T_IEEE_F64LE (double)
Member 'imag': H5T_IEEE_F64LE (double)
ChunkSize: 4x1000
Filters: deflate(0)
FillValue: H5T_COMPOUND
Attributes:
'MATLAB_class': 'double'
HDF5 myCx1000x1000Mat.h5
Group '/'
Dataset 'myCx1000x1000Mat'
Size: 1000x1000
MaxSize: 1000x1000
Datatype: H5T_COMPOUND
Member 'real': H5T_IEEE_F64LE (double)
Member 'imag': H5T_IEEE_F64LE (double)
ChunkSize: []
Filters: none
FillValue: H5T_COMPOUND
Now download the attached file testHDF5.txt, rename it as testHDF5.cpp and compile it with
g++ testHDF5.cpp -larmadillo -o testHDF5
This small C++ demo imports the data generated from MATLAB via the real/imag ASCII files and then compares the complex matrix imported in this way to:
  • The same matrix exported via Armadillo to a HDF5 file
  • The HDF5 file exported from MATLAB via the EasyH5 Toolbox
  • The non-compressed MAT-file
  • The compressed MAT-file
The output is the max difference between the ASCII-imported matrix and these matrices, in absolute value, which on my machine is:
matteo$ ./testHDF5
Max absolute difference between ASCII and load method:
armaHDF5 ---> (0,0)
HDF5 ---> (0,0)
MAT(NC) ---> (0,0)
MAT(C) ---> (0,0)
As you can see, the matrices are all equal to the reference matrix imported via ASCII files. The compressed MAT-file seems to work as well, though I would not blindly trust it; I suggest to either use the uncompressed version or the saveh5 function (unless you really have to save some space).
The C++ program also generates a 'myCx1000x1000Mat_ARMA.h5' file which contains the ASCII-imported matrix subsequently saved in HDF5 via Armadillo. We can check that the matrix saved in Armadillo is the same as the one that we have in MATLAB by first importing the matrix and then checking the max of the absolute difference:
myCx1000x1000Mat_ARMA=loadh5('myCx1000x1000Mat_ARMA.h5');
max(max(abs(myCx1000x1000Mat_ARMA.myCx1000x1000Mat-myCx1000x1000Mat)))
which returns exactly 0 on my machine, so the matrices are equal.
TL;DR
  • from MATLAB to Armadillo: save as a v7.3 MAT-file via the built in save() function with the appropriate options or as a pure HDF5 file via EasyH5's saveh5() function (in this case, you must use the syntax saveh5(...,'ComplexFormat',{'real','imag'}) ); then, in both cases, import the file in Armadillo as a regular HDF5 file.
  • from Armadillo to MATLAB: save as a HDF5 file from Armadillo and then import it into MATLAB by using EasyH5's loadh5() function, without the need to use additional options.

  1 Comment

Charles
Charles on 15 Dec 2019
Thanks, I'll give this a try next time I need to do this.

Sign in to comment.

More Answers (2)

per isakson
per isakson on 3 Nov 2014
Edited: per isakson on 13 Dec 2019
Datatype: H5T_COMPOUND &nbsp is not supported by the Matlab high-level HDF5 functions. The low-level functions, which interact directly with HDF5 library functions, are needed to write to this file. (I find these functions a bit tricky to handle.)
Documentation says: " Using the MATLAB Low-Level HDF5 Functions to Export Data &nbsp - &nbsp MATLAB provides direct access to dozens of functions in the HDF5 library with low-level functions that correspond to the functions in the HDF5 library. In this way, you can access the features of the HDF5 library from MATLAB, such as reading and writing complex data types and using the HDF5 sub-setting capabilities." &nbsp
However see
It might be easier for you to use C++ and make a &nbsp mex-file (See: Build MEX-function from C/C++ or Fortran source code).
Added five years later in response to a comment
HDF5 still doesn't handle complex numbers natively. Thus, I assume we will not see it in Matlab anytime soon.
See the File Exchange submission EasyH5 by Qianqian Fang. Its description says: "The `saveh5.m` can handle almost all MATLAB data types, including [...] real and complex arrays" and "EasyH5 stores complex numerical arrays using a special compound data type in an HDF5 dataset."
And see the File Exchange submission HDF5TOOLS by John [Staff]. It contains a subfunction with a promising name
function write_complex_dataset(datasetId,data,offset,stride,count)

  3 Comments

Matteo Seclì
Matteo Seclì on 12 Dec 2019
Hello,
I'm stuck on the same issue and just found this old topic. Did you eventually managed to sort it out? If yes, via a regular MATLAB function using the low-level API or via a MEX-function approach?
I have to do I/O of very large complex matrices, so I definitely cannot write to raw_ascii with Armadillo and then import it into MATLAB.
Thank you!
per isakson
per isakson on 12 Dec 2019
See the links to the File Exchange, which I've added to my answer.
Matteo Seclì
Matteo Seclì on 13 Dec 2019
Thanks a lot,
the two links you've sent look very promising!

Sign in to comment.


Charles
Charles on 12 Dec 2019
I'm sorry, I did not figure out an HDF5 way of doing this. It was 5 years ago and I used Armadillo's ASCII export format, which sounds like it won't work for you. I don't see why MathWorks wouldn't add this support eventually since it's an obvious shortcoming. Either that or you must store two real-valued matrices, which has its own issues.
Good luck!

  1 Comment

Sign in to comment.

Sign in to answer this question.