Binary format that Matlab can read, and how to convert with C/c++?

15 views (last 30 days)
My current procedure is not optimal. It is:
  1. extract terabytes of data with c++.
  2. Naivly writing that to disk in ascii format
  3. analysing this data using Matlab. I am reading in tons of moderately sized files (half a milliion lines of doubles for each file), rather than a few huge files. Hence tall arrays aren't necessary, just a good format.
So my question is:
  1. Instead of writing to disk in ascii format, is there an easy to use c++ library (or matlab library that Matlab coder can convert to C) that converts the data to some binary format that matlab can read?
  2. Or alternatively, can matlab read unformatted Fortran data? Probably this is not 'so' ideal since the data switches machines
Also,
  1. The reason for preferring binary formats to ascii are file size and speed savings. Also, I was reading about HDF5, and it seemed really hard to use, whereas I would prefer something that will not cost me several days to get up and working.
  2. I realize this question would be perhaps better asked on stackexchange on a C forum, but I was afraid I would get responses which Matlab would not be able to work with, hence I am asking this question from the Matlab side.

Answers (1)

Gayatri Rathod
Gayatri Rathod on 25 May 2023
Hi,
For a more efficient and easy-to-use method to store and read data from C++ in a format that MATLAB can handle. Here are a few options you can consider:
1. Binary File Format: Instead of writing the data to disk in ASCII format, you can write it in a binary format using C++ and then read it in MATLAB. C++ provides functions like “fwrite and fread that allow you to write and read binary data. MATLAB also has corresponding functions (fwrite, fread) that can read binary data. By using a binary format, you can achieve smaller file sizes and faster read/write operations compared to ASCII. fwrite function, fread function.
If you consider saving data into MATLAB, here are a few options you can consider:
  • MATLAB Data Format: MATLAB has its own binary data format called MATLAB MAT-file (.mat) that can store various types of MATLAB data structures, including numeric arrays, matrices, and structs. You can use the MATLAB Engine API for C++ to write data from C++ directly to MAT-files. Saving and loading mat files.
  • MATLAB's save function with -v6 option: If you want to stick with MATLAB, you can use the save function with the -v6 option to save your data in a binary format. You can save your data in MATLAB and then read it back using the load function. Save function, load function.
2. Unformatted Fortran Data: MATLAB has limited support for reading unformatted Fortran data using the “fread function. However, it may not be an ideal solution due to potential issues with different machine architectures and byte order.
Hope it helps!

Categories

Find more on Fortran with MATLAB in Help Center and File Exchange

Products


Release

R2022b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!