problem in parsing data

Hello Everyone,

I am reading a binary file using MATLAB whose size is around 96 MB. After I read in the binary file I convert it into a MATRIX of 11563 rows and 8192 columns. I have written a code for data parsing which takes the first row and the first 12 columns are taken as header of data and the rest is taken as the actual data. Problem is while I am parsing the the initial 1000 rows are parsed quickly and then MATLAB slows down considerably. How can I increase the speed of my MATLAB code. I am storing the data in predefined cell array.

So for example after I read in Binary file my data is uint8 which I save into RawData variable uint8. My MATLAB has been running for 3 hours and I have only been able to extract some 5000 rows using my code.

[dpb]

Crystal orb is in shop yet again today...

Can't tell w/o seeing the code. One conjecture would be you haven't preallocated arrays and so are reallocating memory every pass...

Oh, I see you did say you preallocated a cell array...but, as IA asks, why a cell array instead of array? The cells are preallocated, but the memory for the content of each isn't so it's much like not having allocated an array w/ the added overhead of the cell structure besides. Unless the various records/cells are irregular, cell() is probably the wrong choice.

Also, if you described what parsing you're doing and the structure of the data file, it's quite likely it could be vectorized whereas your description sounds_like looping...

[Ricky]

The reason for preallocating a cell array is the size of array inside each cell is not fixed. In my cell array I have two structures , the first structure contains the header of my data and second structure contains the actual data. Depending upon the sensor used my second structure can be of < 1*8192 uint8 > or < 1*16384 uint8 >. The sensor type is selected using a GUI so I cannot preallocate the array size.

You are right I am looping through my input MATRIX and storing each row in a cell array.

[per isakson]

Did you watch the Windows Task Manager | Performance while your code is running?

It's the fields of the actual and the header data that needs to be pre-allocated.

How much RAM does the computer have?

Answer 1

[Image Analyst]

Why are you using a cell array? That probably slows it down. Also, if you're displaying the data with image() or imshow() without clearing out prior images you've stored, then that will slow it down also. Use "cla" to clear the old images/matrices out of the axes before you display a new one.

[Ricky]

Actually this is not an image data but ultrasonic sensor data.

[Image Analyst]

I still don't see why you can't just use a regular structure for your header data, and then a uint8 or uint16 array for your data. I do that all the time. I just read stuff one at a time and load up a header structure, then when it's time to get the data, just use fread() to load all the data into one 2D array all in one fell swoop (no looping required).

Answer 2

[Muthu Annamalai]

You want to cast your data to right type. Default numeric type in MATLAB in double. You may want to use uint8() on your fread() calls so that you only store byte-sized chunks into your variable.

If you do this consistently you should get a 8x reduction in the data size.

Also as others have commented, it is always good to share your code to provide context to your problem!