You are now following this question
- You will see updates in your followed content feed.
- You may receive emails, depending on your communication preferences.
Faster way of loading data than loading mat file with array? / Faster way than logical indexing or find?
4 views (last 30 days)
Show older comments
Hello everybody,
I need to load a really big amount of data into matlab. I could speed it up by moving from a table to an array and by using v6 to save it. But still it takes almost 50% of the time of the program. Is there an even faster method?
After that, I need to find certain values in this array. This takes about 40% of the time. Is there a faster emthod than logical indexing?
13 Comments
Star Strider
on 22 Jan 2020
Depending on the options available to you, installing a solid-state drive (SSD) in your computer or as an external drive would be the best option. Lacking that (and depending on how large the file is), saving it to an SD card (that currently can have reasonably large capacities) would be something to experiment with.
Guillaume
on 22 Jan 2020
What format is the data stored as? text file? excel? If you mention -v6 does that mean it's stored as a (binary) mat file? If it's a mat file, I doubt that the storage method has much of an impact.
"I need to find certain values in this array" This is too vague for us to answer. Find according to what rule/criteria? To do what with the values afterward?
Gerrit
on 22 Jan 2020
Yeah, it's stored as a binary mat file. At first, I used tables for my program. There I had a loading time of about 2.4s. Now I use arrays and the loading time decreased to 1.1s. Thats why I ask.
This is what I meant by I need to find certain values. For my 4D Interpolation afterwards, I only need values in a specific area.
new_upper_airfoil_all=new_upper_airfoil_all(new_upper_airfoil_all(:,4) < (Ma+delta_ini) & new_upper_airfoil_all(:,4) > (Ma-delta_ini) & new_upper_airfoil_all(:,5) < (C_l+delta_ini) & new_upper_airfoil_all(:,5) > (C_l-delta_ini),2:7);
new_lower_airfoil_all=new_lower_airfoil_all(new_lower_airfoil_all(:,4) < (Ma+delta_ini) & new_lower_airfoil_all(:,4) > (Ma-delta_ini) & new_lower_airfoil_all(:,5) < (C_l+delta_ini) & new_lower_airfoil_all(:,5) > (C_l-delta_ini),2:7);
Guillaume
on 22 Jan 2020
If you're using plain arrays, that's as fast as you'll get with matlab. Similarly, your current indexing is the best you can do.
On the other hand, the problem may come from your algorithm itself, which may be (very) inefficient regardless of the storage method. Do you know the complexity of your algorithm? If it's exponential or worse, then you may need to find something more efficient.
Walter Roberson
on 22 Jan 2020
If you have plain numeric arrays then the fastest way to load them is to use fread() from a binary file.
Walter Roberson
on 22 Jan 2020
The data that you are currently doing the searching for ranges: is that data in sorted order? Because if it is, then binary search can be used to find the boundary in log2 time instead of linear time. I seem to recall that there is a File Exchange contribution to do binary search at the mex level.
James Tursa
on 22 Jan 2020
A mex routine could avoid extracting the temporary arrays such as new_upper_airfoil_all(:,5) etc, but I'm not sure how much time that would save overall. Is the data in the arrays sorted? If so, a mex routine could use a binary search to find the edges and speed up the sub-array extraction.
Gerrit
on 22 Jan 2020
I use load at the moment, I will try fread tomorrow. The data is sorted by the AeroDataId. For every AerodataId there are 70 points in the array. Is plain numeric array something else than a normal array?
Walter Roberson
on 22 Jan 2020
In MATLAB, many kinds of arrays are "normal", including cell arrays. "plain numeric array" is emphasizing on a simple real-valued double array especially one that is vector or 2d.
Guillaume
on 23 Jan 2020
If the data is saved in a mat file, fread won't help (unless you're planning to write your own mat file parser).
Gerrit
on 23 Jan 2020
Yeah, I just tested that. I think, the way I have it, is the best way to do it. Now I try to focus on vectorizing my loops.
If you want, you can take a look at my question. Link: https://de.mathworks.com/matlabcentral/answers/501586-can-i-vectorize-one-of-the-loops
Answers (0)
See Also
Categories
Find more on Logical in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!An Error Occurred
Unable to complete the action because of changes made to the page. Reload the page to see its updated state.
Select a Web Site
Choose a web site to get translated content where available and see local events and offers. Based on your location, we recommend that you select: .
You can also select a web site from the following list
How to Get Best Site Performance
Select the China site (in Chinese or English) for best site performance. Other MathWorks country sites are not optimized for visits from your location.
Americas
- América Latina (Español)
- Canada (English)
- United States (English)
Europe
- Belgium (English)
- Denmark (English)
- Deutschland (Deutsch)
- España (Español)
- Finland (English)
- France (Français)
- Ireland (English)
- Italia (Italiano)
- Luxembourg (English)
- Netherlands (English)
- Norway (English)
- Österreich (Deutsch)
- Portugal (English)
- Sweden (English)
- Switzerland
- United Kingdom (English)
Asia Pacific
- Australia (English)
- India (English)
- New Zealand (English)
- 中国
- 日本Japanese (日本語)
- 한국Korean (한국어)