Working with Large Data Sets

Connect to a Database with Maximum Performance

When you are using MATLAB^® with a database containing large volumes of data, you can experience out-of-memory issues or slow processing. To achieve the fastest performance, connect to your database using the native ODBC interface. If the native ODBC interface does not work, connect to your database using a JDBC driver. For details, see Connect to Database.

Import Large Data Sets into MATLAB

If you are selecting large volumes of data in a database to import into MATLAB, you can experience out-of-memory issues or slow processing. To achieve the fastest performance, you can import the data in batches.

When working with a native ODBC connection, the amount of memory available to MATLAB can restrict you from processing your whole set of data at once. To manage the MATLAB memory, process your data in parts. Use the fetch function to limit the number of rows your query returns by using the 'MaxRows' input argument. Using a MATLAB script, you can import data in increments until all data is retrieved. For an example, see fetch.

If you do not have access to a database and want to import large data sets, you can use the MATLAB interface to SQLite. For details, see Interact with Data in SQLite Database Using MATLAB Interface to SQLite.

Export Large Data Sets from MATLAB

When inserting large volumes of data into a database, you can experience slow processing. To achieve the fastest performance, use the sqlwrite function to export your data from MATLAB.

If you do not have access to a database and want to export large data sets, you can use the insert function with the MATLAB interface to SQLite. For details, see Interact with Data in SQLite Database Using MATLAB Interface to SQLite.

Access Large Data Using a DatabaseDatastore

An alternative for importing large data sets stored in a database into MATLAB is using a DatabaseDatastore. A DatabaseDatastore is a datastore that contains a collection of data stored in a database.

You can analyze data in a DatabaseDatastore using tall arrays with common MATLAB functions, such as mean and histogram. For details, see Analyze Large Data in Database Using Tall Arrays. Or, for more control, you can also write your own algorithms using MapReduce. For details, see Analyze Large Data in Database Using MapReduce.