Feed data into Neural Networks file-by-file
2 views (last 30 days)
Show older comments
Currently for training my neural network, I need to load a huge dataset and it costs almost available RAM in the computer. That's why the training step (using GPU) is very slow. The pseudo-code is following:
Loading and pre-processing dataset (300 files)
Training Neural Network
So how can I don't have to load all at one time, and will be loaded in time the neural network is being trained, like this:
For file in Dataset
Loading and pre-processing file
Training Neural Network
or
Training Neural network
At some epoch:
Loading and Pre-processing a file in dataset
Continue to train.
Does MATLAB currently support this feature?
0 Comments
Answers (1)
Joss Knight
on 1 Nov 2019
Datastores are designed for precisely this purpose. It may be that you're after an imageDatastore processed by a transform.
0 Comments
See Also
Categories
Find more on Image Data Workflows in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!