I have run into a problem where I need to train a LSTM signal classifier with huge amount of data.
Each 1D signal is around 100k samples, every 48 signals are saved in a .mat file. The total number of .mat files is around 2000.
The labels are similarly saved in corresponding .mat files in a different folder.
I would like to know if there's a way to train the network without the necessity of loading the whole thing into memory. (with 64GB ram I can only load ~1300files at once)
Your help will be very much appreciated.