Saving data in background

I am trying to save a large amount of data (2.5GB in a large uint8 array, as well as a much smaller 3MB struct full of a few different types including doubles and char arrays). Due to time limitations, I would rather save this data in the background while other parts of the code are running. I am using the parallel toolbox.

I first tried batch('simple_save_script') [this is literally an .m file with

save('full_path_str','v_large_array', 'strct','-v3.7')

in it..

The error that I get back reads:

Error using batch (line 179)
An unexpected error occurred accessing properties: "CaptureDiary" "CreateDateTime" "CreateTime" "DependentFiles" "Diary" "Error"
"ErrorIdentifier" "ErrorMessage" "FinishDateTime" "FinishTime" "Function" "InputArguments" "DiagnosticWarnings" "Name"
"NumOutputArguments" "OutputArguments" "StartDateTime" "StartTime" "StateEnum" "Worker"

Caused by:
    Error using parallel.internal.cluster.FileSerializer>iSaveMat (line 281)
    Data too large to be saved.

Well darn it.

I then switched to parfeval, since this ought to do exactly as I want. My code is:

P = parfeval(@save, 0, 'full_path', 'v_large_array', 'strct','-v7.3');

When I output P, I get the following error:

                Error: Variable 'v_large_array' not found.

Okay, perhaps this is the size thing again... However, when I try the same code excluding 'v_large_array', I get the same error with 'strct'. In fact, I get the same error no matter what I try to pass into the parfeval function.

To see if it was the syntax that I wasn't understanding, I try the following code with feval:

feval(@save,'full_path', 'v_large_array', 'strct','-v7.3');

This runs as expected, and the data is saved.

So, what the heck can I do to save this data in the background? I have been trying to get this to work for several days now, and I have run out of ideas as this point...

Thank you in advance.

Answer 1

[Edric Ellis]

I fear that handing the data off to another process simply to save it will not actually give you any benefit. The reason for that is you have to send the 2.5GB of data across to the worker, which is a separate process. Sending data to a worker is in some ways equivalent to saving it to a file, and then loading it again on the worker - except that instead of a file, the "saved" data is kept in memory. Anyway. The problem you're having with parfeval is that you are sending to the worker only the name of the array that you would like to be saved, and not the data. This is because the save command itself expects to find the named variable in the calling workspace. To work around this, you need to create a tiny wrapper function. Something like this:

function saveData(fname, data, varargin)
  save(fname, 'data', varargin{:});
end

which then you can use like so:

parfeval(@saveData, 0, 'full_path', largeArray, '-v7.3')

[Zack Fifer]

Thank you for your reply. In regards to the time it takes to send the data to the worker, I am hoping that this can be done faster than saving the data to the disk (roughly 8s without compression), since it is all done in RAM. Actually saving the data is time consuming due to the compression of the file (something that I would like to keep). Saving the file with compression takes ~30s.

In regards to the save-wrapper, I have tried this as well, but I get the following error:

error using save_funct:
Too many output arguments.

This error occurs with the following code:

function save_funct(full_path, v_large_array, strct)
   save( full_path, 'v_large_array', 'strct', '-v7.3' )
end

And the following parfeval call:

parfeval(save_funct, 0, v_large_array, strct)

Once again, I am able to use save_funct in feval without problems.

As an aside, is there a workaround to using batch() for this process? Would batch() be somehow faster than parfeval?

Thank you for your help.

[Zack Fifer]

Bump.

A little help please?

[Walter Roberson]

Edric does not work weekends.

[Walter Roberson]

parfeval(@save_funct, 0, v_large_array, strct)

[Zack Fifer]

Sorry, just trying to increase visibility before Monday morning.

The '-v7.3' flag was a typo when writing this thread. This flag was not included in the actual parfeval call.

Answer 2

[Walter Roberson]

Look in the File Exchange for Shared Matrix.