Using fminunc to optimize neural networks gives out of memory error

I'm working on a neural network problem, the neural network toolbox is good but I wanted to have more control over the behavious of the network, so instead of using the toolbox I wrote my own implementation of the neural networks.

I have a costFunction.m file which takes the input, output, parameters and returns the cost and gradient, then I use the fminunc() to optimize the cost function. My options are:

options = optimoptions('fminunc', 'Algorithm', 'trust-region', 'GradObj', 'on');

It worked fine when I was testing with small inputs, but when I was testing on a larger input (3800 examples with 10000 features each, 100 hidden units, and same number of outputs as inputs), I got out of memory error.

Out of memory. Type HELP MEMORY for your options.

Error in fminunc (line 371)
   Hstr = sparse(ones(n));

I have tried the 'memory' command, and here are the outputs:

Maximum possible array: 32357 MB
Memory available for all arrays: 32357 MB
Memory used by MATLAB: 1313 MB
Physical Memory (RAM): 16308 MB

I'm not familiar with the optimization toolbox, so I'm wondering is it possible that I've used the wrong algorithms or options on this problem? What optimization function and options should I use when I'm working on such huge datasets? Later on I will need to work on a even larger dataset with over 10000 examples, so I really need to solve this problem right now.

Thanks

Answer 1

[Matt J]

Not having worked with NN, I can't tell from your description how many unknowns you are solving for. However, the error message suggests that the number is large enough to make the Hessian difficult to store as a full matrix. If you are going to use the trust-region algorithm, you will need to choose some combination of the options 'Hessian', 'HessMult', and 'HessPattern' to avoid full, explicit computation of the Hessian.