Parfoor loop with 3D matrix

Hello,

I write you today because I have to treat 655Mo of data (matrix 16*512*10000 of double) and before the treatment I use a filtering to reduce noise and center my signals. The structure data contains the signals at the beginning, and also différent parameters like the length of each signals and the number of signals for each sensor. Moreover, the structure parametres contains different variables relative to these signals (sampling frequency, signal frequency, number of sensors). For this, I do:

% Hanning Window
g = ones(data.sig_length, data.nb_signals);
h = hann(31);
g(1:15,:) = repmat(h(1:15), 1, data.nb_signals);
g(data.sig_length-14:data.sig_length, :) = repmat(h(17:31), 1, data.nb_signals);

% Low pass filter coefficient
N = 3;
Fe = parametres.freq_ech;         % 20 MHz
Fce = parametres.freq_central;    % 2  MHz
Wn = (2*Fce)/(Fe/2);
[b, a] = butter(N, Wn, 'low');

% Variable Initialisations
signal_temp = zeros(data.sig_length, data.nb_signals);
moyenne_sig = zeros(data.sig_length, data.nb_signals);
signal_filt = zeros(data.sig_length, data.nb_signals, parametres.text_nb_sensors);

donnees = data.signal_init;
sig_length = data.sig_length;

for sensor = 1:parametres.text_nb_sensors
    % DC Suppression
    signal_temp = squeeze(donnees(:,:,sensor));
    moyenne_sig = repmat(mean(squeeze(donnees(:,:,sensor))), sig_length, 1);
    signal_filt(:,:,sensor) = signal_temp - moyenne_sig;

      % Median filter on 5 points
      for k = 1:1:data.sig_length
          signal_filt(k,:,sensor) = medfilt1(squeeze(signal_filt(k,:,sensor)),5);    
      end

      % Low pass filter with Hanning window
      signal_filt(:,:,sensor) = filtfilt(b,a,g.*(signal_filt(:,:,sensor)));
  end

Until here, no problem, I have the result I want and my treatments after are okay. But this take 19 seconds, so it is too much time for me. I have the toolbox to use the 4 cores of my computers, what I use in another place on my code. My wish is to do the same thing here but after several tries, I always have Matlab which informs me that a variable have not the good indices to be in a parfor loop. For example with the code above, it is the nested-for loop which is the problem.

Somebody could help me to begin somewhere?

I thank you a lot in advance. Best regards.

Answer 1

[Edric Ellis]

I think the problem here is with the way you're using signal_filt. You're using multiple forms of indexing, and the parfor restrictions do not allow that, as described in the documentation. Here's what I think you need to do to fix this, something like this:

    parfor sensor = 1:parametres.text_nb_sensors
        % DC Suppression
        signal_temp = squeeze(donnees(:,:,sensor));
        moyenne_sig = repmat(mean(squeeze(donnees(:,:,sensor))), sig_length, 1);
        tmp = signal_temp - moyenne_sig;
        % Median filter on 5 points
        for k = 1:1:data.sig_length
            tmp(k) = medfilt1(squeeze(tmp(k)),5);
        end
        % Low pass filter with Hanning window
        signal_filt(:,:,sensor) = filtfilt(b,a,g.*(tmp));
    end

Note I'm using tmp to build up the value to be placed back into a slice of signal_filt. This is sufficient to convince parfor that you're not doing anything order-dependent.

[Mathieu]

Thank you! It was just tmp(k, :) in the nested-loop but it was what I was looking for.

When I activate the multicores with parpool(4), it consume 800Mo of RAM in my computer. Is it normal it is so much?

I tested this piece of codes with my larger data (640Mo input, 640Mo output, so 1,3Go total) but, event using the WorkerObjWrapper of FEX, I explode the RAM of my computer which crash because I use too much RAM (0Mo available). So I think I have to rethink how to use this parfor loop, maybe with less workers.