Finding out number of columns with no NaNs

Hello,

I have a large dataset with 923 columns x 56 rows with some missing values (NaN) present. Ultimately I would like to find out the number of columns with no NaNs present in any of the 56 rows.

The data is in a structure flow.data and the column headers are in flow.textdata. I would like to create a new structure which only includes the columns without NaNs and has each column as a separate field with the corresponding column header as the fieldname.

I have written this bit of code but it's not really doing the job. It writes 923 fields in the new structure and in those fields it writes all of the original data (a 56x923 matrix). Could you please help me fix the code or suggest a better way to do things. Also, I was wondering how I can find out the number of fields in the new structure once I have created it successfully? I really appreciate the help!

varnames=flow.textdata(1:end);

for n=1:56     %number of rows
    for i=1:923    %number of columns
         for k=1:length(varnames);

      if annual_flow(:,i) == NaN;
         break

      else annual_flow(:,i) ~= NaN %not equal to NaN;
              new_structure.(varnames{k})(n,i)=annual_flow(n,i);
          end
      end
          end
      end
  end

Answer 1

[Andrei Bobrov]

t = ~any(isnan(flow.data));
a2 = flow.data(:,t);
c2 = [flow.textdata(t);mat2cell(a2,size(a,1),ones(size(a2,2),1))];
new_struct = struct(c2{:});

[Anna]

problem solved, thank you both!

[Anna]

Hi Andrei, could I get your help with one more thing? How can I modify the code so I could write the columns into new_struct that have up to 3 missing values (NaNs)? At the moment it's not allowing for any NaNs but I would like to be able to determine the number of NaNs to test different options. I'm assuming I need to modify the line t = ~any(isnan(flow.data)); but I'm not entirely sure how. thanks again.

[Andrei Bobrov]

Hi Anna!

t = sum(isnan(flow.data))>3;

Answer 2

[Jan]

1. The comparison with NaN must be implemented as isnan, because by definition every direct comparison with NaN is false.

2. If you use a vector sized condition for if, Matlab inserts an all automatically. But you look for any occurrence. Then:

if any(isnan(annual_flow(:, i)))

3. The number of columns without NaNs:

Num = sum(not(any(isnan(annual_flow), 1)))

Answer 3

[Anna]

Thanks a lot for the answer! I can't quite figure out though how to write the columns with no NaNs into a structure using the corresponding column headers as the fieldnames. Initially I only need the number of columns but for further analysis I need to know which column corresponds to which header and also to use the data stored in the columns. thanks again.