Reading multiple nc files in different subfolders within a directory

Hi,

My data is stored in a folder withch contains subfolders cyc001, cyc002, cyc003...cyc258 each with 15 .nc files. I have written this code to explore the main data data directory and each cyc* subdirectories (total 258),read data from 15 nc files, retrieve certain fields in each .nc filesand store/save as output (passes). My code works well for one cyc001 folder but it failes to read the 2nd and subsequen folders.

currentdir= '/Users/Documents/test1'; 
cd(currentdir);
% For each cyc subdirectory
list_cycle= textscan(ls(),'%s');
cyclelist=list_cycle{1,1}(:,:)
j=1;
for c=1:length(cyclelist)
    cd(cyclelist{c,1});%change into current cyc
    list_dir= textscan(ls('-d','*.nc'),'%s');
     dirlist=list_dir{1,1}(:,:)
     %read nc files    
     myFolder=pwd;% folder of each cy
        filePattern = fullfile(myFolder, '*.nc'); 
        theFiles = dir(filePattern);
        findFileInFolder(pwd,{'.nc'})
        % Loop for each nc-file
         for k = 1:length(theFiles)
          baseFileName = theFiles(k).name;
          fullFileName = fullfile(myFolder, baseFileName);
          ncid=netcdf.open(fullFileName, 'NC_NOWRITE');
          varname= netcdf.inqVar(ncid,4);
          varid = netcdf.inqVarID(ncid,varname);
          netcdf.close(ncid);
          end
          ncfiles = dir('*.nc');   % get all nc files in the folder 
          nfiles = length(ncfiles) ;   % total number of files 
end
    % For each pass (.nc file)
    for d=1:length(dirlist)
        cd(dirlist{d,1});
        data=read_nc(fullFileName); % Read the selected nc files using read_nc func 
       fprintf('Reading track: %d/%d of pass %d/%d. To read tracks: %d\n',d, length(dirlist),c,length(cyclelist),i);
       ind=find((data.glat.Value<lat1)&(data.glat.Value>lat2)&(data.glon.Value<lon1|data.glon.Value>lon2));%find the indices 
        passe(j).lat= data.glat.Value(ind);     % copy the fields in the data structure
        passe(j).lon= data.glon.Value(ind); % copy the fields in the data structure
        j=j+1;
       clear data;
      cd('..')
    end
    cd('..')
cd(currentdir)

It gives me eorror that

Error using cd

Cannot CD to cycle002 (Name is nonexistent or not a directory).

Ithink porblem is where I am trying to read nc files in each cyc... subfolder. Anyone, please help to fix this loop? I am not an expert matlab user so maybe I cam forgetting something and doing something wrong here.

[Stephen23]

Do NOT use cd like that. Do NOT change directories just to access data files. It is faster and more robust to use absolute/relative filenames, which you can easily construct using fullfile.

Do NOT use ls in code to get a list of folder contents (it is intended for visual display, not for processing in code). Use dir instead.

Answer 1

[Stephen23]

You should start again, because all of those cd calls are absolute chaos.

Start with something like this:

D = '/Users/Documents/test1';
S = dir(fullfile(D,'cyc*'));
S = {S([S.isdir]).name};
Z = struct('lat',{},'lon',{});
for ii = 1:numel(S)
    T = dir(fullfile(D,S{ii},'*.nc'));
    T = {T.name};
    for jj = 1:numel(T)
        F = fullfile(D,S{ii},T{jj})
        ... your code
        Z(ii,jj).lat = ...
        Z(ii,jj).lon = ...
        end
    end
end

https://www.mathworks.com/help/matlab/import_export/process-a-sequence-of-files.html

[skanwal]

Hi Stephen, I simply delete the redundant lines and rewrite the code according to the format you advised after adding the above code. It worked. I really ppreciate your help. Thank you,

Answer 2

[KSSV]

ncfiles = dir('*nc') ; 
N = length(ncfiles) ;
for i = 1:N
     ncfile = ncfiles(i).name ;
    % do what you want 
end

[skanwal]

Hi KSSV, It did not solve the problem.