Problem extracting 2D array from multiple folders

I want to write a code that will access a folder that contains 10 subfolders each containing 600 of .dat files. The .dat files contains 2500 elements. The aim is to build an array of 2500x600x10 to plot a 3-D image.

I have written a code that can access the subfolders and extract the .dat files from them but I have problem in getting the 2D array from each iteration of the loop. Here is my code and the link to the data ( https://www.dropbox.com/sh/tme38shk7q3reqm/AABiNghFVpZP4DS1da_8dP-7a?dl=0 )

% Start with a folder and get a list of all subfolders.
% Finds and prints names of all .datin that folder and all of its subfolders.
clc;    % Clear the command window.
clear all;close all;
workspace;  % Make sure the workspace panel is showing.
format longg;
format compact;
myFolder= '/Users/.../Dropbox/LV4Total';

% Define a starting folder.
start_path = fullfile(myFolder);
% Ask user to confirm or change.
topLevelFolder = uigetdir(start_path);
if topLevelFolder == 0
	return;
end
% Get list of all subfolders.
allSubFolders = genpath(topLevelFolder);
% Parse into a cell array.
remain = allSubFolders;
listOfFolderNames = {};
while true
	[singleSubFolder, remain] = strtok(remain, ':');
	if isempty(singleSubFolder)
		break;
	end
	listOfFolderNames = [listOfFolderNames singleSubFolder];
end

numberOfFolders = length(listOfFolderNames);


no_a_scans=0;
image_number=1;
 
a_scans_per_image=600;
 
 
 
% Process all image files in those folders.k starts from 2 to exclude the base folder.
for k = 2: numberOfFolders
    % Get this folder and print it out.
    thisFolder = listOfFolderNames{k};
    fprintf('Processing folder %s\n', thisFolder);
    
    
 filePattern = fullfile(thisFolder, '*.dat');
  jpegFiles = dir(filePattern);
  
for p = 1:a_scans_per_image
    
  baseFileName = jpegFiles(p).name;
  no_a_scans=no_a_scans+1;
  image_number=image_number+1;
  fullFileName = fullfile(myFolder, baseFileName);
  fprintf(1, 'Now reading %s\n', fullFileName);
  fullFileName
  A_scan_Array_new(:,no_a_scans,image_number) = readmatrix(fullFileName)';
 
  drawnow; % Force display to update immediately.
end
 
end
no_a_scans
% Data_3D=A_scan_Array_new(:,:,1);

[Ive J]

And what error do you get?

[Akeem Azeez]

Error using importdata (line 139)

Unable to open file.

Error in untitled3 (line 58)

A_scan_Array_new(:,no_a_scans,image_number) = importdata(fullFileName);

[Cris LaPierre]

Please don't change the question after it's been answered. It makes the post unintelligible for those who come across it later. I've added your new code to your corresponding reply below and reverted the code in the question back to what it was originally.

[Akeem Azeez]

Thanks, I am new to this.

Answer 1

[Cris LaPierre]

The only way I could get your code to run was to duplicate a portion of your dropbox folder structure (LV4Total/LV_4 and LV4Total/LV4) and then, when asked to confirm the top level folder, select LV4Total.

I then got the same error message you report. However, when I look at the displayed value of fullFileName, I can see that it is missing the last folder name.

Processing folder C:\Users\...\LV4Total\LV4
                                        ^^^ % file is in folder Lv4
Now reading .\LV4Total\000_100.dat
                       ^ % Path is now missing the LV4 older
fullFileName = '.\LV4Total\000_100.dat'
                           ^ % Path is now missing the LV4 older
Error using importdata (line 139)
Unable to open file.

I think you need to change what is currently line 51 to the following.

fullFileName = fullfile(thisFolder, baseFileName);
                        ^^^^^^^^^^ % use thisFolder instead of myFolder                       

That at least gives a new error that you can work with.

Processing folder C:\Users\...\LV4Total\LV4
Now reading C:\Users\...\LV4Total\LV4\000_100.dat
                                  ^^^ % Path is now correct
fullFileName = 'C:\Users\...\LV4Total\LV4\000_100.dat'
                                      ^^^ % Path is now correct
Index exceeds the number of array elements (1).

Now just look at fixing your indexing (if needed). Again, I only pullied in one dat file. Your code is looking for 600, so if there are 600 in each folder, you should be ok.

[Akeem Azeez]

Thank you for this, I worked overnight on the code and I was able fix the problem, now I am having another challenge of saving the 2500x600 from each iteration. My aim is to harvest the 10 2Ds and concatenate them to get 2500x600x10 3D.

I have modified the code in the question to reflect this. Thanks

% Start with a folder and get a list of all subfolders.
% Finds and prints names of all .datin that folder and all of its subfolders.
clc;    % Clear the command window.
clear all;close all;
workspace;  % Make sure the workspace panel is showing.
format longg;
format compact;


a_scans_per_image=2;

%im=1;             % select Image number



%%
myFolder= '/Users/.../Dropbox/LV4Total';

% Define a starting folder.
start_path = fullfile(myFolder);
% Ask user to confirm or change.
topLevelFolder = uigetdir(start_path);
if topLevelFolder == 0
	return;
end
% Get list of all subfolders.
allSubFolders = genpath(topLevelFolder);
% Parse into a cell array.
remain = allSubFolders;
listOfFolderNames = {};





while true
	[singleSubFolder, remain] = strtok(remain, ':');
	if isempty(singleSubFolder)
		break;
	end
	listOfFolderNames = [listOfFolderNames singleSubFolder];
end
numberOfFolders = length(listOfFolderNames);


no_a_scans=0;
image_number=0;
count=0;


% Process all image files in those folders.
for k = 11: numberOfFolders
	% Get this folder and print it out.
	thisFolder= listOfFolderNames{k};
	%fprintf(1,'%s\n', thisFolder);
    
    new_thisFolder=thisFolder;

    
filePattern = fullfile(thisFolder, '*.dat');
jpegFiles = dir(filePattern);

 image_number=image_number+1;
%     
for k= 1:a_scans_per_image
     baseFileName = jpegFiles(k).name;
  no_a_scans=no_a_scans+1;
  
  fullFileName = fullfile(thisFolder, baseFileName);
  fprintf(1, 'Now reading %s\n', fullFileName);
 A_scan_Array_new(:,no_a_scans) = importdata(fullFileName);
 
  %drawnow; % Force display to update immediately.

end
Data_real{image_number}=A_scan_Array_new(:,:,1);

end

[Cris LaPierre]

If you can create the 2500x600, it should be pretty simple to create the 2500x600x10 using indexing. For example, this code creates a 4x5x2 matrix in a manner similar to what you are trying to do.

for a=1:2
    for b = 1:5
        c(:,b,a)=rand(1,4);
    end
end
c

c = 
c(:,:,1) =

    0.1852    0.3788    0.9872    0.4711    0.9075
    0.6690    0.9966    0.2562    0.5891    0.1045
    0.0584    0.6455    0.6144    0.8767    0.7039
    0.2828    0.4754    0.0368    0.0858    0.1117


c(:,:,2) =

    0.0148    0.6923    0.8732    0.2379    0.8985
    0.7575    0.2233    0.8375    0.2680    0.5318
    0.6784    0.6010    0.5377    0.7956    0.8178
    0.9225    0.4016    0.7719    0.4815    0.4935

[Akeem Azeez]

Thank you very much!!!