Skip reading a file with file datastore

I'm using a file datastore to read log files. I'm using a while loop to read files in datastore. In some cases the file reader is not able to read a file and will throw an error. If this happens, the file datastore does not move forward to the next file and is stucked trying to read the same file.

I would like to be able to continue reading the next file in the datastore if this happens.

Are there a nextfile(fds) function or similar that can change the progress of the datastore?

Alternatively I have to make a recursive call for a new file datastore based on the remaing subset of data and merge the output together. I would like to avoid that.

% Read through the file datastore and store data
ii=1;
reset(fds)
hwait = waitbar(0, '0', 'Name', 'Reading data');
while hasdata(fds)
    try
        Data{ii, :} = read(fds);
    catch
         % flag file and move to next
         flaggedfiles{end+1} 
    end
    ii = ii + 1;
    waitbar(progress(fds), hwait, sprintf('%2.0f %%', progress(fds) * 100));
end

Answer 1

[praguna manvi]

Hi Heine Hørup Pedersen,

As I understand, you are trying to achieve the behavior of "next(fds)" on a "FileDatastore" object "fds" during an "Exception" in a loop while updating the "waitbar". Although there is no built-in support to perform a file-skip, "@fcn" can be configured to "partialfile," which allows a "done" flag for this effect after error handling within the "@fcn" function.

Here is an example that simulates this behavior:

% Create a temporary directory
tempDir = tempname;
mkdir(tempDir);

% Create some text files in the directory
file1 = fullfile(tempDir, 'file1.txt');
file2 = fullfile(tempDir, 'file2.bin'); % Binary file to simulate error
file3 = fullfile(tempDir, 'file3.txt');

% Write data to the files
fid1 = fopen(file1, 'w');
fprintf(fid1, '1\n');
fclose(fid1);

fid2 = fopen(file2, 'w');
fwrite(fid2, 'double');
fclose(fid2);

fid3 = fopen(file3, 'w');
fprintf(fid3, '3\n');
fclose(fid3);

% Create a fileDatastore
fds = fileDatastore(tempDir, 'ReadFcn', @readTextFile, 'FileExtensions', {'.txt', '.bin'});

% Function to read text files
function [data,userdata,done] = readTextFile(filename, userdata)
    fid = fopen(filename, 'r');
    if fid == -1
        error('File cannot be opened.');
    end
    data = fread(fid, '*char')';
    fclose(fid);
    try
        validateattributes(str2double(data), "double", {"nonnan"});
    catch ME
        fprintf('Inner Error message: %s\n', ME.message);
    end
    done = 1;
end

% Initialize variables
ii = 1;
reset(fds);
hwait = waitbar(0, '0', 'Name', 'Reading data');
flaggedfiles = {}; % Initialize the flagged files list

% Read through the file datastore and store data
while hasdata(fds)
    try
        Data{ii, :} = read(fds);
        ii = ii + 1; % Increment the index only if read is successful
    catch ME
        % flaggedfiles{end+1}
        fprintf('Outer Error message: %s\n', ME.message);
    end
    
    % Update the waitbar and introduce a short pause
    waitbar(progress(fds), hwait, sprintf('%2.0f %%', progress(fds) * 100));
    pause(0.5); % Pause for half a second to make the waitbar visible longer
end

close(hwait);

% Display results
disp('Data read from files:');
disp(Data);

For more information about “FileDatastore” properties refer to this link below:

https://www.mathworks.com/help/matlab/ref/matlab.io.datastore.filedatastore.html

Hope this helps!

[Heine Hørup Pedersen]

Thank you Praguna

The solution lies in handling the error with try, catch in the reader file as you suggest.