Downloading Specific Files from Amazon S3 Bucket

Hello,

I am trying to download some files from here: https://fcon_1000.projects.nitrc.org/indi/retro/MPI_LEMON.html. I am using Cyberduck to download files for 200+ people. The problem is that for each person I only need a specific file found within 2-3 subfolders. My current tactic is to just download everything and select the files I am interested in. The problem is that my internet connection is not that great and it will probably take me a week to do so.

Is there a way to do this matlab, so I can download only the specific files I want?

Answer 1

[Samay Sagar]

Hi @Orestis Stylianou,

To download specific files from a website using MATLAB, you can use the “websave” function. This function is generally used for downloading files when you have direct URLs.

If the files are nested within subfolders and you need to navigate through directories, you might need to perform web scraping to list and access the specific files.

If possible, find a pattern in the URLs of the files you want to download. This will allow you to construct the URLs programmatically.

Here is an approach you can follow to automate downloading specific files using MATLAB:

% Base URL
baseURL = "your_base_url";

% List of subjects and specific files
subjects = {'subject1', 'subject2', 'subject3'}; % Replace with actual subject IDs
filename = 'specific_file.ext'; % Replace with the actual filename you need

% Loop through each subject and download the specific file
for i = 1:length(subjects)
    subjectURL = [baseURL, subjects{i}, '/path/to/subfolder/', filename];
    outputFile = fullfile('local_directory', [subjects{i}, '_', filename]); % Save with a unique name
    
    try
        websave(outputFile, subjectURL);
        fprintf('Downloaded: %s\n', outputFile);
    catch
        fprintf('Failed to download: %s\n', subjectURL);
    end
end

Given your slow internet connection, ensure you have a stable connection when running the script. Consider implementing error handling to retry downloads in case of network issues.

For more information about “websave” you can refer the following documentation:

https://www.mathworks.com/help/releases/R2024a/matlab/ref/websave.html

[Orestis]

Hi, I ended up using a Python library to do it. If I need to do it again in the future I will check websave, thanks