Download all hours of all days of the year from CMORPH
7 views (last 30 days)
Show older comments
Augusto Gabriel da Costa Pereira
on 16 Dec 2022
Commented: VBBV
on 20 Dec 2022
This script below wants to download all hours of all days of the year from 2000 to 2005, but the following error appears:
I want to download all time data for 2000 2005 from this link: Index of /data/cmorph-high-resolution-global-precipitation-estimates/access/30min/8km (noaa.gov)
clc
clearvars
% Download the CPC data used in the script below
which_years = (2000:2005); % years to download
httpsUrl = "https://www.ncei.noaa.gov/data/cmorph-high-resolution-global-precipitation-estimates/access/30min/8km/";
% filename_tmp="temp.nc"; % temporary filename (for loop); data will be always stored under this temp name
%% main loop
mm = (1:12); % all monthes in year
% mm = [1 6]; % process only some specific monthes in year
for cc = 1:numel(which_years) % loop over years
this_year = which_years(cc);
for ci = 1:numel(mm) % loop over monthes
days_per_month = datenum(this_year, mm(ci)+1, 1) - datenum(this_year, mm(ci), 1); % compute how many days this month/year
for ck = 1:days_per_month % loop over days
ymd_str = datenum(this_year, mm(ck)+1, 1) - datenum(this_year, mm(ck), 1); % year month day string
for ch = 1:ymd_str
hora_str = strcat(sprintf('%04d',this_year), sprintf('%02d', mm(ci)), sprintf('%02d', ch));
Url = strcat(httpsUrl, sprintf('%04d', this_year), "/", sprintf('%02d', mm(ci)),"/");
filename = strcat("CMORPH_V1.0_ADJ_8km-30min_", hora_str ,".nc"); % create filename according to list above
disp(strcat(filename, " is processed"));
dataUrl = strcat(Url, filename);
data = webread(dataUrl);
filename_tmp=strcat("data_5", ymd_str ,".nc"); % temporary filename (for loop); data will be always stored under this temp name
filename_out = websave(filename_tmp,dataUrl); % save downloaded data
% ncdisp(filename_tmp,'/','min'); % commented, just to avoid filling the command window
precip=ncread(filename_tmp,'cmorph'); % Dimensions: lon,lat,time
long=ncread(filename_tmp,'lon');
lat=ncread(filename_tmp,'lat');
time=ncread(filename_tmp,'time');
end
end
end
end
The error reported is this:
Error using matlab.internal.webservices.HTTPConnector/copyContentToByteArray (line 396)
The server returned the status 404 with message "Not Found" in response to the request to
URL
https://www.ncei.noaa.gov/data/cmorph-high-resolution-global-precipitation-estimates/access/30min/8km/2000/01/CMORPH_V1.0_ADJ_8km-30min_20000101.nc.
Error in readContentFromWebService (line 46)
byteArray = copyContentToByteArray(connection);
Error in webread (line 125)
[varargout{1:nargout}] = readContentFromWebService(connection, options);
Error in Untitled (line 36)
data = webread(dataUrl);
1 Comment
Peter Perkins
on 19 Dec 2022
Two things:
1) You probably want to look at using a datastore for this. They are specifically designed for this kind of multiple file scenario.
2) You are not doing yourself any favors by using datenums. Use datetimes.
Accepted Answer
Augusto Gabriel da Costa Pereira
on 18 Dec 2022
Edited: Augusto Gabriel da Costa Pereira
on 19 Dec 2022
3 Comments
Augusto Gabriel da Costa Pereira
on 18 Dec 2022
Edited: Augusto Gabriel da Costa Pereira
on 18 Dec 2022
VBBV
on 20 Dec 2022
Ok, The problem was with error located in the line
data = webread(dataUrl);
which was resolved, Regarding the download of data, did you look at my updated comment ? . You need to use { } to store all files for years as shown below
filename_out{i+1} = websave(filename_tmp,dataUrl); % save /downloaded all data
More Answers (1)
VBBV
on 17 Dec 2022
Edited: VBBV
on 17 Dec 2022
Note the following changes
clc
clearvars
% Download the CPC data used in the script below
which_years = (2000:2005); % years to download
httpsUrl = "https://www.ncei.noaa.gov/data/cmorph-high-resolution-global-precipitation-estimates/access/30min/8km/";
% filename_tmp="temp.nc"; % temporary filename (for loop); data will be always stored under this temp name
%% main loop
mm = (1:12); % all monthes in year
% mm = [1 6]; % process only some specific monthes in year
for cc = 1:numel(which_years) % loop over years
this_year = which_years(cc);
for ci = 1:numel(mm) % loop over monthes
days_per_month = datenum(this_year, mm(ci)+1, 1) - datenum(this_year, mm(ci), 1); % compute how many days this month/year
for ck = 1:days_per_month % loop over days
ymd_str = datenum(this_year, mm(ci)+1, 1) - datenum(this_year, mm(ci), 1); % year month day string
for ch = 1:ymd_str
hora_str = strcat(sprintf('%04d',this_year), sprintf('%02d', mm(ci)), sprintf('%02d', ch));
Url = strcat(httpsUrl, sprintf('%04d', this_year), "/", sprintf('%02d', mm(ci)),"/",sprintf('%02d',ch),'/');
for i = 0:23
filename = strcat("CMORPH_V1.0_ADJ_8km-30min_", hora_str ,sprintf('%02d',i),".nc"); % create filename according to list above
disp(strcat(filename, " is processed"));
dataUrl = strcat(Url, filename);
data = webread(dataUrl);
filename_tmp=strcat("data_5", num2str(ymd_str) ,".nc"); % temporary filename (for loop); data will be always stored under this temp name
filename_out = websave(filename_tmp,dataUrl); % save downloaded data
% ncdisp(filename_tmp,'/','min'); % commented, just to avoid filling the command window
precip=ncread(filename_tmp,'cmorph'); % Dimensions: lon,lat,time
long=ncread(filename_tmp,'lon');
lat=ncread(filename_tmp,'lat');
time=ncread(filename_tmp,'time');
end
end
end
end
end
See Also
Categories
Find more on Downloads in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!