How do I compute mean values of nine variables over Lat x lon x time?

I am attaching my netcdf matlab code to read the data from netcdf file. Input file netcdf is also attached. Please modify the code to compute mean values of nine meteorological variables from netcdf ERA5 hourly data. This file contains 00 and 12 GMT data. I would be highly obliged for your kind help.

I want daily average values from 00 and 12 GMT observations and also average values over lat and lon data. I have the total number of data points in input file 6961, so output file should have 3480 data points with nine variables(3481,9).

Sanchit

[Cris LaPierre]

User has deleted their original question.

How do I compute mean values of nine variables over Lat x lon x time?

I am attaching my netcdf matlab code to read the data from netcdf file. Input file netcdf is also attached. Please modify the code to compute mean values of nine meteorological variables from netcdf ERA5 hourly data. This file contains 00 and 12 GMT data. I would be highly obliged for your kind help.

I want daily average values from 00 and 12 GMT observations and also average values over lat and lon data. I have the total number of data points in input file 6961, so output file should have 3480 data points with nine variables(3480,9).

Sanchit

[Rena Berman]

(Answers Dev) Restored edit

Answer 1

[Cris LaPierre]

My understanding is that you want to take the mean of all the data for each day and time. This means you would lose the lat/lon and expver information.

I would take a different approach. I would turn the info into a timetable, and then use groupsummary to perform the mean by group.

unzip('matlab.zip','./');
ncfile = 'Lakhimpur_ERA5_daily_00_12_2014_2023.nc';

% Load the grouping data
lat = ncread(ncfile,'latitude');
lon = ncread(ncfile,'longitude');
expver = ncread(ncfile,'expver');
time = ncread(ncfile,'time');
time = datetime(double(time)*60*60,'ConvertFrom','epochtime','Epoch','1901-01-01');

% convert grouping data to 4x4x2x6961 arrays
[Lon,Lat,Expver,Time] = ndgrid(lon,lat,expver,time);

% Load the variables (4x4x2x6961)
var1 = ncread(ncfile,'d2m') ;
var2 = ncread(ncfile,'t2m') ;
var3 = ncread(ncfile,'e') ;
var4 = ncread(ncfile,'pev') ;
var5 = ncread(ncfile,'ssr') ;
var6 = ncread(ncfile,'ssrd') ;
var7 = ncread(ncfile,'tp') ;

d2m = var1 .* 0.00036854 + 288.2675;
d2m = d2m-273.15;
t2m = var2 .* 0.00038766 + 291.3707;
t2m = t2m - 273.15;
e = var3 .* 5.0661e-08 - 0.0030683;
pev = var4 .* 8.5836e-08 - 0.00278;
ssr = var5 .* 183.9069 + 12629668.0466;
ssrd =var6 .* 209.9315 + 14640023.0343;
tp =var7 .* 5.8624e-07 + 0.019209;

% DEWPOINTAND VAPOR PRESSURE DEFICIT EQUATIONS
% From Tetens Formula, the relation between temperature and the partial pressure of water vapor

es = 6.1078 .* exp((17.269 .* t2m) ./ (237.3+t2m));

% where,es is saturated vapor pressure in millibars and T is temperature in degrees C
% and the equation for relative humidity:
% Measure the air temperature T, in °C.
% Find out the dew point temperature Dp, in °C.
% Calculate relative humidity RH using the formula,

rh = 100 .* (exp(17.625 .* d2m ./ (243.04 + d2m)) ./ exp(17.625 .* t2m ./ (243.04 + t2m)));

% Rh=(ea/es)*100
% ea = Rh*es/100
% where, ea is the actual vapor pressure or vapor pressure at dewpoint temperature
% es is the saturation vapor pressure or vapor pressure at air temperature
% And,Vapor Pressure Deficit = es - ea at any instant in millibars

vpd = es - (rh .* es ./ 100);

% Create a timetable of all the data
Lon = Lon(:);
Lat = Lat(:);
Expver = Expver(:);
Time = Time(:);
d2m = d2m(:);
t2m = t2m(:);
e = e(:);
pev= pev(:);
ssr = ssr(:);
ssrd = ssrd(:);
tp = tp(:);
vpd = vpd(:);
rh = rh(:);

dataTbl = timetable(Time,Lon,Lat,Expver,d2m,t2m,e,pev,ssr,ssrd,tp,vpd,rh)

dataTbl = 222752×12 timetable
       Time         Lon      Lat     Expver     d2m       t2m          e           pev          ssr         ssrd          tp        vpd        rh  
    ___________    _____    _____    ______    ______    ______    __________    ________    _________    _________    ________    ______    ______

    01-Jan-2015    93.75     27.5      1        15.22    18.329    -0.0030683    -0.00278    1.263e+07    1.464e+07    0.019209    3.7721    82.096
    01-Jan-2015       94     27.5      1       15.221    18.329    -0.0030683    -0.00278    1.263e+07    1.464e+07    0.019209    3.7723    82.096
    01-Jan-2015    94.25     27.5      1       15.221     18.33    -0.0030683    -0.00278    1.263e+07    1.464e+07    0.019209    3.7725    82.095
    01-Jan-2015     94.5     27.5      1       15.222    18.331    -0.0030683    -0.00278    1.263e+07    1.464e+07    0.019209    3.7728    82.095
    01-Jan-2015    93.75    27.25      1       15.221     18.33    -0.0030683    -0.00278    1.263e+07    1.464e+07    0.019209    3.7725    82.095
    01-Jan-2015       94    27.25      1       15.221     18.33    -0.0030683    -0.00278    1.263e+07    1.464e+07    0.019209    3.7723    82.096
    01-Jan-2015    94.25    27.25      1       15.222    18.331    -0.0030683    -0.00278    1.263e+07    1.464e+07    0.019209    3.7724    82.096
    01-Jan-2015     94.5    27.25      1       15.222    18.331    -0.0030683    -0.00278    1.263e+07    1.464e+07    0.019209    3.7725    82.096
    01-Jan-2015    93.75       27      1       15.222    18.331    -0.0030683    -0.00278    1.263e+07    1.464e+07    0.019209    3.7727    82.095
    01-Jan-2015       94       27      1       15.222    18.331    -0.0030683    -0.00278    1.263e+07    1.464e+07    0.019209    3.7725    82.096
    01-Jan-2015    94.25       27      1       15.222    18.331    -0.0030683    -0.00278    1.263e+07    1.464e+07    0.019209    3.7724    82.097
    01-Jan-2015     94.5       27      1       15.222    18.331    -0.0030683    -0.00278    1.263e+07    1.464e+07    0.019209    3.7724    82.097
    01-Jan-2015    93.75    26.75      1       15.222    18.331    -0.0030683    -0.00278    1.263e+07    1.464e+07    0.019209    3.7726    82.096
    01-Jan-2015       94    26.75      1       15.222    18.331    -0.0030683    -0.00278    1.263e+07    1.464e+07    0.019209    3.7724    82.097
    01-Jan-2015    94.25    26.75      1       15.222     18.33    -0.0030683    -0.00278    1.263e+07    1.464e+07    0.019209    3.7723    82.097
    01-Jan-2015     94.5    26.75      1       15.221     18.33    -0.0030683    -0.00278    1.263e+07    1.464e+07    0.019209    3.7723    82.097

% Calculate the mean over latitude x longitude x time
data = groupsummary(dataTbl,["Time","Time"],["hourofday","day"],"mean",4:12)

data = 6961×12 table
    hourofday_Time     day_Time      GroupCount    mean_d2m    mean_t2m      mean_e      mean_pev    mean_ssr     mean_ssrd    mean_tp     mean_vpd    mean_rh
    ______________    ___________    __________    ________    ________    __________    ________    _________    _________    ________    ________    _______

          0           01-Jan-2015        32         15.222       18.33     -0.0030683    -0.00278    1.263e+07    1.464e+07    0.019209     3.7724     82.096 
          0           02-Jan-2015        32         15.221       18.33     -0.0030683    -0.00278    1.263e+07    1.464e+07    0.019209     3.7724     82.096 
          0           03-Jan-2015        32         15.221       18.33     -0.0030683    -0.00278    1.263e+07    1.464e+07    0.019209     3.7725     82.096 
          0           04-Jan-2015        32         15.221       18.33     -0.0030683    -0.00278    1.263e+07    1.464e+07    0.019209     3.7726     82.095 
          0           05-Jan-2015        32         15.222       18.33     -0.0030683    -0.00278    1.263e+07    1.464e+07    0.019209     3.7725     82.096 
          0           06-Jan-2015        32         15.222       18.33     -0.0030683    -0.00278    1.263e+07    1.464e+07    0.019209     3.7726     82.095 
          0           07-Jan-2015        32         15.222      18.331     -0.0030683    -0.00278    1.263e+07    1.464e+07    0.019209     3.7728     82.095 
          0           08-Jan-2015        32         15.221       18.33     -0.0030683    -0.00278    1.263e+07    1.464e+07    0.019209     3.7723     82.096 
          0           09-Jan-2015        32         15.221      18.329     -0.0030683    -0.00278    1.263e+07    1.464e+07    0.019209     3.7723     82.095 
          0           10-Jan-2015        32         15.221       18.33     -0.0030683    -0.00278    1.263e+07    1.464e+07    0.019209     3.7725     82.095 
          0           11-Jan-2015        32         15.221       18.33     -0.0030683    -0.00278    1.263e+07    1.464e+07    0.019209     3.7728     82.094 
          0           12-Jan-2015        32         15.222      18.332     -0.0030683    -0.00278    1.263e+07    1.464e+07    0.019209     3.7732     82.094 
          0           13-Jan-2015        32         15.222      18.331     -0.0030683    -0.00278    1.263e+07    1.464e+07    0.019209     3.7732     82.093 
          0           14-Jan-2015        32         15.221      18.331     -0.0030683    -0.00278    1.263e+07    1.464e+07    0.019209     3.7727     82.095 
          0           15-Jan-2015        32         15.221       18.33     -0.0030683    -0.00278    1.263e+07    1.464e+07    0.019209     3.7727     82.095 
          0           16-Jan-2015        32         15.222      18.331     -0.0030683    -0.00278    1.263e+07    1.464e+07    0.019209     3.7732     82.094 

You can then go on to process the data as you desire. See the Access Data in Tables page for help on working with table data. If you keep the results in a table, you will want to use writetable instead of writematrix.

[Devendra]

My understanding is that Sanchit wants daily average of all the data each day. That means daily average from 00 and 12 GMT also. Please modify your code to take daily average of all the data including time also. So the output table will have dimensions of (3480,11). Therefore hourofday_time will also vanish along with lat and Lon and expver information.

Thanks a lot for your kind help.

Devendra

[Cris LaPierre]

The table has the data for both times in it. Please use the links in my post to learn how to continue processing the data, including extracting the data for 00 and 12 GMT from the summary table created by groupsummary (3481 and 00, 3480 at 12).

[Torsten]

No. We may like to restore your question.

[Cris LaPierre]

Bringing the converstaion back here from your duplicate post.The summary table contains both data sets. Since I set the grouping as ["hourofday","day"], the first half of the table is 00 GMT, and the second half is 12 GMT. The only problem you have left to solve is splitting the table.

You can do that using any method of indexing you'd like. Again, from the linked page, in you want an array instead of a table as the result, your syntax would be T{rows,vars}

Make an attempt and share your code here if it doesn't work for more help.

[Devendra]

Hi Cris,

Is it possible to take mean of 00 and 12 GMT observations from the above code itself rather than splitting the table for 00 and 12 GMT and then computing the mean of 00 and 12GMT?

I also wanted to use your code if it computes the average of all the variables from 00 and 12 GMT daily observations for each day.

Devendra

[Sanchit]

Hello Cris,

Could you please suggest me how to write data in a csv file without truncated by matlab?

data = groupsummary(dataTbl,["Time","Time"],["hourofday","day"],"mean",4:12);

Thank you for your help

Sanchit

[Cris LaPierre]

The varialbe data is a table, so you should use writetable, as explained here.

What do you mean by truncated? What are you seeing that is unexpected?

[Sanchit]

I am using the following line to normalize the data between 0 to 1 directly from your code as follow;

data = groupsummary(dataTbl,["Time","Time"],["hourofday","day"],"mean",4:12)

% Normalize the data matrix\r\noutput = normalize(data,"range")

% Write normalized data in csv format

filename = 'c:/dummy.csv'

writematrix(output,filename);

but it is giving error.

May I request you to kindly suggest me how to get normalized data from above lines?

Thank you very much.

Sanchit

[Sanchit]

Alternatevly how to write the data upto 10 decimal points in a csv file.

Sanchit

[Walter Roberson]

EDIT: Adding deleted posts here

I am using the following line to normalize the data between 0 to 1 directly from your code as follow;

data = groupsummary(dataTbl,["Time","Time"],["hourofday","day"],"mean",4:12) 
% Normalize the data matrix 
output = normalize(data,"range"); 

% Write normalized data in csv format 
filename = 'c:/dummy.csv' writematrix(output,filename);

but it is giving error. May I request you to kindly suggest me how to get normalized data from above lines? Thank you very much.

Sanchit

Alternatevly how to write the data upto 10 decimal points in a csv file.

Sanchit

____________________________________________________________________________

@Sanchit

I am unclear about the datatypes of the variables to be written. If they are all double precision, then see @doc:dlmwrite and use a 'precision' option if you need more precision than writematrix() gives you. writematrix() is defined to use the equivalent of "format long g" which is %.15g . If you need to be able to reproduce double precision values exactly then you need at least %.17g (not sure at the moment if you ever need %.18g)

[Cris LaPierre]

@Sanchit, the error is because output is a table, and you are using writematrix to create the csv. You either need to use writetable (see here and here), or extract your numeric data to a numeric array (see here).

That still won't give you your data to 10 significant figures, but it will correct the error.

Also, please stop deleting your posts.

[Sanchit]

Hi Cris,

I have downloaded the ERA5 data of 3,6,9 and 12 GMT with 8 variables as netcdf file. I am using your code to calculate mean of 8 variables over lat x lon x time. This is giving the following error

Error using matlab.internal.math.checkDataVariables

Invalid data variables.

Error in groupsummary (line 252)

dataVars = matlab.internal.math.checkDataVariables(T, varargin{indStart},

"groupsummary", "Data");

Error in Read_netCDF_file (line 67)

data = groupsummary(dataTbl,["Time","Time"],["hourofday","day"],"mean",4:12)

I humbaly request you to please suggest me how to fix it. I am attaching the input file alongwith matlab code.

I would appreciate your kind help.

Sanchit

[Cris LaPierre]

The issue is exactly what the error message says: "invalid data variables" in Read_netCDF_file (line 67)

data = groupsummary(dataTbl,["Time","Time"],["hourofday","day"],"mean",4:12)

For reference, here is the syntax that line of code is using:

G = groupsummary(T,groupvars,groupbins,method,datavars)

[Sanchit]

May I request you to please provide me the correct form of variables in the following line

data = groupsummary(dataTbl,["Time","Time"],["hourofday","day"],"mean",4:12)

I would be very grateful to you for your kind help.

Sanchit

[Cris LaPierre]

Let's start simple. Look at your data. How many variables (columns) are there? Which columns of the table do you want to take the mean of?

[Sanchit]

There are 11 variables including time.

Time Lon Lat d2m t2m cdir ssr ssrc ssrdc ssrd tp

I want to take mean of 8 variables. Please guide me to fix it.

Sanchit

[Cris LaPierre]

As a technicality, in a timetable, the Time column doesn't count, so technically there are 10.

I'm not sure if you remember, but in the original data file that spawned this question, there were 12 columns, and the code you are using was written for the original data set, not your new one. You need to update the datavars input accordingly.

For reference, here is the syntax my code is using:

G = groupsummary(T,groupvars,groupbins,method,datavars)