Read NetCDF Data Directly from Remote Locations

MATLAB^® supports reading and writing netCDF data on local file systems. MATLAB also supports reading netCDF data directly from remote file systems in these ways:

• You can use the MATLAB HDF5 interfaces with the URL of the file when these two conditions hold:

  • The file has format netcdf4.

  • The file is hosted on one of the platforms that MATLAB supports. For more information, see Supported Cloud Platforms.

• You can use the MATLAB netCDF interfaces with the URL of the file (with #mode=bytes appended to the end) when these two conditions hold:

  • The file is publicly hosted.

  • The file is hosted on a server that supports byte-range reading.

• You can use the MATLAB netCDF interfaces with the URL of the file when the file is hosted on an OPeNDAP server.

Read Cloud-Based NetCDF Data Using HDF5 Interface

The MATLAB high- and low-level HDF5 interfaces are cloud enabled, while the high- and low-level netCDF interfaces are not. However, because netCDF-4 files are based on the HDF5 format, you can use the MATLAB HDF5 interfaces to read netCDF-4 files on the cloud as well. The cloud storage location can be public or private. If the cloud storage location is private, follow the relevant authentication procedures.

This example shows how to read data from a remote netCDF-4 file that is publicly hosted on Amazon S3 using the MATLAB high-level HDF5 interface.

Get information about a netCDF-4 file using the h5info function.

urlFilePath = "s3://noaa-goes16/ABI-L1b-RadC/2017/242/00/";
urlFileName = "OR_ABI-L1b-RadC-M3C01_G16_s20172420002168_e20172420004540_c20172420004583.nc";
url = urlFilePath + urlFileName;

info = h5info(url)

info = struct with fields:
      Filename: 's3://noaa-goes16/ABI-L1b-RadC/2017/242/00/OR_ABI-L1b-RadC-M3C01_G16_s20172420002168_e20172420004540_c20172420004583.nc'
          Name: '/'
        Groups: []
      Datasets: [44×1 struct]
     Datatypes: []
         Links: []
    Attributes: [29×1 struct]

Select a variable to investigate from the file.

varName = info.Datasets(7).Name

varName = 
'band_wavelength'

Get a descriptive name of this variable using the h5readatt function.

varPath = "/" + varName;
longName = h5readatt(url,varPath,info.Datasets(7).Attributes(1).Name)

longName = 
'ABI band central wavelength'

Alternatively, you can get this descriptive name from the info structure.

info.Datasets(7).Attributes(1).Value

ans = 
'ABI band central wavelength'

Read the value of this variable using the h5read function.

bandWavelength = h5read(url,varPath)

bandWavelength = single
    0.4700

These caveats apply to using the HDF5 interfaces to read data from netCDF-4 files.

• This workflow involves using one interface (HDF5) to read data from a file of a different format (netCDF). You should have expertise in both netCDF and HDF5 file formats and their programmatic interfaces to use this workflow.

• The MATLAB HDF5 interfaces may not read data correctly for all netCDF-4 files.

Read Remote NetCDF Data Using Byte-Range Reading

This example shows how to read data from a remote netCDF file using the MATLAB high-level netCDF interface and byte-range reading. Reading netCDF data using byte-range reading can be slower than reading from other sources.

Get information about a netCDF file using the ncinfo function.

urlFilePath = "https://remotetest.unidata.ucar.edu/thredds/fileServer/testdata/";
urlFileName = "2004050300_eta_211.nc";
url = urlFilePath + urlFileName + "#mode=bytes";

info = ncinfo(url)

info = struct with fields:
      Filename: 'https://remotetest.unidata.ucar.edu/thredds/fileServer/testdata/2004050300_eta_211.nc#mode=bytes'
          Name: '/'
    Dimensions: [1×8 struct]
     Variables: [1×23 struct]
    Attributes: [1×5 struct]
        Groups: []
        Format: 'classic'
     Datatypes: []

Select a variable to investigate from the file.

varName = info.Variables(3).Name

varName = 
'datetime'

Read the value of the selected variable using the ncread function.

dateChars = ncread(url,varName)'

dateChars = 
'2003-09-25 00:00:00Z '

Read Remote NetCDF Data from OPeNDAP Server

This example shows how to read data from a remote netCDF file that is hosted on an OPeNDAP server using the MATLAB high-level netCDF interface. Reading netCDF data from an OPeNDAP server can be slower than reading from other sources if the data is stored on the server in a compressed format.

Get information about a netCDF file using the ncinfo function.

urlFilePath = "http://test.opendap.org/dap/data/nc/";
urlFileName = "bears_comp.nc.gz";
url = urlFilePath + urlFileName;

info = ncinfo(url)

info = struct with fields:
      Filename: 'http://test.opendap.org/dap/data/nc/bears_comp.nc.gz'
          Name: '/'
    Dimensions: [1×4 struct]
     Variables: [1×8 struct]
    Attributes: [1×2 struct]
        Groups: []
        Format: 'classic'
     Datatypes: []

Select a variable to investigate from the file.

varName = info.Variables(5).Name

varName = 
'order'

Read the value of the selected variable using the ncread function.

order = ncread(url,varName)

order = 3×2 int16 matrix

   1   4
   2   5
   3   6

Supported Cloud Platforms

MATLAB supports these cloud platforms:

• Amazon S3™ (Simple Storage Service)

• Azure^® Blob Storage (previously known as Windows Azure^® Storage Blob (WASB))

• Hadoop^® Distributed File System (HDFS™)

For more information about supported cloud platforms, see Work with Remote Data.

Note

Using copyfile to make a local copy of a remote netCDF file before working with it can be inefficient, especially if you want to read a small amount of data from a large file.