Interfacing to Data at NCI

PyEarthTools provides a standardised interface to data. Code is already written to nicely interface to the full ERA5 that’s on disk at NCI (not just the 5.625 degree data from weatherbench). We’ll start by taking a look at the PyEarthTools ERA5 interface, then walk through the process of connecting to the low-res dataset to see how to connect to a new data source.

import pyearthtools.data
import pyearthtools.pipeline
import warnings

with warnings.catch_warnings(action="ignore"):
    import site_archive_nci

pyearthtools.data.archive.NCI?

Type:        module
String form: <module 'site_archive_nci' from '/g/data/kd24/tjl/src/nci/src/site_archive_nci/__init__.py'>
File:        /g/data/kd24/tjl/src/nci/src/site_archive_nci/__init__.py
Docstring:
National Computing Infrastructure specific Indexes

| Name        | Description |
| :---        |       ----: |
| [ERA5][site_archive_nci.ERA5]                | ECWMF ReAnalysis v5       |
| [ACCESS][site_archive_nci.ACCESS]            | Australian Community Climate and Earth-System Simulator       |
| [AGCD][site_archive_nci.AGCD]                | Australian Gridded Climate Data        |
| [BRAN][site_archive_nci.BRAN]                | Bluelink ReANalysis        |
| [OceanMaps][site_archive_nci.OceanMaps]      | Ocean Modelling and Analysis Prediction System        |
| [MODIS][site_archive_nci.MODIS]              | MODerate resolution Imaging Spectroradiometer       |
| [Himawari][site_archive_nci.Himawari]        | Himawari 8/9 satellite data       |
| [BARRA][site_archive_nci.BARRA]              | Bureau of meteorology Atmospheric high-resolution Regional Reanalysis for Australia       |
| [BARPA][site_archive_nci.BARPA]              | Bureau of Meteorology Atmospheric Regional Projections for Australia       |
| [BARRA_V2][site_archive_nci.BARRA_V2]        | Bureau of meteorology Atmospheric high-resolution Regional Reanalysis for Australia v2    |

var=['u', 'v']  # Note - there is no really straightforward way to just list the variables in the archive
                # However, mismatches will cause PyEarthTools to list what's available with a "did you mean" prompt
                # A specific listing function should be added in future.
UandV = pyearthtools.data.archive.ERA5(var)
UandV

ERA5
    Description                    ECWMF ReAnalysis v5
             range                          '1970-current'
             Documentation                  'https://confluence.ecmwf.int/display/CKB/ERA5%3A+data+documentation'


    Initialisation
             level_value                    None
             product                        'reanalysis'
             variables                      ['u', 'v']
    Transforms
             StandardCoordinateNames        {'latitude': "['lat', 'Latitude', 'yt_ocean', 'yt']", 'longitude': "['lon', 'Longitude', 'xt_ocean', 'xt']", 'replacement_dictionary': 'None', 'time': "['Time']"}
             Rename                         {'names': {'t2m': "'2t'", 'u10': "'10u'", 'v10': "'10v'", 'siconc': "'ci'"}}

ERA5

Description:

ECWMF ReAnalysis v5

• range
  '1970-current'
• Documentation
  https://confluence.ecmwf.int/display/CKB/ERA5%3A+data+documentation

• Initialisation:
  • level_value
    None
    None
  • product
    'reanalysis'
    'reanalysis'
  • variables
    ['u', 'v']
    - u
    - v
• Transforms:
  • StandardCoordinateNames
    {'latitude': ['lat', 'Latitude', 'yt_ocean', 'yt'], 'longitude': ['lon', 'Longitude', 'xt_ocean', 'xt'], 'replacement_dictionary': None, 'time': ['Time']}
    latitude : ['lat', 'Latitude', 'yt_ocean', 'yt']
    longitude : ['lon', 'Longitude', 'xt_ocean', 'xt']
    replacement_dictionary : None
    time : ['Time']
  • Rename
    {'names': {'t2m': '2t', 'u10': '10u', 'v10': '10v', 'siconc': 'ci'}}
    names : {'t2m': '2t', 'u10': '10u', 'v10': '10v', 'siconc': 'ci'}

UandV['1984-01-01']  # ERA5 is an analysis product, so is indexed by its analysis time. This request
                     # fetches all of the data from a particular date

<xarray.Dataset> Size: 15GB
Dimensions:    (longitude: 1440, latitude: 721, level: 37, time: 24)
Coordinates:
  * longitude  (longitude) float32 6kB -180.0 -179.8 -179.5 ... 179.5 179.8
  * latitude   (latitude) float32 3kB 90.0 89.75 89.5 ... -89.5 -89.75 -90.0
  * level      (level) int32 148B 1 2 3 5 7 10 20 ... 875 900 925 950 975 1000
  * time       (time) datetime64[ns] 192B 1984-01-01 ... 1984-01-01T23:00:00
Data variables:
    u          (time, level, latitude, longitude) float64 7GB dask.array<chunksize=(4, 5, 405, 900), meta=np.ndarray>
    v          (time, level, latitude, longitude) float64 7GB dask.array<chunksize=(4, 5, 405, 900), meta=np.ndarray>
Attributes:
    Conventions:  CF-1.6
    license:      Licence to use Copernicus Products: https://apps.ecmwf.int/...
    summary:      ERA5 is the fifth generation ECMWF atmospheric reanalysis o...

xarray.Dataset

• Dimensions:
  • longitude: 1440
  • latitude: 721
  • level: 37
  • time: 24
• Coordinates: (4)
  • longitude
    (longitude)
    float32
    -180.0 -179.8 ... 179.5 179.8

    units :

    degrees_east

    long_name :

    longitude

    array([-180.  , -179.75, -179.5 , ...,  179.25,  179.5 ,  179.75],
          dtype=float32)

  • latitude
    (latitude)
    float32
    90.0 89.75 89.5 ... -89.75 -90.0

    units :

    degrees_north

    long_name :

    latitude

    array([ 90.  ,  89.75,  89.5 , ..., -89.5 , -89.75, -90.  ], dtype=float32)

  • level
    (level)
    int32
    1 2 3 5 7 ... 900 925 950 975 1000

    units :

    millibars

    long_name :

    pressure_level

    array([   1,    2,    3,    5,    7,   10,   20,   30,   50,   70,  100,  125,
            150,  175,  200,  225,  250,  300,  350,  400,  450,  500,  550,  600,
            650,  700,  750,  775,  800,  825,  850,  875,  900,  925,  950,  975,
           1000], dtype=int32)

  • time
    (time)
    datetime64[ns]
    1984-01-01 ... 1984-01-01T23:00:00

    long_name :

    time

    array(['1984-01-01T00:00:00.000000000', '1984-01-01T01:00:00.000000000',
           '1984-01-01T02:00:00.000000000', '1984-01-01T03:00:00.000000000',
           '1984-01-01T04:00:00.000000000', '1984-01-01T05:00:00.000000000',
           '1984-01-01T06:00:00.000000000', '1984-01-01T07:00:00.000000000',
           '1984-01-01T08:00:00.000000000', '1984-01-01T09:00:00.000000000',
           '1984-01-01T10:00:00.000000000', '1984-01-01T11:00:00.000000000',
           '1984-01-01T12:00:00.000000000', '1984-01-01T13:00:00.000000000',
           '1984-01-01T14:00:00.000000000', '1984-01-01T15:00:00.000000000',
           '1984-01-01T16:00:00.000000000', '1984-01-01T17:00:00.000000000',
           '1984-01-01T18:00:00.000000000', '1984-01-01T19:00:00.000000000',
           '1984-01-01T20:00:00.000000000', '1984-01-01T21:00:00.000000000',
           '1984-01-01T22:00:00.000000000', '1984-01-01T23:00:00.000000000'],
          dtype='datetime64[ns]')

• Data variables: (2)
  • u
    (time, level, latitude, longitude)
    float64
    dask.array<chunksize=(4, 5, 405, 900), meta=np.ndarray>

    units :

    m s**-1

    long_name :

    U component of wind

    standard_name :

    eastward_wind

    Array Chunk Bytes 6.87 GiB 55.62 MiB Shape (24, 37, 721, 1440) (4, 5, 405, 900) Dask graph 192 chunks in 4 graph layers Data type float64 numpy.ndarray

  • v
    (time, level, latitude, longitude)
    float64
    dask.array<chunksize=(4, 5, 405, 900), meta=np.ndarray>

    units :

    m s**-1

    long_name :

    V component of wind

    standard_name :

    northward_wind

    Array Chunk Bytes 6.87 GiB 55.62 MiB Shape (24, 37, 721, 1440) (4, 5, 405, 900) Dask graph 192 chunks in 4 graph layers Data type float64 numpy.ndarray

• Indexes: (4)
  • longitude
    PandasIndex

    PandasIndex(Index([ -180.0, -179.75,  -179.5, -179.25,  -179.0, -178.75,  -178.5, -178.25,
            -178.0, -177.75,
           ...
             177.5,  177.75,   178.0,  178.25,   178.5,  178.75,   179.0,  179.25,
             179.5,  179.75],
          dtype='float32', name='longitude', length=1440))

  • latitude
    PandasIndex

    PandasIndex(Index([  90.0,  89.75,   89.5,  89.25,   89.0,  88.75,   88.5,  88.25,   88.0,
            87.75,
           ...
           -87.75,  -88.0, -88.25,  -88.5, -88.75,  -89.0, -89.25,  -89.5, -89.75,
            -90.0],
          dtype='float32', name='latitude', length=721))

  • level
    PandasIndex

    PandasIndex(Index([   1,    2,    3,    5,    7,   10,   20,   30,   50,   70,  100,  125,
            150,  175,  200,  225,  250,  300,  350,  400,  450,  500,  550,  600,
            650,  700,  750,  775,  800,  825,  850,  875,  900,  925,  950,  975,
           1000],
          dtype='int32', name='level'))

  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['1984-01-01 00:00:00', '1984-01-01 01:00:00',
                   '1984-01-01 02:00:00', '1984-01-01 03:00:00',
                   '1984-01-01 04:00:00', '1984-01-01 05:00:00',
                   '1984-01-01 06:00:00', '1984-01-01 07:00:00',
                   '1984-01-01 08:00:00', '1984-01-01 09:00:00',
                   '1984-01-01 10:00:00', '1984-01-01 11:00:00',
                   '1984-01-01 12:00:00', '1984-01-01 13:00:00',
                   '1984-01-01 14:00:00', '1984-01-01 15:00:00',
                   '1984-01-01 16:00:00', '1984-01-01 17:00:00',
                   '1984-01-01 18:00:00', '1984-01-01 19:00:00',
                   '1984-01-01 20:00:00', '1984-01-01 21:00:00',
                   '1984-01-01 22:00:00', '1984-01-01 23:00:00'],
                  dtype='datetime64[ns]', name='time', freq=None))

• Attributes: (3)

  Conventions :

  CF-1.6

  license :

  Licence to use Copernicus Products: https://apps.ecmwf.int/datasets/licences/copernicus/

  summary :

  ERA5 is the fifth generation ECMWF atmospheric reanalysis of the global climate. This file is part of the ERA5 replica hosted at NCI Australia. For more information please see http://dx.doi.org/10.25914/5f48874388857

What this buys you

PyEarthTools has abstracted the relationship between the data loading and the data source, in this case the filesystem. By writing a specific connector to the filesystem, it is possible to thereafter treat all data sources in a compatible and generic fashion, and also modularise the code which deals with filesystem connections (such as handling different filesystem layouts e.g. YYYY, YYMM, YYYMMDD layouts, or others)

It also applies standard coordinate naming, so that data sources can be more easily combined. It is common, for example for one data set to pick ‘lat’ and ‘lon’ instead of ‘latitude’ and ‘longitude’, or ‘t2m’ instead of ‘2t’ and suchlike.

At this point, it would be possible to write a simple loop such as:

for every date in a training period: load it call a training update function

This is a valid use case. The next notebook, “Data Pipelines with PyEarthTools”, will explore how to make further use of these data objects for data processing and options for presentation to the ML training frameworks.