Basic Data Indexing

import warnings
import pyearthtools.data

# Catch name override warning of ERA5 using site installation
with warnings.catch_warnings(action="ignore"):
    import site_archive_nci

Variables

doi = '2022-04-01'
var = 'tmax'

DataIndexes

One of the base classes of pyearthtools.data is the DataIndex, which provides a unified interface to accessing data, and common data retrieval techniques.

As of the writing of this notebook, the following data sources have been implemented:

• ERA5

• ACCESS G/C

• BRAN

• OceanMAPS

• Himiwari 8/9

Each of these sources requires it’s own initalisation parameters such that when attempting to retrieve data, only the timestep is needed.

This Notebook shall show the features available with examples with the ERA5 dataset

era5 = pyearthtools.data.archive.ERA5(var,)

era5

ERA5
    Description                    ECWMF ReAnalysis v5
             range                          '1970-current'
             Documentation                  'https://confluence.ecmwf.int/display/CKB/ERA5%3A+data+documentation'


    Initialisation
             level_value                    None
             product                        'reanalysis'
             variables                      ['tmax']
    Transforms
             StandardCoordinateNames        {'latitude': "['lat', 'Latitude', 'yt_ocean', 'yt']", 'longitude': "['lon', 'Longitude', 'xt_ocean', 'xt']", 'replacement_dictionary': 'None', 'time': "['Time']"}
             Rename                         {'names': {'t2m': "'2t'", 'u10': "'10u'", 'v10': "'10v'", 'siconc': "'ci'"}}

ERA5

Description:

ECWMF ReAnalysis v5

• range
  '1970-current'
• Documentation
  https://confluence.ecmwf.int/display/CKB/ERA5%3A+data+documentation

• Initialisation:
  • level_value
    None
    None
  • product
    'reanalysis'
    'reanalysis'
  • variables
    ['tmax']
    - tmax
• Transforms:
  • StandardCoordinateNames
    {'latitude': ['lat', 'Latitude', 'yt_ocean', 'yt'], 'longitude': ['lon', 'Longitude', 'xt_ocean', 'xt'], 'replacement_dictionary': None, 'time': ['Time']}
    latitude : ['lat', 'Latitude', 'yt_ocean', 'yt']
    longitude : ['lon', 'Longitude', 'xt_ocean', 'xt']
    replacement_dictionary : None
    time : ['Time']
  • Rename
    {'names': {'t2m': '2t', 'u10': '10u', 'v10': '10v', 'siconc': 'ci'}}
    names : {'t2m': '2t', 'u10': '10u', 'v10': '10v', 'siconc': 'ci'}

Data Loading

Using the search function allows retrieval of the filepaths to each requested variable at a given time.

From this, the data can be automatically loaded simply by calling the DataIndex with a date, which provides a couple of useful features, which will be explored in later notebooks.

era5.search(doi)

[{'tmax': PosixPath('/g/data/rt52/era5/single-levels/reanalysis/tmax/2022/tmax_era5_wave_sfc_20220401-20220430.nc')}]

era5_ds = era5(doi)

era5_ds

<xarray.Dataset> Size: 50MB
Dimensions:    (longitude: 720, latitude: 361, time: 24)
Coordinates:
  * longitude  (longitude) float32 3kB -180.0 -179.5 -179.0 ... 179.0 179.5
  * latitude   (latitude) float32 1kB 90.0 89.5 89.0 88.5 ... -89.0 -89.5 -90.0
  * time       (time) datetime64[ns] 192B 2022-04-01 ... 2022-04-01T23:00:00
Data variables:
    tmax       (time, latitude, longitude) float64 50MB dask.array<chunksize=(24, 182, 360), meta=np.ndarray>
Attributes:
    Conventions:  CF-1.6
    history:      2022-08-01 09:25:07 UTC+1000 by era5_replication_tools-1.10...
    license:      Licence to use Copernicus Products: https://apps.ecmwf.int/...
    summary:      ERA5 is the fifth generation ECMWF atmospheric reanalysis o...
    title:        ERA5 single-levels reanalysis period_corresponding_to_maxim...

xarray.Dataset

• Dimensions:
  • longitude: 720
  • latitude: 361
  • time: 24
• Coordinates: (3)
  • longitude
    (longitude)
    float32
    -180.0 -179.5 ... 179.0 179.5

    units :

    degrees_east

    long_name :

    longitude

    array([-180. , -179.5, -179. , ...,  178.5,  179. ,  179.5], dtype=float32)

  • latitude
    (latitude)
    float32
    90.0 89.5 89.0 ... -89.5 -90.0

    units :

    degrees_north

    long_name :

    latitude

    array([ 90. ,  89.5,  89. , ..., -89. , -89.5, -90. ], dtype=float32)

  • time
    (time)
    datetime64[ns]
    2022-04-01 ... 2022-04-01T23:00:00

    long_name :

    time

    array(['2022-04-01T00:00:00.000000000', '2022-04-01T01:00:00.000000000',
           '2022-04-01T02:00:00.000000000', '2022-04-01T03:00:00.000000000',
           '2022-04-01T04:00:00.000000000', '2022-04-01T05:00:00.000000000',
           '2022-04-01T06:00:00.000000000', '2022-04-01T07:00:00.000000000',
           '2022-04-01T08:00:00.000000000', '2022-04-01T09:00:00.000000000',
           '2022-04-01T10:00:00.000000000', '2022-04-01T11:00:00.000000000',
           '2022-04-01T12:00:00.000000000', '2022-04-01T13:00:00.000000000',
           '2022-04-01T14:00:00.000000000', '2022-04-01T15:00:00.000000000',
           '2022-04-01T16:00:00.000000000', '2022-04-01T17:00:00.000000000',
           '2022-04-01T18:00:00.000000000', '2022-04-01T19:00:00.000000000',
           '2022-04-01T20:00:00.000000000', '2022-04-01T21:00:00.000000000',
           '2022-04-01T22:00:00.000000000', '2022-04-01T23:00:00.000000000'],
          dtype='datetime64[ns]')

• Data variables: (1)
  • tmax
    (time, latitude, longitude)
    float64
    dask.array<chunksize=(24, 182, 360), meta=np.ndarray>

    units :

    s

    long_name :

    Period corresponding to maximum individual wave height

    Array Chunk Bytes 47.59 MiB 12.00 MiB Shape (24, 361, 720) (24, 182, 360) Dask graph 4 chunks in 4 graph layers Data type float64 numpy.ndarray

• Indexes: (3)
  • longitude
    PandasIndex

    PandasIndex(Index([-180.0, -179.5, -179.0, -178.5, -178.0, -177.5, -177.0, -176.5, -176.0,
           -175.5,
           ...
            175.0,  175.5,  176.0,  176.5,  177.0,  177.5,  178.0,  178.5,  179.0,
            179.5],
          dtype='float32', name='longitude', length=720))

  • latitude
    PandasIndex

    PandasIndex(Index([ 90.0,  89.5,  89.0,  88.5,  88.0,  87.5,  87.0,  86.5,  86.0,  85.5,
           ...
           -85.5, -86.0, -86.5, -87.0, -87.5, -88.0, -88.5, -89.0, -89.5, -90.0],
          dtype='float32', name='latitude', length=361))

  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2022-04-01 00:00:00', '2022-04-01 01:00:00',
                   '2022-04-01 02:00:00', '2022-04-01 03:00:00',
                   '2022-04-01 04:00:00', '2022-04-01 05:00:00',
                   '2022-04-01 06:00:00', '2022-04-01 07:00:00',
                   '2022-04-01 08:00:00', '2022-04-01 09:00:00',
                   '2022-04-01 10:00:00', '2022-04-01 11:00:00',
                   '2022-04-01 12:00:00', '2022-04-01 13:00:00',
                   '2022-04-01 14:00:00', '2022-04-01 15:00:00',
                   '2022-04-01 16:00:00', '2022-04-01 17:00:00',
                   '2022-04-01 18:00:00', '2022-04-01 19:00:00',
                   '2022-04-01 20:00:00', '2022-04-01 21:00:00',
                   '2022-04-01 22:00:00', '2022-04-01 23:00:00'],
                  dtype='datetime64[ns]', name='time', freq=None))

• Attributes: (5)

  Conventions :

  CF-1.6

  history :

  2022-08-01 09:25:07 UTC+1000 by era5_replication_tools-1.10.2: mv /g/data/rt52/admin/incoming/era5/single-levels/reanalysis/tmax/2022/tmax_era5_wave_sfc_20220401-20220430.tmp /g/data/rt52/era5/single-levels/reanalysis/tmax/2022/tmax_era5_wave_sfc_20220401-20220430.nc 2022-08-01 09:24:59 UTC+1000 by era5_replication_tools-1.10.2: nccopy -ctime/93,latitude/91,longitude/180 -m2G -h2G -k4 -d2 -s /g/data/id28/admin/incoming/era5/single-levels/reanalysis/tmax/2022/tmax_era5_wave_sfc_20220401-20220430.nc /g/data/rt52/admin/incoming/era5/single-levels/reanalysis/tmax/2022/tmax_era5_wave_sfc_20220401-20220430.tmp 2022-08-01 09:24:20 UTC+1000 by era5_replication_tools-1.10.2: curl --connect-timeout 20 --show-error --silent --max-time 36000 -o /g/data/id28/admin/incoming/era5/single-levels/reanalysis/tmax/2022/tmax_era5_wave_sfc_20220401-20220430.nc https://download-0006-clone.copernicus-climate.eu/cache-compute-0006/cache/data6/adaptor.mars.internal-1659309838.3742871-24036-8-d566e723-ff3a-4cbc-ab32-be54c65c429e.nc 2022-07-31 23:24:04 GMT by grib_to_netcdf-2.25.1: /opt/ecmwf/mars-client/bin/grib_to_netcdf.bin -S param -o /cache/data6/adaptor.mars.internal-1659309838.3742871-24036-8-d566e723-ff3a-4cbc-ab32-be54c65c429e.nc /cache/tmp/d566e723-ff3a-4cbc-ab32-be54c65c429e-adaptor.mars.internal-1659309825.5342174-24036-14-tmp.grib

  license :

  Licence to use Copernicus Products: https://apps.ecmwf.int/datasets/licences/copernicus/

  summary :

  ERA5 is the fifth generation ECMWF atmospheric reanalysis of the global climate. This file is part of the ERA5 replica hosted at NCI Australia. For more information please see http://dx.doi.org/10.25914/5f48874388857

  title :

  ERA5 single-levels reanalysis period_corresponding_to_maximum_individual_wave_height 20220401-20220430

era5_ds[var][0].plot(figsize = (16,10))

<matplotlib.collections.QuadMesh at 0x1478e86b9a90>