Blending Data from Multiple Sources

This is an example showing how to process data from three data sources. We will be showing how to work with two similar data sets that can be merged, and how to deal with a source that can’t be easily merged.

Introduction to Each Data Source

The first of the two “mergeable” datasets is the surface global irradiance from the Himawari satellite. This data is masked over the ocean away from the coast, and it also has a slightly different grid spacing to the ACCESS model data.

The second of the two “mergable” datasets is the ACCESS model data, from which we will extract the mean sea level pressure. This is a generally compatible region (lat/lon bounding box), but has a slightly different grid spacing.

The third data set is radar data from one radar in the Australian region. It only has X and Y coordinates, so it’s not straightforward to create a simple merged data sets.

In all these examples, a simple linear interpolation is used for merging datasets. More complex merge operations are possible.

Himawari Data

The tutorial will first introduce a single data set. Datasets can be merged using a pre-defined grid (such as a Mercator projection with a specified grid spacing), or, and this is often simplest, one of the datasets can be used to define the target grid structure without needing a formal specification of it. Let’s examine the Himawari data now.

%%capture

import pyearthtools.data as petdata
import pyearthtools.pipeline as petpipe
import tempfile
import site_archive_nci

# We specify the date, hour, and minute for querying data
doi = '2021-06-09T0600'

himawari = petdata.archive.Himawari('surface_global_irradiance')

sample_sgi = himawari[doi].surface_global_irradiance
sample_sgi.plot()

<matplotlib.collections.QuadMesh at 0x14c80e7d0530>

# Uncomment the line below to see the data structure
# sample_sgi

ACCESS Model Data

Here we see how to access a single sample of ACCESS model data

# We want the ACCESS data trimmed to the same bounding box as the Himawari data.
# The ACCESS data is global, but we only want the Australian region
like_himawari = petdata.transform.region.like(sample_sgi)
access_g = petdata.archive.ACCESS( variables='sfc/mslp', datatype='an', region='g',transforms=like_himawari)

%%capture
sample_mslp = access_g[doi].mslp

sample_mslp.plot()

<matplotlib.collections.QuadMesh at 0x14c7fccb8230>

Radar Data

The radar data sample comes from just a single radar, and its geospatial data is not known. We can retrieve samples by date, but it’s not easy to geolocate them. Let’s examine some data.

radar = petdata.archive.Rainfields3(variables='prcp-crate')

radar_sample = radar[doi]
radar_sample

<xarray.Dataset> Size: 40MB
Dimensions:     (y: 2050, n2: 2, x: 2450)
Coordinates:
  * y           (y) float64 16kB -1.001e+03 -1.003e+03 ... -5.097e+03 -5.099e+03
  * x           (x) float64 20kB -2.299e+03 -2.297e+03 ... 2.597e+03 2.599e+03
Dimensions without coordinates: n2
Data variables:
    valid_time  datetime64[ns] 8B 2021-06-09T06:00:00
    proj        int8 1B 0
    y_bounds    (y, n2) float64 33kB -1e+03 -1.002e+03 ... -5.098e+03 -5.1e+03
    x_bounds    (x, n2) float64 39kB -2.3e+03 -2.298e+03 ... 2.598e+03 2.6e+03
    rain_rate   (y, x) float64 40MB nan nan nan nan nan ... nan nan nan nan nan
Attributes:
    Conventions:         CF-1.7
    contributing_sites:  [ 2  3  4  5  6  8  9 14 15 16 17 19 22 23 24 25 27 ...
    institution:         Commonwealth of Australia, Bureau of Meteorology (AB...
    licence:             http://www.bom.gov.au/other/copyright.shtml
    source:              rainfields 3.2.4 drs-rainfields 2021-02-02
    station_id:          310
    station_name:        Ausm310
    title:               Bias Corrected Rainfall Rate Mosaic

xarray.Dataset

• Dimensions:
  • y: 2050
  • n2: 2
  • x: 2450
• Coordinates: (2)
  • y
    (y)
    float64
    -1.001e+03 ... -5.099e+03

    standard_name :

    projection_y_coordinate

    units :

    km

    bounds :

    y_bounds

    array([-1001., -1003., -1005., ..., -5095., -5097., -5099.], shape=(2050,))

  • x
    (x)
    float64
    -2.299e+03 -2.297e+03 ... 2.599e+03

    standard_name :

    projection_x_coordinate

    units :

    km

    bounds :

    x_bounds

    array([-2299., -2297., -2295., ...,  2595.,  2597.,  2599.], shape=(2450,))

• Data variables: (5)
  • valid_time
    ()
    datetime64[ns]
    2021-06-09T06:00:00

    long_name :

    Valid time

    standard_name :

    time

    array('2021-06-09T06:00:00.000000000', dtype='datetime64[ns]')

  • proj
    ()
    int8
    0

    grid_mapping_name :

    albers_conical_equal_area

    standard_parallel :

    [-18. -36.]

    longitude_of_central_meridian :

    132.0

    latitude_of_projection_origin :

    0.0

    false_easting :

    0.0

    false_northing :

    0.0

    towgs84 :

    [0. 0. 0. 0. 0. 0. 0.]

    array(0, dtype=int8)

  • y_bounds
    (y, n2)
    float64
    -1e+03 -1.002e+03 ... -5.1e+03

    array([[-1000., -1002.],
           [-1002., -1004.],
           [-1004., -1006.],
           ...,
           [-5094., -5096.],
           [-5096., -5098.],
           [-5098., -5100.]], shape=(2050, 2))

  • x_bounds
    (x, n2)
    float64
    -2.3e+03 -2.298e+03 ... 2.6e+03

    array([[-2300., -2298.],
           [-2298., -2296.],
           [-2296., -2294.],
           ...,
           [ 2594.,  2596.],
           [ 2596.,  2598.],
           [ 2598.,  2600.]], shape=(2450, 2))

  • rain_rate
    (y, x)
    float64
    nan nan nan nan ... nan nan nan nan

    grid_mapping :

    proj

    long_name :

    Bias corrected rainfall rate

    standard_name :

    rainfall_rate

    units :

    mm hr-1

    array([[nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan],
           ...,
           [nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan]], shape=(2050, 2450))

• Indexes: (2)
  • y
    PandasIndex

    PandasIndex(Index([-1001.0, -1003.0, -1005.0, -1007.0, -1009.0, -1011.0, -1013.0, -1015.0,
           -1017.0, -1019.0,
           ...
           -5081.0, -5083.0, -5085.0, -5087.0, -5089.0, -5091.0, -5093.0, -5095.0,
           -5097.0, -5099.0],
          dtype='float64', name='y', length=2050))

  • x
    PandasIndex

    PandasIndex(Index([-2299.0, -2297.0, -2295.0, -2293.0, -2291.0, -2289.0, -2287.0, -2285.0,
           -2283.0, -2281.0,
           ...
            2581.0,  2583.0,  2585.0,  2587.0,  2589.0,  2591.0,  2593.0,  2595.0,
            2597.0,  2599.0],
          dtype='float64', name='x', length=2450))

• Attributes: (8)

  Conventions :

  CF-1.7

  contributing_sites :

  [ 2 3 4 5 6 8 9 14 15 16 17 19 22 23 24 25 27 28 29 31 32 33 36 38 40 41 42 44 46 49 50 52 53 54 55 58 63 64 66 67 68 69 70 71 72 73 75 76 77 79 93 95 97]

  institution :

  Commonwealth of Australia, Bureau of Meteorology (ABN 92 637 533 532)

  licence :

  http://www.bom.gov.au/other/copyright.shtml

  source :

  rainfields 3.2.4 drs-rainfields 2021-02-02

  station_id :

  310

  station_name :

  Ausm310

  title :

  Bias Corrected Rainfall Rate Mosaic

radar_sample.rain_rate.plot()

<matplotlib.collections.QuadMesh at 0x14c7fccb83b0>

Putting them together, the simple way, without “merging” them

Despite their differences, these data sources can easily be co-indexed. If all you want to do is retrieve the relevant samples by date, than can be done quickly and easily using a pipeline.

pipe = petpipe.Pipeline(
    (himawari, access_g, radar),
)
pipe

Pipeline
    Description                    `pyearthtools.pipeline` Data Pipeline


    Initialisation
             exceptions_to_ignore           None
             iterator                       None
             sampler                        None
    Steps
             Himawari                       {'Himawari': {'data_interval': "(10, 'm')", 'file_regex': "'*{date_info}*{time_info}*.nc'", 'variables': "['surface_global_irradiance']"}}
             ACCESS.ACCESS_Analysis         {'ACCESS_Analysis': {'data_interval': "(6, 'h')", 'datatype': "'an'", 'level_value': 'None', 'region': "'g'", 'variables': "['sfc/mslp']"}}
             _.Rainfields3                  {'Rainfields3': {'variables': "'prcp-crate'"}}

Pipeline

Description:

`pyearthtools.pipeline` Data Pipeline

• Initialisation:
  • exceptions_to_ignore
    None
    None
  • iterator
    None
    None
  • sampler
    None
    None
• Steps:
  • Himawari
    {'Himawari': {'data_interval': (10, 'm'), 'file_regex': '*{date_info}*{time_info}*.nc', 'variables': ['surface_global_irradiance']}}
    Himawari : {'data_interval': (10, 'm'), 'file_regex': '*{date_info}*{time_info}*.nc', 'variables': ['surface_global_irradiance']}
  • ACCESS.ACCESS_Analysis
    {'ACCESS_Analysis': {'data_interval': (6, 'h'), 'datatype': 'an', 'level_value': None, 'region': 'g', 'variables': ['sfc/mslp']}}
    ACCESS_Analysis : {'data_interval': (6, 'h'), 'datatype': 'an', 'level_value': None, 'region': 'g', 'variables': ['sfc/mslp']}
  • _.Rainfields3
    {'Rainfields3': {'variables': 'prcp-crate'}}
    Rainfields3 : {'variables': 'prcp-crate'}

Graph

Fontconfig error: Cannot load default config file: No such file: (null)

%%capture
all_sources = pipe[doi]

# Indexing into the pipeline by the data of interest returns a tuple, with
# the retrieved item from each data source according to that data of interest
print(type(all_sources))
print(len(all_sources))

<class 'tuple'>
3

all_sources[2]  #

<xarray.Dataset> Size: 40MB
Dimensions:     (y: 2050, n2: 2, x: 2450)
Coordinates:
  * y           (y) float64 16kB -1.001e+03 -1.003e+03 ... -5.097e+03 -5.099e+03
  * x           (x) float64 20kB -2.299e+03 -2.297e+03 ... 2.597e+03 2.599e+03
Dimensions without coordinates: n2
Data variables:
    valid_time  datetime64[ns] 8B 2021-06-09T06:00:00
    proj        int8 1B 0
    y_bounds    (y, n2) float64 33kB -1e+03 -1.002e+03 ... -5.098e+03 -5.1e+03
    x_bounds    (x, n2) float64 39kB -2.3e+03 -2.298e+03 ... 2.598e+03 2.6e+03
    rain_rate   (y, x) float64 40MB nan nan nan nan nan ... nan nan nan nan nan
Attributes:
    Conventions:         CF-1.7
    contributing_sites:  [ 2  3  4  5  6  8  9 14 15 16 17 19 22 23 24 25 27 ...
    institution:         Commonwealth of Australia, Bureau of Meteorology (AB...
    licence:             http://www.bom.gov.au/other/copyright.shtml
    source:              rainfields 3.2.4 drs-rainfields 2021-02-02
    station_id:          310
    station_name:        Ausm310
    title:               Bias Corrected Rainfall Rate Mosaic

xarray.Dataset

• Dimensions:
  • y: 2050
  • n2: 2
  • x: 2450
• Coordinates: (2)
  • y
    (y)
    float64
    -1.001e+03 ... -5.099e+03

    standard_name :

    projection_y_coordinate

    units :

    km

    bounds :

    y_bounds

    array([-1001., -1003., -1005., ..., -5095., -5097., -5099.], shape=(2050,))

  • x
    (x)
    float64
    -2.299e+03 -2.297e+03 ... 2.599e+03

    standard_name :

    projection_x_coordinate

    units :

    km

    bounds :

    x_bounds

    array([-2299., -2297., -2295., ...,  2595.,  2597.,  2599.], shape=(2450,))

• Data variables: (5)
  • valid_time
    ()
    datetime64[ns]
    2021-06-09T06:00:00

    long_name :

    Valid time

    standard_name :

    time

    array('2021-06-09T06:00:00.000000000', dtype='datetime64[ns]')

  • proj
    ()
    int8
    0

    grid_mapping_name :

    albers_conical_equal_area

    standard_parallel :

    [-18. -36.]

    longitude_of_central_meridian :

    132.0

    latitude_of_projection_origin :

    0.0

    false_easting :

    0.0

    false_northing :

    0.0

    towgs84 :

    [0. 0. 0. 0. 0. 0. 0.]

    array(0, dtype=int8)

  • y_bounds
    (y, n2)
    float64
    -1e+03 -1.002e+03 ... -5.1e+03

    array([[-1000., -1002.],
           [-1002., -1004.],
           [-1004., -1006.],
           ...,
           [-5094., -5096.],
           [-5096., -5098.],
           [-5098., -5100.]], shape=(2050, 2))

  • x_bounds
    (x, n2)
    float64
    -2.3e+03 -2.298e+03 ... 2.6e+03

    array([[-2300., -2298.],
           [-2298., -2296.],
           [-2296., -2294.],
           ...,
           [ 2594.,  2596.],
           [ 2596.,  2598.],
           [ 2598.,  2600.]], shape=(2450, 2))

  • rain_rate
    (y, x)
    float64
    nan nan nan nan ... nan nan nan nan

    grid_mapping :

    proj

    long_name :

    Bias corrected rainfall rate

    standard_name :

    rainfall_rate

    units :

    mm hr-1

    array([[nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan],
           ...,
           [nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan]], shape=(2050, 2450))

• Indexes: (2)
  • y
    PandasIndex

    PandasIndex(Index([-1001.0, -1003.0, -1005.0, -1007.0, -1009.0, -1011.0, -1013.0, -1015.0,
           -1017.0, -1019.0,
           ...
           -5081.0, -5083.0, -5085.0, -5087.0, -5089.0, -5091.0, -5093.0, -5095.0,
           -5097.0, -5099.0],
          dtype='float64', name='y', length=2050))

  • x
    PandasIndex

    PandasIndex(Index([-2299.0, -2297.0, -2295.0, -2293.0, -2291.0, -2289.0, -2287.0, -2285.0,
           -2283.0, -2281.0,
           ...
            2581.0,  2583.0,  2585.0,  2587.0,  2589.0,  2591.0,  2593.0,  2595.0,
            2597.0,  2599.0],
          dtype='float64', name='x', length=2450))

• Attributes: (8)

  Conventions :

  CF-1.7

  contributing_sites :

  [ 2 3 4 5 6 8 9 14 15 16 17 19 22 23 24 25 27 28 29 31 32 33 36 38 40 41 42 44 46 49 50 52 53 54 55 58 63 64 66 67 68 69 70 71 72 73 75 76 77 79 93 95 97]

  institution :

  Commonwealth of Australia, Bureau of Meteorology (ABN 92 637 533 532)

  licence :

  http://www.bom.gov.au/other/copyright.shtml

  source :

  rainfields 3.2.4 drs-rainfields 2021-02-02

  station_id :

  310

  station_name :

  Ausm310

  title :

  Bias Corrected Rainfall Rate Mosaic

The coordinates of the radar data are specified as “X” and “Y”. Because the radar data has X and Y, we have no way to correlate it with the latitude and longitude dimensions of the Himawari and ACCESS data.

Merging datasets onto a common grid

(Please note! There is no reason to assume that you want the data on a common grid for machine learning. Many ML algorithms are happy with (and may prefer) unhomogenised data!)

It can be useful to try to merge data sources onto a common grid, so that they all have data at the exact same spots. For example, consider the following data sets:

x1=[1,  3,   6,   9, 10],   y1=[1, 9, 36, 81, 100]
x2=[1, 2, 4, 6, 8,   10],   y2=[1, 4, 36, 64, 100]

They both fulfil y=x^2, but with slightly different coordinates, but both ranging from 1 to 10 inclusive. The semantics of the ordinal indexing (place value) do not match the semantics of the cadinal indexing (the actual value of the coordinate). It can be really useful to line up two datasets with the same x values. When doing so, it’s necessary to guess the corresponding y value in some fashion. In leiu of a clear formula, a linear interpolation of the y values is used.

Either both y1 and y2 could each be interpolated to provide values for, say x=[1,2,3,4,5,6,7,8,9,10], or y1 could be interpolated to give values for the x1 coordinates.

In earth system modelling terms, the most common interpolation is between coordinate reference systems, possibly including different spatial resolutions. “x” in that case is the latitude and longitude values, and “y” would be the data values.

PyEarthTools supports a number of transformations. This notebook demonstrates the “InterpLike” transformation which will map one dataset onto the coordinates of another using linear interpolation.

The first step will be looking at merging Himawari with ACCESS, and then subsequently we will look at how to deal with the fact the radar data cannot be simply merged.

pipe = petpipe.Pipeline(
    (himawari, access_g),
    petpipe.operations.xarray.join.InterpLike(sample_mslp[0])
)
pipe

Pipeline
    Description                    `pyearthtools.pipeline` Data Pipeline


    Initialisation
             exceptions_to_ignore           None
             iterator                       None
             sampler                        None
    Steps
             Himawari                       {'Himawari': {'data_interval': "(10, 'm')", 'file_regex': "'*{date_info}*{time_info}*.nc'", 'variables': "['surface_global_irradiance']"}}
             ACCESS.ACCESS_Analysis         {'ACCESS_Analysis': {'data_interval': "(6, 'h')", 'datatype': "'an'", 'level_value': 'None', 'region': "'g'", 'variables': "['sfc/mslp']"}}
             join.InterpLike                {'InterpLike': {'merge_kwargs': 'None', 'method': "'nearest'", 'reference_dataset': "<xarray.DataArray 'mslp' (latitude: 295, longitude: 252)> Size: 297kB\ndask.array<getitem, shape=(295, 252), dtype=float32, chunksize=(295, 252), chunktype=numpy.ndarray>\nCoordinates:\n    time       datetime64[ns] 8B 2021-06-09T06:00:00\n  * latitude   (latitude) float32 1kB -10.02 -10.14 -10.25 ... -44.36 -44.47\n  * longitude  (longitude) float32 1kB 112.1 112.2 112.4 ... 155.8 156.0 156.2\nAttributes:\n    grid_type:   spatial\n    level_type:  single\n    units:       Pa\n    long_name:   mean sea level pressure\n    stash_code:  16222\n    accum_type:  instantaneous", 'reference_index': 'None'}}

Pipeline

Description:

`pyearthtools.pipeline` Data Pipeline

• Initialisation:
  • exceptions_to_ignore
    None
    None
  • iterator
    None
    None
  • sampler
    None
    None
• Steps:
  • Himawari
    {'Himawari': {'data_interval': (10, 'm'), 'file_regex': '*{date_info}*{time_info}*.nc', 'variables': ['surface_global_irradiance']}}
    Himawari : {'data_interval': (10, 'm'), 'file_regex': '*{date_info}*{time_info}*.nc', 'variables': ['surface_global_irradiance']}
  • ACCESS.ACCESS_Analysis
    {'ACCESS_Analysis': {'data_interval': (6, 'h'), 'datatype': 'an', 'level_value': None, 'region': 'g', 'variables': ['sfc/mslp']}}
    ACCESS_Analysis : {'data_interval': (6, 'h'), 'datatype': 'an', 'level_value': None, 'region': 'g', 'variables': ['sfc/mslp']}
  • join.InterpLike
    {'InterpLike': {'merge_kwargs': None, 'method': 'nearest', 'reference_dataset': <xarray.DataArray 'mslp' (latitude: 295, longitude: 252)> Size: 297kB dask.array<getitem, shape=(295, 252), dtype=float32, chunksize=(295, 252), chunktype=numpy.ndarray> Coordinates: time datetime64[ns] 8B 2021-06-09T06:00:00 * latitude (latitude) float32 1kB -10.02 -10.14 -10.25 ... -44.36 -44.47 * longitude (longitude) float32 1kB 112.1 112.2 112.4 ... 155.8 156.0 156.2 Attributes: grid_type: spatial level_type: single units: Pa long_name: mean sea level pressure stash_code: 16222 accum_type: instantaneous, 'reference_index': None}}
    InterpLike : {'merge_kwargs': None, 'method': 'nearest', 'reference_dataset': Size: 297kB dask.array Coordinates: time datetime64[ns] 8B 2021-06-09T06:00:00 * latitude (latitude) float32 1kB -10.02 -10.14 -10.25 ... -44.36 -44.47 * longitude (longitude) float32 1kB 112.1 112.2 112.4 ... 155.8 156.0 156.2 Attributes: grid_type: spatial level_type: single units: Pa long_name: mean sea level pressure stash_code: 16222 accum_type: instantaneous, 'reference_index': None}

Graph

Fontconfig error: Cannot load default config file: No such file: (null)

%%capture
merged_sample = pipe[doi]

merged_sample

<xarray.Dataset> Size: 894kB
Dimensions:                    (time: 1, latitude: 295, longitude: 252)
Coordinates:
  * time                       (time) datetime64[ns] 8B 2021-06-09T06:00:00
  * latitude                   (latitude) float32 1kB -10.02 -10.14 ... -44.47
  * longitude                  (longitude) float32 1kB 112.1 112.2 ... 156.2
Data variables:
    surface_global_irradiance  (time, latitude, longitude) float64 595kB dask.array<chunksize=(1, 295, 252), meta=np.ndarray>
    mslp                       (time, latitude, longitude) float32 297kB dask.array<chunksize=(1, 295, 252), meta=np.ndarray>
Attributes: (12/58)
    Conventions:                      CF-1.7
    Metadata_Conventions:             Unidata Dataset Discovery v1.0
    acknowledgment:                   The following acknowledgement is requir...
    cdm_data_type:                    grid
    comment:                          Solar radiation data derived from satel...
    contributor_name:                 Mines ParisTech; Commonwealth of Austra...
    ...                               ...
    quality:                          0
    quality_meaning:                  0: no_known_issues 1: known_issue
    project:                          Gridded Solar Observations
    history:                          Mon Mar  4 01:55:13 2024: ncatted -a re...
    references:                       Poulsen C., Majewski L. J. (2022) Gridd...
    NCO:                              netCDF Operators version 4.7.7 (Homepag...

xarray.Dataset

• Dimensions:
  • time: 1
  • latitude: 295
  • longitude: 252
• Coordinates: (3)
  • time
    (time)
    datetime64[ns]
    2021-06-09T06:00:00

    long_name :

    time

    comment :

    Start time of the corresponding satellite observation period

    array(['2021-06-09T06:00:00.000000000'], dtype='datetime64[ns]')

  • latitude
    (latitude)
    float32
    -10.02 -10.14 ... -44.36 -44.47

    long_name :

    latitudes

    type :

    uniform

    units :

    degrees_north

    valid_min :

    -90.0

    valid_max :

    90.0

    axis :

    Y

    array([-10.019531, -10.136719, -10.253906, ..., -44.23828 , -44.35547 ,
           -44.472656], shape=(295,), dtype=float32)

  • longitude
    (longitude)
    float32
    112.1 112.2 112.4 ... 156.0 156.2

    long_name :

    longitudes

    type :

    uniform

    units :

    degrees_east

    valid_min :

    -180.0

    valid_max :

    360.0

    axis :

    X

    array([112.06055, 112.23633, 112.41211, ..., 155.83008, 156.00586, 156.18164],
          shape=(252,), dtype=float32)

• Data variables: (2)
  • surface_global_irradiance
    (time, latitude, longitude)
    float64
    dask.array<chunksize=(1, 295, 252), meta=np.ndarray>

    standard_name :

    surface_downwelling_shortwave_flux_in_air

    long_name :

    Total horizontal solar irradiance at surface with bias correction applied

    comment :

    Bias correction has been applied. Surface downwelling shortwave is the sum of direct and diffuse solar radiation incident on the surface, and is sometimes called "global radiation"

    units :

    W m-2

    valid_min :

    0

    valid_max :

    140000

    grid_mapping :

    crs

    Array Chunk Bytes 580.78 kiB 580.78 kiB Shape (1, 295, 252) (1, 295, 252) Dask graph 1 chunks in 23 graph layers Data type float64 numpy.ndarray

  • mslp
    (time, latitude, longitude)
    float32
    dask.array<chunksize=(1, 295, 252), meta=np.ndarray>

    grid_type :

    spatial

    level_type :

    single

    units :

    Pa

    long_name :

    mean sea level pressure

    stash_code :

    16222

    accum_type :

    instantaneous

    Array Chunk Bytes 290.39 kiB 290.39 kiB Shape (1, 295, 252) (1, 295, 252) Dask graph 1 chunks in 21 graph layers Data type float32 numpy.ndarray

• Indexes: (3)
  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2021-06-09 06:00:00'], dtype='datetime64[ns]', name='time', freq=None))

  • latitude
    PandasIndex

    PandasIndex(Index([-10.01953125, -10.13671875, -10.25390625, -10.37109375, -10.48828125,
           -10.60546875, -10.72265625, -10.83984375, -10.95703125, -11.07421875,
           ...
           -43.41796875, -43.53515625, -43.65234375, -43.76953125, -43.88671875,
           -44.00390625, -44.12109375, -44.23828125, -44.35546875, -44.47265625],
          dtype='float32', name='latitude', length=295))

  • longitude
    PandasIndex

    PandasIndex(Index([112.060546875, 112.236328125, 112.412109375, 112.587890625,
           112.763671875, 112.939453125, 113.115234375, 113.291015625,
           113.466796875, 113.642578125,
           ...
           154.599609375, 154.775390625, 154.951171875, 155.126953125,
           155.302734375, 155.478515625, 155.654296875, 155.830078125,
           156.005859375, 156.181640625],
          dtype='float32', name='longitude', length=252))

• Attributes: (58)

  Conventions :

  CF-1.7

  Metadata_Conventions :

  Unidata Dataset Discovery v1.0

  acknowledgment :

  The following acknowledgement is required "Solar radiation data derived from satellite imagery processed by the Bureau of Meteorology from the geostationary satellite Himawari-8 operated by the Japan Meteorological Agency using Heliosat-4 software provided by Mines ParisTech. Contains modified Copernicus Atmosphere Monitoring Service Information 2021."

  cdm_data_type :

  grid

  comment :

  Solar radiation data derived from satellite imagery processed by the Bureau of Meteorology from the geostationary satellite Himawari-8 operated by the Japan Meteorological Agency using Heliosat-4 software provided by Mines ParisTech

  contributor_name :

  Mines ParisTech; Commonwealth of Australia, Bureau of Meteorology, Science and Innovation, Research, Weather and Environmental Prediction, Satellite Applications; Commonwealth of Australia, Bureau of Meteorology, Data and Digital, Digital Services, Business Solutions Services

  contributor_role :

  Development; Development; Operations

  creator_email :

  sat_support@bom.gov.au

  creator_name :

  Commonwealth of Australia, Bureau of Meteorology, Data and Digital, Digital Services, Business Solutions Services

  creator_url :

  http://www.bom.gov.au/satellite/

  disclaimer :

  You are solely responsible for your use of this information and you accept all risks and consequences that might arise from your use of this information. The Bureau of Meteorology ('Bureau') is not in any way liable for losses, damages, costs, expenses and liability of any kind that you or any other person may suffer or incur directly or indirectly from you using this information. This information: is subject to the uncertainties of scientific and technical research; may not be accurate, current or complete; is subject to change without notice. Please refer to http://www.bom.gov.au/other/disclaimer.shtml for disclaimer details. Neither the European Commission nor ECMWF is responsible for any use that may be made of the information it contains.

  geospatial_lat_units :

  degree_north

  geospatial_lon_units :

  degree_east

  geospatial_vertical_max :

  0

  geospatial_vertical_min :

  0

  geospatial_vertical_positive :

  up

  geospatial_vertical_resolution :

  0

  geospatial_vertical_units :

  km

  id :

  ABOM_HS4_SSI_P1S_GRID_v1.0

  institution :

  ABOM

  instrument :

  AHI

  keywords :

  EARTH SCIENCE

  keywords_vocabulary :

  NASA Global Change Master Directory (GCMD) Science Keywords

  license :

  Unless provided under a different license by the Bureau of Meteorology, this information in this data file is provided under the CC BY-NC 4.0 International Licence. The CC BY-NC 4.0 International (Creative Commons Attribution Non-Commercial CC BY-NC v.4.0 international) license summary may be found here: [ https://creativecommons.org/licenses/by-nc/4.0/]. The full license text may be found here: [ https://creativecommons.org/licenses/by-nc/4.0/legalcode ]. For commercial use, please contact Bureau of Meteorology Climate Data Services at climatedata@bom.gov.au

  naming_authority :

  au.gov.bom

  platform :

  Himawari-8

  processing_level :

  L3

  processing_version :

  heliosat-4

  product_specification :

  ABOM_HS4_SSI_P1S_GRID_v1.0

  product_suite :

  ABOM_HS4_SSI

  product_version :

  v1.1

  publisher_email :

  sat_support@bom.gov.au

  publisher_name :

  Commonwealth of Australia, Bureau of Meteorology, Data and Digital, Digital Services, Business Solutions Services

  publisher_url :

  http://www.bom.gov.au/satellite

  standard_name_vocabulary :

  CF-1.7

  summary :

  Solar radiation data derived from satellite imagery processed by the Bureau of Meteorology from the geostationary satellite Himawari-8 operated by the Japan Meteorological Agency using Heliosat-4 software provided by Mines ParisTech. Contains modified Copernicus Atmosphere Monitoring Service Information 2021.

  time_coverage_duration :

  P1S

  time_coverage_resolution :

  P1S

  title :

  Heliosat-4 Solar Radiation - Instantaneous

  geospatial_lat_max :

  -9.99

  geospatial_lat_min :

  -44.51

  geospatial_lon_max :

  156.26999999999998

  geospatial_lon_min :

  111.99

  geospatial_lat_resolution :

  0.02

  geospatial_lon_resolution :

  0.02

  time_coverage_start :

  20210609T060556Z

  time_coverage_end :

  20210609T060835Z

  source :

  IDY25006.APS3.all-flds.slv.2021060900.006.surface.grb2; IDY25006.APS3.all-flds.slv.2021060900.009.surface.grb2; gfs_4_20210609_0000_006.grb2; gfs_4_20210609_0600_006.grb2; CAMS-CLIM; S_NWC_HS4C_HIMA08_HIMA-N-NR_20210609T060000Z.h5; S_NWC_HS4S_HIMA08_HIMA-N-NR_20210609T060000Z.h5; model_BC_ghi_clear_ACCESS_CAMS-CLIM_all01_all12_60_30_1_10; model_BC_ghi_cloud_ACCESS_CAMS-CLIM_all01_all12_60_30_1_10; model_BC_dni_cloud_ACCESS_CAMS-CLIM_all01_all12_60_30_1_10; model_BC_dni_cloud_ACCESS_CAMS-CLIM_all01_all12_60_30_1_10

  uuid :

  538d58e4-9ef3-4901-8f0a-0154a3dade3b

  date_created :

  20211219T225111Z

  bias_correction_applied :

  1

  bias_correction_applied_meaning :

  0: not applied; 1:applied

  quality :

  0

  quality_meaning :

  0: no_known_issues 1: known_issue

  project :

  Gridded Solar Observations

  history :

  Mon Mar 4 01:55:13 2024: ncatted -a references,global,o,c,Poulsen C., Majewski L. J. (2022) Gridded satellite solar observations Project: Implementation of Heliosat-4 and application of bias correction, Australian Bureau of Meteorology, Melbourne, Victoria. Qu A., Oumbe A., Blanc P., Espinar B., Gesell G., et al. (2017) Fast radiative transfer parameterisation for assessing the surface solar irradiance: The Heliosat-4 method. Meteorologische Zeitschrift, v26, p33-57. 10.1127/metz/2016/0781 . hal-01512589 /g/data/rv74/satellite-products/arc/der/himawari-ahi/solar/p1s/v1.1/2021/06/09/IDE00326.202106090600.nc Mon Mar 4 00:54:18 2024: ncatted -a references,global,a,c,Poulsen C., Majewski L. J. (2022) Gridded satellite solar observations Project: Implementation of Heliosat-4 and application of bias correction, Australian Bureau of Meteorology, Melbourne, Victoria. Qu A., Oumbe A., Blanc P., Espinar B., Gesell G., et al. (2017) Fast radiative transfer parameterisation for assessing the surface solar irradiance: The Heliosat-4 method. Meteorologische Zeitschrift, v26, p33-57. 10.1127/metz/2016/0781 . hal-01512589 /g/data/rv74/satellite-products/arc/der/himawari-ahi/solar/p1s/v1.1/2021/06/09/IDE00326.202106090600.nc heliosat-4; post_process_p1s.py

  references :

  Poulsen C., Majewski L. J. (2022) Gridded satellite solar observations Project: Implementation of Heliosat-4 and application of bias correction, Australian Bureau of Meteorology, Melbourne, Victoria. Qu A., Oumbe A., Blanc P., Espinar B., Gesell G., et al. (2017) Fast radiative transfer parameterisation for assessing the surface solar irradiance: The Heliosat-4 method. Meteorologische Zeitschrift, v26, p33-57. 10.1127/metz/2016/0781 . hal-01512589

  NCO :

  netCDF Operators version 4.7.7 (Homepage = http://nco.sf.net, Code = http://github.com/nco/nco)

# If we want to, we can now operate on the samples with a common grid
(merged_sample.surface_global_irradiance / merged_sample.mslp).plot()

<matplotlib.collections.QuadMesh at 0x14c7f4ded4c0>

Pipelines with Unmergeable Data

In this example, we will merge the Himawari and ACCESS data into a dataset which has a common coordinate reference system, but leave the radar data to be indexed separately. Each request to the pipeline according to the date of interest will return the merged data, together with the relevant unmerged data, as a tuple. The user is then able to interpret each data source as they wish.

pipe = petpipe.Pipeline(
    (himawari, access_g),
    (petpipe.operations.xarray.join.InterpLike(sample_mslp[0]), radar)
)
pipe

Pipeline
    Description                    `pyearthtools.pipeline` Data Pipeline


    Initialisation
             exceptions_to_ignore           None
             iterator                       None
             sampler                        None
    Steps
             Himawari                       {'Himawari': {'data_interval': "(10, 'm')", 'file_regex': "'*{date_info}*{time_info}*.nc'", 'variables': "['surface_global_irradiance']"}}
             ACCESS.ACCESS_Analysis         {'ACCESS_Analysis': {'data_interval': "(6, 'h')", 'datatype': "'an'", 'level_value': 'None', 'region': "'g'", 'variables': "['sfc/mslp']"}}
             join.InterpLike                {'InterpLike': {'merge_kwargs': 'None', 'method': "'nearest'", 'reference_dataset': "<xarray.DataArray 'mslp' (latitude: 295, longitude: 252)> Size: 297kB\ndask.array<getitem, shape=(295, 252), dtype=float32, chunksize=(295, 252), chunktype=numpy.ndarray>\nCoordinates:\n    time       datetime64[ns] 8B 2021-06-09T06:00:00\n  * latitude   (latitude) float32 1kB -10.02 -10.14 -10.25 ... -44.36 -44.47\n  * longitude  (longitude) float32 1kB 112.1 112.2 112.4 ... 155.8 156.0 156.2\nAttributes:\n    grid_type:   spatial\n    level_type:  single\n    units:       Pa\n    long_name:   mean sea level pressure\n    stash_code:  16222\n    accum_type:  instantaneous", 'reference_index': 'None'}}
             _.Rainfields3                  {'Rainfields3': {'variables': "'prcp-crate'"}}

Pipeline

Description:

`pyearthtools.pipeline` Data Pipeline

• Initialisation:
  • exceptions_to_ignore
    None
    None
  • iterator
    None
    None
  • sampler
    None
    None
• Steps:
  • Himawari
    {'Himawari': {'data_interval': (10, 'm'), 'file_regex': '*{date_info}*{time_info}*.nc', 'variables': ['surface_global_irradiance']}}
    Himawari : {'data_interval': (10, 'm'), 'file_regex': '*{date_info}*{time_info}*.nc', 'variables': ['surface_global_irradiance']}
  • ACCESS.ACCESS_Analysis
    {'ACCESS_Analysis': {'data_interval': (6, 'h'), 'datatype': 'an', 'level_value': None, 'region': 'g', 'variables': ['sfc/mslp']}}
    ACCESS_Analysis : {'data_interval': (6, 'h'), 'datatype': 'an', 'level_value': None, 'region': 'g', 'variables': ['sfc/mslp']}
  • join.InterpLike
    {'InterpLike': {'merge_kwargs': None, 'method': 'nearest', 'reference_dataset': <xarray.DataArray 'mslp' (latitude: 295, longitude: 252)> Size: 297kB dask.array<getitem, shape=(295, 252), dtype=float32, chunksize=(295, 252), chunktype=numpy.ndarray> Coordinates: time datetime64[ns] 8B 2021-06-09T06:00:00 * latitude (latitude) float32 1kB -10.02 -10.14 -10.25 ... -44.36 -44.47 * longitude (longitude) float32 1kB 112.1 112.2 112.4 ... 155.8 156.0 156.2 Attributes: grid_type: spatial level_type: single units: Pa long_name: mean sea level pressure stash_code: 16222 accum_type: instantaneous, 'reference_index': None}}
    InterpLike : {'merge_kwargs': None, 'method': 'nearest', 'reference_dataset': Size: 297kB dask.array Coordinates: time datetime64[ns] 8B 2021-06-09T06:00:00 * latitude (latitude) float32 1kB -10.02 -10.14 -10.25 ... -44.36 -44.47 * longitude (longitude) float32 1kB 112.1 112.2 112.4 ... 155.8 156.0 156.2 Attributes: grid_type: spatial level_type: single units: Pa long_name: mean sea level pressure stash_code: 16222 accum_type: instantaneous, 'reference_index': None}
  • _.Rainfields3
    {'Rainfields3': {'variables': 'prcp-crate'}}
    Rainfields3 : {'variables': 'prcp-crate'}

Graph

Fontconfig error: Cannot load default config file: No such file: (null)

%%capture
sample = pipe[doi]

sample[0]

<xarray.Dataset> Size: 894kB
Dimensions:                    (time: 1, latitude: 295, longitude: 252)
Coordinates:
  * time                       (time) datetime64[ns] 8B 2021-06-09T06:00:00
  * latitude                   (latitude) float32 1kB -10.02 -10.14 ... -44.47
  * longitude                  (longitude) float32 1kB 112.1 112.2 ... 156.2
Data variables:
    surface_global_irradiance  (time, latitude, longitude) float64 595kB dask.array<chunksize=(1, 295, 252), meta=np.ndarray>
    mslp                       (time, latitude, longitude) float32 297kB dask.array<chunksize=(1, 295, 252), meta=np.ndarray>
Attributes: (12/58)
    Conventions:                      CF-1.7
    Metadata_Conventions:             Unidata Dataset Discovery v1.0
    acknowledgment:                   The following acknowledgement is requir...
    cdm_data_type:                    grid
    comment:                          Solar radiation data derived from satel...
    contributor_name:                 Mines ParisTech; Commonwealth of Austra...
    ...                               ...
    quality:                          0
    quality_meaning:                  0: no_known_issues 1: known_issue
    project:                          Gridded Solar Observations
    history:                          Mon Mar  4 01:55:13 2024: ncatted -a re...
    references:                       Poulsen C., Majewski L. J. (2022) Gridd...
    NCO:                              netCDF Operators version 4.7.7 (Homepag...

xarray.Dataset

• Dimensions:
  • time: 1
  • latitude: 295
  • longitude: 252
• Coordinates: (3)
  • time
    (time)
    datetime64[ns]
    2021-06-09T06:00:00

    long_name :

    time

    comment :

    Start time of the corresponding satellite observation period

    array(['2021-06-09T06:00:00.000000000'], dtype='datetime64[ns]')

  • latitude
    (latitude)
    float32
    -10.02 -10.14 ... -44.36 -44.47

    long_name :

    latitudes

    type :

    uniform

    units :

    degrees_north

    valid_min :

    -90.0

    valid_max :

    90.0

    axis :

    Y

    array([-10.019531, -10.136719, -10.253906, ..., -44.23828 , -44.35547 ,
           -44.472656], shape=(295,), dtype=float32)

  • longitude
    (longitude)
    float32
    112.1 112.2 112.4 ... 156.0 156.2

    long_name :

    longitudes

    type :

    uniform

    units :

    degrees_east

    valid_min :

    -180.0

    valid_max :

    360.0

    axis :

    X

    array([112.06055, 112.23633, 112.41211, ..., 155.83008, 156.00586, 156.18164],
          shape=(252,), dtype=float32)

• Data variables: (2)
  • surface_global_irradiance
    (time, latitude, longitude)
    float64
    dask.array<chunksize=(1, 295, 252), meta=np.ndarray>

    standard_name :

    surface_downwelling_shortwave_flux_in_air

    long_name :

    Total horizontal solar irradiance at surface with bias correction applied

    comment :

    Bias correction has been applied. Surface downwelling shortwave is the sum of direct and diffuse solar radiation incident on the surface, and is sometimes called "global radiation"

    units :

    W m-2

    valid_min :

    0

    valid_max :

    140000

    grid_mapping :

    crs

    Array Chunk Bytes 580.78 kiB 580.78 kiB Shape (1, 295, 252) (1, 295, 252) Dask graph 1 chunks in 23 graph layers Data type float64 numpy.ndarray

  • mslp
    (time, latitude, longitude)
    float32
    dask.array<chunksize=(1, 295, 252), meta=np.ndarray>

    grid_type :

    spatial

    level_type :

    single

    units :

    Pa

    long_name :

    mean sea level pressure

    stash_code :

    16222

    accum_type :

    instantaneous

    Array Chunk Bytes 290.39 kiB 290.39 kiB Shape (1, 295, 252) (1, 295, 252) Dask graph 1 chunks in 21 graph layers Data type float32 numpy.ndarray

• Indexes: (3)
  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2021-06-09 06:00:00'], dtype='datetime64[ns]', name='time', freq=None))

  • latitude
    PandasIndex

    PandasIndex(Index([-10.01953125, -10.13671875, -10.25390625, -10.37109375, -10.48828125,
           -10.60546875, -10.72265625, -10.83984375, -10.95703125, -11.07421875,
           ...
           -43.41796875, -43.53515625, -43.65234375, -43.76953125, -43.88671875,
           -44.00390625, -44.12109375, -44.23828125, -44.35546875, -44.47265625],
          dtype='float32', name='latitude', length=295))

  • longitude
    PandasIndex

    PandasIndex(Index([112.060546875, 112.236328125, 112.412109375, 112.587890625,
           112.763671875, 112.939453125, 113.115234375, 113.291015625,
           113.466796875, 113.642578125,
           ...
           154.599609375, 154.775390625, 154.951171875, 155.126953125,
           155.302734375, 155.478515625, 155.654296875, 155.830078125,
           156.005859375, 156.181640625],
          dtype='float32', name='longitude', length=252))

• Attributes: (58)

  Conventions :

  CF-1.7

  Metadata_Conventions :

  Unidata Dataset Discovery v1.0

  acknowledgment :

  The following acknowledgement is required "Solar radiation data derived from satellite imagery processed by the Bureau of Meteorology from the geostationary satellite Himawari-8 operated by the Japan Meteorological Agency using Heliosat-4 software provided by Mines ParisTech. Contains modified Copernicus Atmosphere Monitoring Service Information 2021."

  cdm_data_type :

  grid

  comment :

  Solar radiation data derived from satellite imagery processed by the Bureau of Meteorology from the geostationary satellite Himawari-8 operated by the Japan Meteorological Agency using Heliosat-4 software provided by Mines ParisTech

  contributor_name :

  Mines ParisTech; Commonwealth of Australia, Bureau of Meteorology, Science and Innovation, Research, Weather and Environmental Prediction, Satellite Applications; Commonwealth of Australia, Bureau of Meteorology, Data and Digital, Digital Services, Business Solutions Services

  contributor_role :

  Development; Development; Operations

  creator_email :

  sat_support@bom.gov.au

  creator_name :

  Commonwealth of Australia, Bureau of Meteorology, Data and Digital, Digital Services, Business Solutions Services

  creator_url :

  http://www.bom.gov.au/satellite/

  disclaimer :

  You are solely responsible for your use of this information and you accept all risks and consequences that might arise from your use of this information. The Bureau of Meteorology ('Bureau') is not in any way liable for losses, damages, costs, expenses and liability of any kind that you or any other person may suffer or incur directly or indirectly from you using this information. This information: is subject to the uncertainties of scientific and technical research; may not be accurate, current or complete; is subject to change without notice. Please refer to http://www.bom.gov.au/other/disclaimer.shtml for disclaimer details. Neither the European Commission nor ECMWF is responsible for any use that may be made of the information it contains.

  geospatial_lat_units :

  degree_north

  geospatial_lon_units :

  degree_east

  geospatial_vertical_max :

  0

  geospatial_vertical_min :

  0

  geospatial_vertical_positive :

  up

  geospatial_vertical_resolution :

  0

  geospatial_vertical_units :

  km

  id :

  ABOM_HS4_SSI_P1S_GRID_v1.0

  institution :

  ABOM

  instrument :

  AHI

  keywords :

  EARTH SCIENCE

  keywords_vocabulary :

  NASA Global Change Master Directory (GCMD) Science Keywords

  license :

  Unless provided under a different license by the Bureau of Meteorology, this information in this data file is provided under the CC BY-NC 4.0 International Licence. The CC BY-NC 4.0 International (Creative Commons Attribution Non-Commercial CC BY-NC v.4.0 international) license summary may be found here: [ https://creativecommons.org/licenses/by-nc/4.0/]. The full license text may be found here: [ https://creativecommons.org/licenses/by-nc/4.0/legalcode ]. For commercial use, please contact Bureau of Meteorology Climate Data Services at climatedata@bom.gov.au

  naming_authority :

  au.gov.bom

  platform :

  Himawari-8

  processing_level :

  L3

  processing_version :

  heliosat-4

  product_specification :

  ABOM_HS4_SSI_P1S_GRID_v1.0

  product_suite :

  ABOM_HS4_SSI

  product_version :

  v1.1

  publisher_email :

  sat_support@bom.gov.au

  publisher_name :

  Commonwealth of Australia, Bureau of Meteorology, Data and Digital, Digital Services, Business Solutions Services

  publisher_url :

  http://www.bom.gov.au/satellite

  standard_name_vocabulary :

  CF-1.7

  summary :

  Solar radiation data derived from satellite imagery processed by the Bureau of Meteorology from the geostationary satellite Himawari-8 operated by the Japan Meteorological Agency using Heliosat-4 software provided by Mines ParisTech. Contains modified Copernicus Atmosphere Monitoring Service Information 2021.

  time_coverage_duration :

  P1S

  time_coverage_resolution :

  P1S

  title :

  Heliosat-4 Solar Radiation - Instantaneous

  geospatial_lat_max :

  -9.99

  geospatial_lat_min :

  -44.51

  geospatial_lon_max :

  156.26999999999998

  geospatial_lon_min :

  111.99

  geospatial_lat_resolution :

  0.02

  geospatial_lon_resolution :

  0.02

  time_coverage_start :

  20210609T060556Z

  time_coverage_end :

  20210609T060835Z

  source :

  IDY25006.APS3.all-flds.slv.2021060900.006.surface.grb2; IDY25006.APS3.all-flds.slv.2021060900.009.surface.grb2; gfs_4_20210609_0000_006.grb2; gfs_4_20210609_0600_006.grb2; CAMS-CLIM; S_NWC_HS4C_HIMA08_HIMA-N-NR_20210609T060000Z.h5; S_NWC_HS4S_HIMA08_HIMA-N-NR_20210609T060000Z.h5; model_BC_ghi_clear_ACCESS_CAMS-CLIM_all01_all12_60_30_1_10; model_BC_ghi_cloud_ACCESS_CAMS-CLIM_all01_all12_60_30_1_10; model_BC_dni_cloud_ACCESS_CAMS-CLIM_all01_all12_60_30_1_10; model_BC_dni_cloud_ACCESS_CAMS-CLIM_all01_all12_60_30_1_10

  uuid :

  538d58e4-9ef3-4901-8f0a-0154a3dade3b

  date_created :

  20211219T225111Z

  bias_correction_applied :

  1

  bias_correction_applied_meaning :

  0: not applied; 1:applied

  quality :

  0

  quality_meaning :

  0: no_known_issues 1: known_issue

  project :

  Gridded Solar Observations

  history :

  Mon Mar 4 01:55:13 2024: ncatted -a references,global,o,c,Poulsen C., Majewski L. J. (2022) Gridded satellite solar observations Project: Implementation of Heliosat-4 and application of bias correction, Australian Bureau of Meteorology, Melbourne, Victoria. Qu A., Oumbe A., Blanc P., Espinar B., Gesell G., et al. (2017) Fast radiative transfer parameterisation for assessing the surface solar irradiance: The Heliosat-4 method. Meteorologische Zeitschrift, v26, p33-57. 10.1127/metz/2016/0781 . hal-01512589 /g/data/rv74/satellite-products/arc/der/himawari-ahi/solar/p1s/v1.1/2021/06/09/IDE00326.202106090600.nc Mon Mar 4 00:54:18 2024: ncatted -a references,global,a,c,Poulsen C., Majewski L. J. (2022) Gridded satellite solar observations Project: Implementation of Heliosat-4 and application of bias correction, Australian Bureau of Meteorology, Melbourne, Victoria. Qu A., Oumbe A., Blanc P., Espinar B., Gesell G., et al. (2017) Fast radiative transfer parameterisation for assessing the surface solar irradiance: The Heliosat-4 method. Meteorologische Zeitschrift, v26, p33-57. 10.1127/metz/2016/0781 . hal-01512589 /g/data/rv74/satellite-products/arc/der/himawari-ahi/solar/p1s/v1.1/2021/06/09/IDE00326.202106090600.nc heliosat-4; post_process_p1s.py

  references :

  Poulsen C., Majewski L. J. (2022) Gridded satellite solar observations Project: Implementation of Heliosat-4 and application of bias correction, Australian Bureau of Meteorology, Melbourne, Victoria. Qu A., Oumbe A., Blanc P., Espinar B., Gesell G., et al. (2017) Fast radiative transfer parameterisation for assessing the surface solar irradiance: The Heliosat-4 method. Meteorologische Zeitschrift, v26, p33-57. 10.1127/metz/2016/0781 . hal-01512589

  NCO :

  netCDF Operators version 4.7.7 (Homepage = http://nco.sf.net, Code = http://github.com/nco/nco)

sample[1]

<xarray.Dataset> Size: 40MB
Dimensions:     (y: 2050, n2: 2, x: 2450)
Coordinates:
  * y           (y) float64 16kB -1.001e+03 -1.003e+03 ... -5.097e+03 -5.099e+03
  * x           (x) float64 20kB -2.299e+03 -2.297e+03 ... 2.597e+03 2.599e+03
Dimensions without coordinates: n2
Data variables:
    valid_time  datetime64[ns] 8B 2021-06-09T06:00:00
    proj        int8 1B 0
    y_bounds    (y, n2) float64 33kB -1e+03 -1.002e+03 ... -5.098e+03 -5.1e+03
    x_bounds    (x, n2) float64 39kB -2.3e+03 -2.298e+03 ... 2.598e+03 2.6e+03
    rain_rate   (y, x) float64 40MB nan nan nan nan nan ... nan nan nan nan nan
Attributes:
    Conventions:         CF-1.7
    contributing_sites:  [ 2  3  4  5  6  8  9 14 15 16 17 19 22 23 24 25 27 ...
    institution:         Commonwealth of Australia, Bureau of Meteorology (AB...
    licence:             http://www.bom.gov.au/other/copyright.shtml
    source:              rainfields 3.2.4 drs-rainfields 2021-02-02
    station_id:          310
    station_name:        Ausm310
    title:               Bias Corrected Rainfall Rate Mosaic

xarray.Dataset

• Dimensions:
  • y: 2050
  • n2: 2
  • x: 2450
• Coordinates: (2)
  • y
    (y)
    float64
    -1.001e+03 ... -5.099e+03

    standard_name :

    projection_y_coordinate

    units :

    km

    bounds :

    y_bounds

    array([-1001., -1003., -1005., ..., -5095., -5097., -5099.], shape=(2050,))

  • x
    (x)
    float64
    -2.299e+03 -2.297e+03 ... 2.599e+03

    standard_name :

    projection_x_coordinate

    units :

    km

    bounds :

    x_bounds

    array([-2299., -2297., -2295., ...,  2595.,  2597.,  2599.], shape=(2450,))

• Data variables: (5)
  • valid_time
    ()
    datetime64[ns]
    2021-06-09T06:00:00

    long_name :

    Valid time

    standard_name :

    time

    array('2021-06-09T06:00:00.000000000', dtype='datetime64[ns]')

  • proj
    ()
    int8
    0

    grid_mapping_name :

    albers_conical_equal_area

    standard_parallel :

    [-18. -36.]

    longitude_of_central_meridian :

    132.0

    latitude_of_projection_origin :

    0.0

    false_easting :

    0.0

    false_northing :

    0.0

    towgs84 :

    [0. 0. 0. 0. 0. 0. 0.]

    array(0, dtype=int8)

  • y_bounds
    (y, n2)
    float64
    -1e+03 -1.002e+03 ... -5.1e+03

    array([[-1000., -1002.],
           [-1002., -1004.],
           [-1004., -1006.],
           ...,
           [-5094., -5096.],
           [-5096., -5098.],
           [-5098., -5100.]], shape=(2050, 2))

  • x_bounds
    (x, n2)
    float64
    -2.3e+03 -2.298e+03 ... 2.6e+03

    array([[-2300., -2298.],
           [-2298., -2296.],
           [-2296., -2294.],
           ...,
           [ 2594.,  2596.],
           [ 2596.,  2598.],
           [ 2598.,  2600.]], shape=(2450, 2))

  • rain_rate
    (y, x)
    float64
    nan nan nan nan ... nan nan nan nan

    grid_mapping :

    proj

    long_name :

    Bias corrected rainfall rate

    standard_name :

    rainfall_rate

    units :

    mm hr-1

    array([[nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan],
           ...,
           [nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan]], shape=(2050, 2450))

• Indexes: (2)
  • y
    PandasIndex

    PandasIndex(Index([-1001.0, -1003.0, -1005.0, -1007.0, -1009.0, -1011.0, -1013.0, -1015.0,
           -1017.0, -1019.0,
           ...
           -5081.0, -5083.0, -5085.0, -5087.0, -5089.0, -5091.0, -5093.0, -5095.0,
           -5097.0, -5099.0],
          dtype='float64', name='y', length=2050))

  • x
    PandasIndex

    PandasIndex(Index([-2299.0, -2297.0, -2295.0, -2293.0, -2291.0, -2289.0, -2287.0, -2285.0,
           -2283.0, -2281.0,
           ...
            2581.0,  2583.0,  2585.0,  2587.0,  2589.0,  2591.0,  2593.0,  2595.0,
            2597.0,  2599.0],
          dtype='float64', name='x', length=2450))

• Attributes: (8)

  Conventions :

  CF-1.7

  contributing_sites :

  [ 2 3 4 5 6 8 9 14 15 16 17 19 22 23 24 25 27 28 29 31 32 33 36 38 40 41 42 44 46 49 50 52 53 54 55 58 63 64 66 67 68 69 70 71 72 73 75 76 77 79 93 95 97]

  institution :

  Commonwealth of Australia, Bureau of Meteorology (ABN 92 637 533 532)

  licence :

  http://www.bom.gov.au/other/copyright.shtml

  source :

  rainfields 3.2.4 drs-rainfields 2021-02-02

  station_id :

  310

  station_name :

  Ausm310

  title :

  Bias Corrected Rainfall Rate Mosaic