DataProcessor

The first step in any modelling pipeline is to normalise and standardise the data. In DeepSensor, this is achieved with the DataProcessor class. Let’s load some environmental data from deepsensor.data.sources and see how it works!

Note

Some of the data downloader functions used here require additional dependencies. To run this yourself you will need to run:

pip install rioxarray

to install the rioxarray package and

pip install git+https://github.com/scott-hosking/get-station-data.git

to install the get_station_data package.

Show code cell source Hide code cell source

import logging
logging.captureWarnings(True)

import xarray as xr
import pandas as pd

# Using the same settings allows use to use pre-downloaded cached data
data_range = ("2015-06-25", "2015-06-30")
extent = "europe"
station_var_IDs = ["TAVG", "PRCP"]
era5_var_IDs = ["2m_temperature", "10m_u_component_of_wind", "10m_v_component_of_wind"]
aux_var_IDs = ["elevation", "tpi"]
cache_dir = "../../.datacache"

from deepsensor.data import DataProcessor
from deepsensor.data.sources import get_ghcnd_station_data, get_era5_reanalysis_data, get_earthenv_auxiliary_data, get_gldas_land_mask

station_raw_df = get_ghcnd_station_data(station_var_IDs, extent, date_range=data_range, cache=True, cache_dir=cache_dir)
era5_raw_ds = get_era5_reanalysis_data(era5_var_IDs, extent, date_range=data_range, cache=True, cache_dir=cache_dir)
aux_raw_ds = get_earthenv_auxiliary_data(aux_var_IDs, extent, "1KM", cache=True, cache_dir=cache_dir)
land_mask_raw_ds = get_gldas_land_mask(extent, cache=True, cache_dir=cache_dir)

100%|████████████████████████████████████████████████████████████████| 3142/3142 [02:42<00:00, 19.33it/s]

Initialising a DataProcessor

To initialise a DataProcessor object, provide it with the names of the spatiotemporal dimensions in your data (defaults to time, x1, x2).

data_processor = DataProcessor(x1_name="lat", x2_name="lon")
print(data_processor)

DataProcessor with normalisation params:
{'coords': {'time': {'name': 'time'},
            'x1': {'map': None, 'name': 'lat'},
            'x2': {'map': None, 'name': 'lon'}}}

Normalising data with a DataProcessor

Calling a DataProcessor on an xarray or pandas object will compute normalisation parameters for each variable in the object (if not already computed) and return the normalised object/s.

era5_ds = data_processor(era5_raw_ds)
era5_ds

<xarray.Dataset>
Dimensions:                  (time: 6, x1: 141, x2: 221)
Coordinates:
  * time                     (time) datetime64[ns] 2015-06-25 ... 2015-06-30
  * x1                       (x1) float32 0.6364 0.6318 0.6273 ... 0.004545 0.0
  * x2                       (x2) float32 0.0 0.004545 0.009091 ... 0.9955 1.0
Data variables:
    2m_temperature           (time, x1, x2) float32 -2.652 -2.635 ... 2.322
    10m_u_component_of_wind  (time, x1, x2) float32 1.987 1.985 ... 1.572 1.529
    10m_v_component_of_wind  (time, x1, x2) float32 1.054 1.018 ... -0.724

xarray.Dataset

• Dimensions:
  • time: 6
  • x1: 141
  • x2: 221
• Coordinates: (3)
  • time
    (time)
    datetime64[ns]
    2015-06-25 ... 2015-06-30

    array(['2015-06-25T00:00:00.000000000', '2015-06-26T00:00:00.000000000',
           '2015-06-27T00:00:00.000000000', '2015-06-28T00:00:00.000000000',
           '2015-06-29T00:00:00.000000000', '2015-06-30T00:00:00.000000000'],
          dtype='datetime64[ns]')

  • x1
    (x1)
    float32
    0.6364 0.6318 ... 0.004545 0.0

    array([0.636364, 0.631818, 0.627273, 0.622727, 0.618182, 0.613636, 0.609091,
           0.604545, 0.6     , 0.595455, 0.590909, 0.586364, 0.581818, 0.577273,
           0.572727, 0.568182, 0.563636, 0.559091, 0.554545, 0.55    , 0.545455,
           0.540909, 0.536364, 0.531818, 0.527273, 0.522727, 0.518182, 0.513636,
           0.509091, 0.504545, 0.5     , 0.495455, 0.490909, 0.486364, 0.481818,
           0.477273, 0.472727, 0.468182, 0.463636, 0.459091, 0.454545, 0.45    ,
           0.445455, 0.440909, 0.436364, 0.431818, 0.427273, 0.422727, 0.418182,
           0.413636, 0.409091, 0.404545, 0.4     , 0.395455, 0.390909, 0.386364,
           0.381818, 0.377273, 0.372727, 0.368182, 0.363636, 0.359091, 0.354545,
           0.35    , 0.345455, 0.340909, 0.336364, 0.331818, 0.327273, 0.322727,
           0.318182, 0.313636, 0.309091, 0.304545, 0.3     , 0.295455, 0.290909,
           0.286364, 0.281818, 0.277273, 0.272727, 0.268182, 0.263636, 0.259091,
           0.254545, 0.25    , 0.245455, 0.240909, 0.236364, 0.231818, 0.227273,
           0.222727, 0.218182, 0.213636, 0.209091, 0.204545, 0.2     , 0.195455,
           0.190909, 0.186364, 0.181818, 0.177273, 0.172727, 0.168182, 0.163636,
           0.159091, 0.154545, 0.15    , 0.145455, 0.140909, 0.136364, 0.131818,
           0.127273, 0.122727, 0.118182, 0.113636, 0.109091, 0.104545, 0.1     ,
           0.095455, 0.090909, 0.086364, 0.081818, 0.077273, 0.072727, 0.068182,
           0.063636, 0.059091, 0.054545, 0.05    , 0.045455, 0.040909, 0.036364,
           0.031818, 0.027273, 0.022727, 0.018182, 0.013636, 0.009091, 0.004545,
           0.      ], dtype=float32)

  • x2
    (x2)
    float32
    0.0 0.004545 ... 0.9955 1.0

    array([0.      , 0.004545, 0.009091, ..., 0.990909, 0.995455, 1.      ],
          dtype=float32)

• Data variables: (3)
  • 2m_temperature
    (time, x1, x2)
    float32
    -2.652 -2.635 ... 2.235 2.322

    array([[[-2.6523552 , -2.6348572 , -2.6191788 , ..., -1.190525  ,
             -1.1986656 , -1.2125969 ],
            [-2.615448  , -2.6060944 , -2.5952187 , ..., -1.1386185 ,
             -1.1468468 , -1.160923  ],
            [-2.5641403 , -2.5547295 , -2.545517  , ..., -1.0934029 ,
             -1.1080284 , -1.1220665 ],
            ...,
            [ 0.7250366 ,  0.71790314,  0.7080231 , ...,  1.9699783 ,
              2.1188316 ,  2.3057442 ],
            [ 0.7476959 ,  0.7334976 ,  0.7221947 , ...,  1.8910828 ,
              2.107647  ,  2.3647995 ],
            [ 0.7528763 ,  0.7448845 ,  0.73334885, ...,  2.2095985 ,
              2.31279   ,  2.398697  ]],
    
           [[-2.5501785 , -2.5397987 , -2.5316696 , ..., -1.46101   ,
             -1.459446  , -1.4568634 ],
            [-2.4556313 , -2.4541702 , -2.4519768 , ..., -1.4176788 ,
             -1.4160271 , -1.4134521 ],
            [-2.3554993 , -2.3542175 , -2.3529587 , ..., -1.376133  ,
             -1.3760033 , -1.3716049 ],
    ...
            [ 0.71673584,  0.69192123,  0.66566086, ...,  1.9801445 ,
              2.1486855 ,  2.3426056 ],
            [ 0.7229271 ,  0.69088745,  0.66340256, ...,  1.9233055 ,
              2.1533813 ,  2.4106102 ],
            [ 0.7176094 ,  0.6897087 ,  0.6639595 , ...,  2.2028618 ,
              2.3241425 ,  2.4215965 ]],
    
           [[-2.3398476 , -2.3420486 , -2.34449   , ..., -1.8318138 ,
             -1.8369827 , -1.8382492 ],
            [-2.3311996 , -2.3352852 , -2.3385582 , ..., -1.7985497 ,
             -1.8037567 , -1.8049812 ],
            [-2.3111572 , -2.315361  , -2.3195343 , ..., -1.76437   ,
             -1.769947  , -1.7740784 ],
            ...,
            [ 0.8371887 ,  0.82435226,  0.8161011 , ...,  1.8790741 ,
              2.038414  ,  2.2233696 ],
            [ 0.85030365,  0.8317833 ,  0.8202286 , ...,  1.8408623 ,
              2.0677109 ,  2.3173485 ],
            [ 0.85694885,  0.83976364,  0.8279114 , ...,  2.1220284 ,
              2.2351494 ,  2.3223839 ]]], dtype=float32)

  • 10m_u_component_of_wind
    (time, x1, x2)
    float32
    1.987 1.985 1.985 ... 1.572 1.529

    array([[[ 1.9867022 ,  1.9852841 ,  1.9854782 , ..., -0.03774244,
             -0.01398502,  0.01424471],
            [ 2.0099676 ,  2.0157323 ,  2.0208912 , ...,  0.02567162,
              0.04982929,  0.07827641],
            [ 1.9669826 ,  1.9727471 ,  1.9787295 , ...,  0.02890863,
              0.06622045,  0.10174783],
            ...,
            [ 0.107547  ,  0.09797318,  0.05313496, ...,  3.6755922 ,
              3.618241  ,  3.3195407 ],
            [ 0.11976318,  0.08455609,  0.02544282, ...,  3.5628335 ,
              3.531126  ,  3.1831043 ],
            [ 0.13624574,  0.06425302, -0.01961268, ...,  3.0371287 ,
              3.0529816 ,  2.9593139 ]],
    
           [[ 1.9374385 ,  1.9820731 ,  2.0263605 , ...,  1.8701108 ,
              1.9066098 ,  1.9328997 ],
            [ 1.8437095 ,  1.8855307 ,  1.9292805 , ...,  1.9600194 ,
              1.997056  ,  2.0236034 ],
            [ 1.5285546 ,  1.5705444 ,  1.6131843 , ...,  2.0331178 ,
              2.056102  ,  2.0738413 ],
    ...
            [-0.24281344, -0.2311048 , -0.21245493, ...,  2.0393054 ,
              1.952157  ,  1.7797613 ],
            [-0.30337298, -0.29068774, -0.27305645, ...,  1.8711834 ,
              1.8161476 ,  1.6167116 ],
            [-0.3554582 , -0.34528267, -0.3267063 , ...,  1.503752  ,
              1.5538417 ,  1.5262661 ]],
    
           [[-0.88259035, -0.9220995 , -0.9566686 , ...,  1.8813791 ,
              1.9043632 ,  1.9191835 ],
            [-1.0911965 , -1.1048253 , -1.1266619 , ...,  1.9713376 ,
              1.9946496 ,  2.0095243 ],
            [-1.3249073 , -1.3380989 , -1.350875  , ...,  1.9883761 ,
              2.018781  ,  2.0378637 ],
            ...,
            [ 1.1508583 ,  1.1197207 ,  1.0529324 , ...,  2.0892966 ,
              2.0107918 ,  1.8264158 ],
            [ 1.074452  ,  1.0030296 ,  0.92812055, ...,  1.8889747 ,
              1.855717  ,  1.6535263 ],
            [ 0.97419816,  0.8868299 ,  0.8100632 , ...,  1.524638  ,
              1.5723332 ,  1.5288672 ]]], dtype=float32)

  • 10m_v_component_of_wind
    (time, x1, x2)
    float32
    1.054 1.018 0.983 ... -0.778 -0.724

    array([[[ 1.0540881e+00,  1.0182923e+00,  9.8300970e-01, ...,
              1.4420865e+00,  1.4325457e+00,  1.4366663e+00],
            [ 1.0086231e+00,  9.7437513e-01,  9.3983626e-01, ...,
              1.5612113e+00,  1.5515506e+00,  1.5557227e+00],
            [ 9.7015196e-01,  9.3555319e-01,  9.0095431e-01, ...,
              1.6508585e+00,  1.6414031e+00,  1.6372997e+00],
            ...,
            [-7.2610891e-01, -7.6200718e-01, -7.6854742e-01, ...,
             -7.1113938e-01, -8.8267940e-01, -8.7818265e-01],
            [-8.4160048e-01, -8.6288816e-01, -8.5716861e-01, ...,
             -1.2555542e+00, -1.1878699e+00, -1.0134059e+00],
            [-9.1483313e-01, -9.2714417e-01, -9.1954392e-01, ...,
             -1.1848691e+00, -9.7225833e-01, -8.0542868e-01]],
    
           [[ 7.6629186e-01,  7.3474604e-01,  7.0416361e-01, ...,
             -7.0524555e-01, -6.3118845e-01, -5.4704404e-01],
            [ 8.4295106e-01,  8.2335114e-01,  7.9528624e-01, ...,
             -5.4133314e-01, -4.6632135e-01, -3.8118848e-01],
            [ 9.2358929e-01,  9.0448791e-01,  8.8523436e-01, ...,
             -3.9005920e-01, -3.1745508e-01, -2.4389644e-01],
    ...
            [-1.1776097e+00, -1.2850400e+00, -1.4078401e+00, ...,
             -5.3842860e-01, -6.8481743e-01, -6.9051790e-01],
            [-1.2283620e+00, -1.3342761e+00, -1.4526581e+00, ...,
             -8.7780237e-01, -8.4295344e-01, -7.6150787e-01],
            [-1.2609743e+00, -1.3704449e+00, -1.4837912e+00, ...,
             -7.6381052e-01, -6.4306021e-01, -5.7118100e-01]],
    
           [[ 1.6113994e-01,  1.6557619e-01,  1.6842957e-01, ...,
             -1.3953098e+00, -1.3844689e+00, -1.3523219e+00],
            [ 3.0843057e-02,  1.6971957e-02,  1.4732994e-02, ...,
             -1.3481355e+00, -1.3371387e+00, -1.3046267e+00],
            [-7.2961196e-02, -8.7894514e-02, -1.0261957e-01, ...,
             -1.2933800e+00, -1.2916409e+00, -1.2693762e+00],
            ...,
            [ 3.5955313e-01,  3.4301606e-01,  2.8095013e-01, ...,
             -5.0673562e-01, -6.5742260e-01, -6.8586248e-01],
            [ 3.1950191e-01,  2.9112437e-01,  2.1389596e-01, ...,
             -8.8749373e-01, -8.6598921e-01, -8.0972373e-01],
            [ 2.2508034e-01,  1.8238398e-01,  9.3731761e-02, ...,
             -8.9222151e-01, -7.7802432e-01, -7.2397697e-01]]], dtype=float32)

• Indexes: (3)
  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2015-06-25', '2015-06-26', '2015-06-27', '2015-06-28',
                   '2015-06-29', '2015-06-30'],
                  dtype='datetime64[ns]', name='time', freq='D'))

  • x1
    PandasIndex

    PandasIndex(Index([  0.6363636255264282,   0.6318181753158569,   0.6272727251052856,
             0.6227272748947144,   0.6181818246841431,   0.6136363744735718,
             0.6090909242630005,   0.6045454740524292,   0.6000000238418579,
             0.5954545736312866,
           ...
           0.040909089148044586,    0.036363635212183,  0.03181818127632141,
           0.027272727340459824, 0.022727273404598236,   0.0181818176060915,
           0.013636363670229912,  0.00909090880304575, 0.004545454401522875,
                            0.0],
          dtype='float32', name='x1', length=141))

  • x2
    PandasIndex

    PandasIndex(Index([                 0.0, 0.004545454401522875,  0.00909090880304575,
           0.013636363670229912,   0.0181818176060915, 0.022727273404598236,
           0.027272727340459824,  0.03181818127632141,    0.036363635212183,
           0.040909089148044586,
           ...
             0.9590908885002136,   0.9636363387107849,   0.9681817889213562,
             0.9727272987365723,   0.9772727489471436,   0.9818181991577148,
             0.9863636493682861,   0.9909090995788574,   0.9954545497894287,
                            1.0],
          dtype='float32', name='x2', length=221))

• Attributes: (0)

station_df = data_processor(station_raw_df)
station_df

				PRCP	TAVG
time	x1	x2	station
2015-06-25	0.000309	0.246364	AGM00060531	-0.278121	0.759107
	0.001818	0.239091	AGE00147716	-0.278121	0.836200
	0.002127	0.940909	SYM00040030	NaN	1.221668
	0.003036	0.314855	AGM00060514	-0.278121	1.318035
	0.003636	0.261509	AGM00060520	-0.278121	1.125301
...	...	...	...	...	...
2015-06-30	0.198782	0.412727	ITM00016052	NaN	-2.575188
	0.061218	0.263636	SPM00008359	-0.278121	1.645682
	0.370600	0.940000	RSM00027611	-0.278121	-0.358749
	0.437818	0.597455	SWE00138750	-0.278121	NaN
	0.522909	0.498727	SWE00140158	-0.134323	NaN

16922 rows × 2 columns

You can also process multiple variables in one DataProcessor call, and choose to normalise the data with "min_max", which will scale the data to the range [-1, 1]. For example:

aux_ds, land_mask_ds = data_processor([aux_raw_ds, land_mask_raw_ds], method="min_max")
aux_ds

<xarray.Dataset>
Dimensions:    (x1: 4200, x2: 6600)
Coordinates:
  * x1         (x1) float64 0.6363 0.6361 0.636 ... 0.0002273 7.576e-05
  * x2         (x2) float64 7.576e-05 0.0002273 0.0003788 ... 0.9998 0.9999
Data variables:
    elevation  (x1, x2) float32 -0.921 -0.921 -0.921 ... -0.8095 -0.8095 -0.8099
    tpi        (x1, x2) float32 -0.09401 -0.09401 -0.09401 ... -0.09305 -0.09199

xarray.Dataset

• Dimensions:
  • x1: 4200
  • x2: 6600
• Coordinates: (2)
  • x1
    (x1)
    float64
    0.6363 0.6361 ... 7.576e-05

    array([6.362879e-01, 6.361364e-01, 6.359848e-01, ..., 3.787879e-04,
           2.272727e-04, 7.575758e-05])

  • x2
    (x2)
    float64
    7.576e-05 0.0002273 ... 0.9999

    array([7.575758e-05, 2.272727e-04, 3.787879e-04, ..., 9.996212e-01,
           9.997727e-01, 9.999242e-01])

• Data variables: (2)
  • elevation
    (x1, x2)
    float32
    -0.921 -0.921 ... -0.8095 -0.8099

    array([[-0.92104805, -0.92104805, -0.92104805, ..., -0.92104805,
            -0.92104805, -0.92104805],
           [-0.92104805, -0.92104805, -0.92104805, ..., -0.92104805,
            -0.92104805, -0.92104805],
           [-0.92104805, -0.92104805, -0.92104805, ..., -0.92104805,
            -0.92104805, -0.92104805],
           ...,
           [-0.92104805, -0.92104805, -0.92104805, ..., -0.8093372 ,
            -0.80907065, -0.80955046],
           [-0.92104805, -0.92104805, -0.92104805, ..., -0.8094705 ,
            -0.8091773 , -0.8100036 ],
           [-0.92104805, -0.92104805, -0.92104805, ..., -0.8094971 ,
            -0.80952376, -0.8099236 ]], dtype=float32)

  • tpi
    (x1, x2)
    float32
    -0.09401 -0.09401 ... -0.09199

    array([[-0.0940097 , -0.0940097 , -0.0940097 , ..., -0.0940097 ,
            -0.0940097 , -0.0940097 ],
           [-0.0940097 , -0.0940097 , -0.0940097 , ..., -0.0940097 ,
            -0.0940097 , -0.0940097 ],
           [-0.0940097 , -0.0940097 , -0.0940097 , ..., -0.0940097 ,
            -0.0940097 , -0.0940097 ],
           ...,
           [-0.0940097 , -0.0940097 , -0.0940097 , ..., -0.08949416,
            -0.09103137, -0.09256858],
           [-0.0940097 , -0.0940097 , -0.0940097 , ..., -0.09199212,
            -0.09439401, -0.09804487],
           [-0.0940097 , -0.0940097 , -0.0940097 , ..., -0.09352933,
            -0.09304895, -0.09199212]], dtype=float32)

• Indexes: (2)
  • x1
    PandasIndex

    PandasIndex(Index([    0.6362878787878788,     0.6361363636363638,     0.6359848484848486,
               0.6358333333333333,     0.6356818181818182,      0.635530303030303,
                0.635378787878788,     0.6352272727272728,     0.6350757575757577,
               0.6349242424242424,
           ...
            0.0014393939393940514,  0.0012878787878789085,  0.0011363636363637655,
            0.0009848484848486226,  0.0008333333333333505,  0.0006818181818182077,
            0.0005303030303030647, 0.00037878787878792183, 0.00022727272727277896,
            7.575757575763605e-05],
          dtype='float64', name='x1', length=4200))

  • x2
    PandasIndex

    PandasIndex(Index([ 7.575757575750686e-05, 0.00022727272727252057,  0.0003787878787875343,
            0.0005303030303030647,  0.0006818181818180784,  0.0008333333333330922,
            0.0009848484848486226,  0.0011363636363636363,    0.00128787878787865,
            0.0014393939393936637,
           ...
               0.9985606060606058,     0.9987121212121208,     0.9988636363636364,
               0.9990151515151514,     0.9991666666666664,     0.9993181818181819,
               0.9994696969696969,     0.9996212121212119,     0.9997727272727269,
               0.9999242424242425],
          dtype='float64', name='x2', length=6600))

• Attributes: (0)

DataProcessor configuration

The DataProcessor keeps track of the normalisation parameters used to transform the data.

When the DataProcessor is called on data with a variable ID that matches one in the config dictionary, those normalisation parameters are used.

print(data_processor)

DataProcessor with normalisation params:
{'10m_u_component_of_wind': {'method': 'mean_std',
                             'params': {'mean': 0.644899845123291,
                                        'std': 2.8509252071380615}},
 '10m_v_component_of_wind': {'method': 'mean_std',
                             'params': {'mean': -0.19969606399536133,
                                        'std': 3.2448606491088867}},
 '2m_temperature': {'method': 'mean_std',
                    'params': {'mean': 289.39849853515625,
                               'std': 5.538551330566406}},
 'GLDAS_mask': {'method': 'min_max', 'params': {'max': 1.0, 'min': 0.0}},
 'PRCP': {'method': 'mean_std',
          'params': {'mean': 1.1604694953552597, 'std': 4.172541647864038}},
 'TAVG': {'method': 'mean_std',
          'params': {'mean': 19.0613726868119, 'std': 5.188503809362459}},
 'coords': {'time': {'name': 'time'},
            'x1': {'map': (35.0, 90.0), 'name': 'lat'},
            'x2': {'map': (-15.0, 40.0), 'name': 'lon'}},
 'elevation': {'method': 'min_max',
               'params': {'max': 4504.4375, 'min': -185.125}},
 'tpi': {'method': 'min_max', 'params': {'max': 88.9609375, 'min': -73.671875}}}

Unnormalising data

Keeping track of the normalisation parameters allows us to easily unnormalise data back to the original coordinates and units:

era5_raw_ds_unnormalised = data_processor.unnormalise(era5_ds)
xr.testing.assert_allclose(era5_raw_ds, era5_raw_ds_unnormalised, atol=1e-6)
era5_raw_ds_unnormalised

<xarray.Dataset>
Dimensions:                  (time: 6, lat: 141, lon: 221)
Coordinates:
  * time                     (time) datetime64[ns] 2015-06-25 ... 2015-06-30
  * lat                      (lat) float32 70.0 69.75 69.5 ... 35.5 35.25 35.0
  * lon                      (lon) float32 -15.0 -14.75 -14.5 ... 39.75 40.0
Data variables:
    2m_temperature           (time, lat, lon) float32 274.7 274.8 ... 302.3
    10m_u_component_of_wind  (time, lat, lon) float32 6.309 6.305 ... 5.004
    10m_v_component_of_wind  (time, lat, lon) float32 3.221 3.105 ... -2.549

xarray.Dataset

• Dimensions:
  • time: 6
  • lat: 141
  • lon: 221
• Coordinates: (3)
  • time
    (time)
    datetime64[ns]
    2015-06-25 ... 2015-06-30

    array(['2015-06-25T00:00:00.000000000', '2015-06-26T00:00:00.000000000',
           '2015-06-27T00:00:00.000000000', '2015-06-28T00:00:00.000000000',
           '2015-06-29T00:00:00.000000000', '2015-06-30T00:00:00.000000000'],
          dtype='datetime64[ns]')

  • lat
    (lat)
    float32
    70.0 69.75 69.5 ... 35.5 35.25 35.0

    array([70.  , 69.75, 69.5 , 69.25, 69.  , 68.75, 68.5 , 68.25, 68.  , 67.75,
           67.5 , 67.25, 67.  , 66.75, 66.5 , 66.25, 66.  , 65.75, 65.5 , 65.25,
           65.  , 64.75, 64.5 , 64.25, 64.  , 63.75, 63.5 , 63.25, 63.  , 62.75,
           62.5 , 62.25, 62.  , 61.75, 61.5 , 61.25, 61.  , 60.75, 60.5 , 60.25,
           60.  , 59.75, 59.5 , 59.25, 59.  , 58.75, 58.5 , 58.25, 58.  , 57.75,
           57.5 , 57.25, 57.  , 56.75, 56.5 , 56.25, 56.  , 55.75, 55.5 , 55.25,
           55.  , 54.75, 54.5 , 54.25, 54.  , 53.75, 53.5 , 53.25, 53.  , 52.75,
           52.5 , 52.25, 52.  , 51.75, 51.5 , 51.25, 51.  , 50.75, 50.5 , 50.25,
           50.  , 49.75, 49.5 , 49.25, 49.  , 48.75, 48.5 , 48.25, 48.  , 47.75,
           47.5 , 47.25, 47.  , 46.75, 46.5 , 46.25, 46.  , 45.75, 45.5 , 45.25,
           45.  , 44.75, 44.5 , 44.25, 44.  , 43.75, 43.5 , 43.25, 43.  , 42.75,
           42.5 , 42.25, 42.  , 41.75, 41.5 , 41.25, 41.  , 40.75, 40.5 , 40.25,
           40.  , 39.75, 39.5 , 39.25, 39.  , 38.75, 38.5 , 38.25, 38.  , 37.75,
           37.5 , 37.25, 37.  , 36.75, 36.5 , 36.25, 36.  , 35.75, 35.5 , 35.25,
           35.  ], dtype=float32)

  • lon
    (lon)
    float32
    -15.0 -14.75 -14.5 ... 39.75 40.0

    array([-15.  , -14.75, -14.5 , ...,  39.5 ,  39.75,  40.  ], dtype=float32)

• Data variables: (3)
  • 2m_temperature
    (time, lat, lon)
    float32
    274.7 274.8 274.9 ... 301.8 302.3

    array([[[274.70828, 274.8052 , 274.89203, ..., 282.80472, 282.7596 ,
             282.68246],
            [274.91272, 274.9645 , 275.02475, ..., 283.0922 , 283.04663,
             282.96866],
            [275.19687, 275.249  , 275.30002, ..., 283.34262, 283.26163,
             283.18387],
            ...,
            [293.41415, 293.37463, 293.31992, ..., 300.30933, 301.13376,
             302.16898],
            [293.53964, 293.461  , 293.3984 , ..., 299.87234, 301.0718 ,
             302.49606],
            [293.56833, 293.52408, 293.46017, ..., 301.63647, 302.208  ,
             302.6838 ]],
    
           [[275.2742 , 275.3317 , 275.3767 , ..., 281.3066 , 281.31528,
             281.3296 ],
            [275.79785, 275.80594, 275.81808, ..., 281.5466 , 281.55576,
             281.57   ],
            [276.35245, 276.35956, 276.36652, ..., 281.7767 , 281.77744,
             281.8018 ],
    ...
            [293.36816, 293.23074, 293.0853 , ..., 300.36563, 301.2991 ,
             302.37314],
            [293.40247, 293.225  , 293.07278, ..., 300.0508 , 301.3251 ,
             302.7498 ],
            [293.37302, 293.21848, 293.07587, ..., 301.59915, 302.27087,
             302.81064]],
    
           [[276.43915, 276.42694, 276.41342, ..., 279.2529 , 279.22427,
             279.21725],
            [276.48703, 276.46442, 276.4463 , ..., 279.43713, 279.4083 ,
             279.40152],
            [276.59802, 276.57477, 276.55164, ..., 279.62643, 279.59555,
             279.57266],
            ...,
            [294.0353 , 293.9642 , 293.91852, ..., 299.80585, 300.68835,
             301.71274],
            [294.10794, 294.00537, 293.94138, ..., 299.5942 , 300.85062,
             302.23325],
            [294.14474, 294.04956, 293.98392, ..., 301.15146, 301.77798,
             302.26114]]], dtype=float32)

  • 10m_u_component_of_wind
    (time, lat, lon)
    float32
    6.309 6.305 6.305 ... 5.128 5.004

    array([[[ 6.3088393 ,  6.304796  ,  6.3053493 , ...,  0.537299  ,
              0.6050296 ,  0.68551046],
            [ 6.375167  ,  6.391602  ,  6.4063096 , ...,  0.7180877 ,
              0.7869594 ,  0.86806005],
            [ 6.25262   ,  6.2690544 ,  6.2861094 , ...,  0.72731614,
              0.8336894 ,  0.93497527],
            ...,
            [ 0.9515083 ,  0.92421407,  0.7963836 , ..., 11.123737  ,
             10.960234  , 10.108662  ],
            [ 0.98633575,  0.88596296,  0.7174354 , ..., 10.802271  ,
             10.711876  ,  9.719692  ],
            [ 1.0333263 ,  0.8280804 ,  0.58898556, ...,  9.303526  ,
              9.348722  ,  9.081682  ]],
    
           [[ 6.168392  ,  6.295642  ,  6.421902  , ...,  5.9764457 ,
              6.0805016 ,  6.1554523 ],
            [ 5.9011774 ,  6.0204067 ,  6.1451344 , ...,  6.2327685 ,
              6.338357  ,  6.414042  ],
            [ 5.0026946 ,  5.1224046 ,  5.2439675 , ...,  6.4411664 ,
              6.506693  ,  6.557266  ],
    ...
            [-0.04734313, -0.01396269,  0.03920674, ...,  6.458807  ,
              6.2103534 ,  5.7188663 ],
            [-0.21999383, -0.18382919, -0.13356364, ...,  5.9795036 ,
              5.822601  ,  5.2540236 ],
            [-0.36848485, -0.3394752 , -0.28651536, ...,  4.9319844 ,
              5.074786  ,  4.9961705 ]],
    
           [[-1.8712993 , -1.9839368 , -2.082491  , ...,  6.008571  ,
              6.0740967 ,  6.1163483 ],
            [-2.4660199 , -2.5048742 , -2.567129  , ...,  6.2650356 ,
              6.3314967 ,  6.3739033 ],
            [-3.1323118 , -3.16992   , -3.206344  , ...,  6.3136115 ,
              6.4002934 ,  6.454697  ],
            ...,
            [ 3.9259107 ,  3.8371398 ,  3.6467314 , ...,  6.601328  ,
              6.3775167 ,  5.851875  ],
            [ 3.7080822 ,  3.5044622 ,  3.2909021 , ...,  6.0302253 ,
              5.93541   ,  5.3589797 ],
            [ 3.422266  ,  3.1731856 ,  2.9543295 , ...,  4.991529  ,
              5.1275043 ,  5.003586  ]]], dtype=float32)

  • 10m_v_component_of_wind
    (time, lat, lon)
    float32
    3.221 3.105 2.99 ... -2.724 -2.549

    array([[[ 3.220673  ,  3.1045206 ,  2.9900334 , ...,  4.4796734 ,
              4.448715  ,  4.4620857 ],
            [ 3.0731454 ,  2.9620154 ,  2.8499417 , ...,  4.866217  ,
              4.8348694 ,  4.8484073 ],
            [ 2.9483118 ,  2.8360436 ,  2.7237751 , ...,  5.1571097 ,
              5.126428  ,  5.113113  ],
            ...,
            [-2.5558183 , -2.6723032 , -2.6935253 , ..., -2.5072443 ,
             -3.0638678 , -3.0492764 ],
            [-2.9305723 , -2.9996479 , -2.9810889 , ..., -4.2737947 ,
             -4.0541687 , -3.4880571 ],
            [-3.1682022 , -3.2081497 , -3.183488  , ..., -4.0444307 ,
             -3.354539  , -2.8132    ]],
    
           [[ 2.2868142 ,  2.1844525 ,  2.0852168 , ..., -2.4881196 ,
             -2.2478147 , -1.9747777 ],
            [ 2.5355628 ,  2.4719636 ,  2.380897  , ..., -1.9562466 ,
             -1.7128439 , -1.4365996 ],
            [ 2.7972224 ,  2.7352412 ,  2.672766  , ..., -1.4653838 ,
             -1.2297935 , -0.99110603],
    ...
            [-4.0208755 , -4.369472  , -4.767941  , ..., -1.9468218 ,
             -2.4218333 , -2.4403305 ],
            [-4.1855593 , -4.529236  , -4.913369  , ..., -3.0480425 ,
             -2.9349625 , -2.670683  ],
            [-4.291382  , -4.646599  , -5.014392  , ..., -2.6781547 ,
             -2.286337  , -2.0530987 ]],
    
           [[ 0.32318056,  0.3375756 ,  0.34683442, ..., -4.727282  ,
             -4.692105  , -4.587792  ],
            [-0.09961464, -0.14462443, -0.15188955, ..., -4.574208  ,
             -4.5385246 , -4.4330277 ],
            [-0.436445  , -0.48490152, -0.5326823 , ..., -4.396534  ,
             -4.3908906 , -4.318645  ],
            ...,
            [ 0.9670037 ,  0.9133432 ,  0.711948  , ..., -1.8439826 ,
             -2.3329408 , -2.4252243 ],
            [ 0.83704305,  0.744962  ,  0.49436653, ..., -3.0794895 ,
             -3.0097103 , -2.8271368 ],
            [ 0.5306583 ,  0.39211452,  0.10445043, ..., -3.0948305 ,
             -2.7242765 , -2.5489004 ]]], dtype=float32)

• Indexes: (3)
  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2015-06-25', '2015-06-26', '2015-06-27', '2015-06-28',
                   '2015-06-29', '2015-06-30'],
                  dtype='datetime64[ns]', name='time', freq='D'))

  • lat
    PandasIndex

    PandasIndex(Index([ 70.0, 69.75,  69.5, 69.25,  69.0, 68.75,  68.5, 68.25,  68.0, 67.75,
           ...
           37.25,  37.0, 36.75,  36.5, 36.25,  36.0, 35.75,  35.5, 35.25,  35.0],
          dtype='float32', name='lat', length=141))

  • lon
    PandasIndex

    PandasIndex(Index([ -15.0, -14.75,  -14.5, -14.25,  -14.0, -13.75,  -13.5, -13.25,  -13.0,
           -12.75,
           ...
            37.75,   38.0,  38.25,   38.5,  38.75,   39.0,  39.25,   39.5,  39.75,
             40.0],
          dtype='float32', name='lon', length=221))

• Attributes: (0)

station_df_unnormalised = data_processor.unnormalise(station_df)
pd.testing.assert_frame_equal(station_raw_df, station_df_unnormalised)
station_df_unnormalised

				PRCP	TAVG
time	lat	lon	station
2015-06-25	35.017	-1.450	AGM00060531	0.0	23.0
	35.100	-1.850	AGE00147716	0.0	23.4
	35.117	36.750	SYM00040030	NaN	25.4
	35.167	2.317	AGM00060514	0.0	25.9
	35.200	-0.617	AGM00060520	0.0	24.9
...	...	...	...	...	...
2015-06-30	45.933	7.700	ITM00016052	NaN	5.7
	38.367	-0.500	SPM00008359	0.0	27.6
	55.383	36.700	RSM00027611	0.0	17.2
	59.080	17.860	SWE00138750	0.0	NaN
	63.760	12.430	SWE00140158	0.6	NaN

16922 rows × 2 columns

Notice how the units of the unnormalised data are the same as the original data.

This functionality will be used under the hood later to map model predictions back to the original units!

Saving and loading a DataProcessor

The DataProcessor configuration can be saved and loaded to avoid re-computing the normalisation parameters in new sessions.

This is done using the save method to save the DataProcessor configuration to a folder, and then instantiating a new DataProcessor with the same folder will recover the same DataProcessor object.

data_processor.save("../deepsensor_config/")
data_processor2 = DataProcessor("../deepsensor_config/")
print(data_processor2)

DataProcessor with normalisation params:
{'10m_u_component_of_wind': {'method': 'mean_std',
                             'params': {'mean': 0.644899845123291,
                                        'std': 2.8509252071380615}},
 '10m_v_component_of_wind': {'method': 'mean_std',
                             'params': {'mean': -0.19969606399536133,
                                        'std': 3.2448606491088867}},
 '2m_temperature': {'method': 'mean_std',
                    'params': {'mean': 289.39849853515625,
                               'std': 5.538551330566406}},
 'GLDAS_mask': {'method': 'min_max', 'params': {'max': 1.0, 'min': 0.0}},
 'PRCP': {'method': 'mean_std',
          'params': {'mean': 1.1604694953552597, 'std': 4.172541647864038}},
 'TAVG': {'method': 'mean_std',
          'params': {'mean': 19.0613726868119, 'std': 5.188503809362459}},
 'coords': {'time': {'name': 'time'},
            'x1': {'map': (35.0, 90.0), 'name': 'lat'},
            'x2': {'map': (-15.0, 40.0), 'name': 'lon'}},
 'elevation': {'method': 'min_max',
               'params': {'max': 4504.4375, 'min': -185.125}},
 'tpi': {'method': 'min_max', 'params': {'max': 88.9609375, 'min': -73.671875}}}

Computing normalisation parameters over a subset of the data

Want to compute normalisation parameters only over a training period? No problem: just slice the data before passing it to the DataProcessor.

_ = data_processor(era5_raw_ds.sel(time=slice("2015-06-25", "2015-06-27")))
era5_ds = data_processor(era5_raw_ds)  # Will use the normalisation parameters computed above when called on the full dataset