data_sources.nwp
NWP data sources and functions
data_sources.nwp.nwp_data_source
NWP Data Source
NWPDataSource Objects
@dataclass
class NWPDataSource(ZarrDataSource)
NWP Data Source (Numerical Weather Predictions)
Args (for init): filename: The base path in which we find '2018_1-6', etc.
Attributes:
_data
- xr.DataArray of Numerical Weather Predictions, opened by open(). x is left-to-right. y is top-to-bottom. Access using public nwp property.consolidated
- Whether or not the Zarr store is consolidated.channels
- The NWP forecast parameters to load. If None then don't filter. The available params are: t : Temperature in Kelvin. dswrf : Downward short-wave radiation flux in W/m^2 (irradiance). prate : Precipitation rate in kg/m^2/s. r : Relative humidty in %. sde : Snow depth in meters. si10 : 10-meter wind speed in m/s. vis : Visibility in meters. lcc : Low-level cloud cover in %. mcc : Medium-level cloud cover in %. hcc : High-level cloud cover in %.
__post_init__
def __post_init__(image_size_pixels: int, meters_per_pixel: int)
Post init
Arguments:
image_size_pixels
- number of pixels in imagemeters_per_pixel
- how many meteres for each pixel
open
def open() -> None
Open NWP data
We don't want to open_sat_data in init. If we did that, then we couldn't copy NWPDataSource instances into separate processes. Instead, call open() after creating separate processes.
get_batch
def get_batch(t0_datetimes: pd.DatetimeIndex, x_locations: Iterable[Number], y_locations: Iterable[Number]) -> NWP
Get batch data
Arguments:
t0_datetimes
- list of timstampsx_locations
- list of x locations, where the batch data is for-
y_locations
- list of y locations, where the batch data is for -
Returns
- batch data
datetime_index
def datetime_index() -> pd.DatetimeIndex
Returns a complete list of all available datetimes
open_nwp
def open_nwp(filename: str, consolidated: bool) -> xr.Dataset
Open The NWP data
Arguments:
filename
- filename must start with 'gs://' if it's on GCP.-
consolidated
- consolidate the zarr file? -
Returns
- nwp data
data_sources.nwp.nwp_model
Model for output of NWP data
NWP Objects
class NWP(DataSourceOutput)
Class to store NWP data as a xr.Dataset with some validation
model_validation
@classmethod
def model_validation(cls, v)
Check that all values are not NaNs
NWPML Objects
class NWPML(DataSourceOutputML)
Model for output of NWP data
fake
@staticmethod
def fake(batch_size=32, seq_length_5=19, image_size_pixels=64, number_nwp_channels=7, time_5=None)
Create fake data
get_datetime_index
def get_datetime_index() -> Array
Get the datetime index of this data
from_xr_dataset
@staticmethod
def from_xr_dataset(xr_dataset: xr.Dataset)
Change xr dataset to model with tensors