data_sources.satellite
Satellite data sources and functions
data_sources.satellite.satellite_model
Model for output of satellite data
Satellite Objects
class Satellite(DataSourceOutput)
Class to store satellite data as a xr.Dataset with some validation
model_validation
@classmethod
def model_validation(cls, v)
Check that all values are non negative
data_sources.satellite.satellite_data_source
Satellite Data Source
SatelliteDataSource Objects
@dataclass
class SatelliteDataSource(ZarrDataSource)
Satellite Data Source
filename: Must start with 'gs://' if on GCP.
__post_init__
def __post_init__(image_size_pixels: int, meters_per_pixel: int)
Post Init
open
def open() -> None
Open Satellite data
We don't want to open_sat_data in init. If we did that, then we couldn't copy SatelliteDataSource instances into separate processes. Instead, call open() after creating separate processes.
get_batch
def get_batch(t0_datetimes: pd.DatetimeIndex, x_locations: Iterable[Number], y_locations: Iterable[Number]) -> Satellite
Get batch data
Load the first _n_timesteps_per_batch concurrently. This loads the timesteps from disk concurrently, and fills the cache. If we try loading all examples concurrently, then SatelliteDataSource will try reading from empty caches, and things are much slower!
Arguments:
t0_datetimes
- list of timestamps for the datetime of the batches. The batch will also include data for historic and future depending on 'history_minutes' and 'future_minutes'.x_locations
- x center batch locations-
y_locations
- y center batch locations -
Returns
- Batch data
datetime_index
def datetime_index(remove_night: bool = True) -> pd.DatetimeIndex
Returns a complete list of all available datetimes
Arguments:
remove_night
- If True then remove datetimes a night.
open_sat_data
def open_sat_data(filename: str, consolidated: bool) -> xr.DataArray
Lazily opens the Zarr store.
Adds 1 minute to the 'time' coordinates, so the timestamps are at 00, 05, ..., 55 past the hour.
Arguments:
filename
- Cloud URL or local path. If GCP URL, must start with 'gs://'consolidated
- Whether or not the Zarr metadata is consolidated.