data_sources.satellite
Satellite data sources and functions
data_sources.satellite.satellite_model
Model for output of satellite data
Satellite Objects
class Satellite(DataSourceOutput)
Model for output of satellite data
x_coordinates_shape
@validator("sat_x_coords")
def x_coordinates_shape(cls, v, values)
Validate 'sat_x_coords'
y_coordinates_shape
@validator("sat_y_coords")
def y_coordinates_shape(cls, v, values)
Validate 'sat_y_coords'
fake
@staticmethod
def fake(batch_size=32, seq_length_5=19, satellite_image_size_pixels=64, number_sat_channels=7, time_5=None)
Create fake data
get_datetime_index
def get_datetime_index() -> Array
Get the datetime index of this data
to_xr_dataset
def to_xr_dataset(i)
Make a xr dataset
from_xr_dataset
@staticmethod
def from_xr_dataset(xr_dataset)
Change xr dataset to model. If data does not exist, then return None
data_sources.satellite.satellite_data_source
Satellite Data Source
SatelliteDataSource Objects
@dataclass
class SatelliteDataSource(ZarrDataSource)
Satellite Data Source
filename: Must start with 'gs://' if on GCP.
__post_init__
def __post_init__(image_size_pixels: int, meters_per_pixel: int)
Post Init
open
def open() -> None
Open Satellite data
We don't want to open_sat_data in init. If we did that, then we couldn't copy SatelliteDataSource instances into separate processes. Instead, call open() after creating separate processes.
get_batch
def get_batch(t0_datetimes: pd.DatetimeIndex, x_locations: Iterable[Number], y_locations: Iterable[Number]) -> Satellite
Get batch data
Load the first _n_timesteps_per_batch concurrently. This loads the timesteps from disk concurrently, and fills the cache. If we try loading all examples concurrently, then SatelliteDataSource will try reading from empty caches, and things are much slower!
Arguments:
t0_datetimes
- list of timestamps for the datetime of the batches. The batch will also include data for historic and future depending on 'history_minutes' and 'future_minutes'.x_locations
- x center batch locations-
y_locations
- y center batch locations -
Returns
- Batch data
datetime_index
def datetime_index(remove_night: bool = True) -> pd.DatetimeIndex
Returns a complete list of all available datetimes
Arguments:
remove_night
- If True then remove datetimes a night.
open_sat_data
def open_sat_data(filename: str, consolidated: bool) -> xr.DataArray
Lazily opens the Zarr store.
Adds 1 minute to the 'time' coordinates, so the timestamps are at 00, 05, ..., 55 past the hour.
Arguments:
filename
- Cloud URL or local path. If GCP URL, must start with 'gs://'consolidated
- Whether or not the Zarr metadata is consolidated.