config
Configuration of the dataset
config.load
Loading configuration functions
load_yaml_configuration
def load_yaml_configuration(filename: Union[str, Pathy]) -> Configuration
Load a yaml file which has a configuration in it
Arguments:
filename
- the file name that you want to load. Will load from local, AWS, or GCP depending on the protocol suffix (e.g. 's3://bucket/config.yaml').
Returns:pydantic class
load_configuration_from_gcs
def load_configuration_from_gcs(gcp_dir: str, bucket: str = "solar-pv-nowcasting-data", filename: str = "configuration.yaml") -> Configuration
Load configuration from gcs
gcp_dir: the directory where the configruation is saved bucket: the gcs bucket to load from filename: the filename that will be loaded
Returns: configuration class
config.save
Save functions for the configuration model
save_yaml_configuration
def save_yaml_configuration(configuration: Configuration, filename: Optional[Union[str, Pathy]] = None)
Save a local yaml file which has the a configuration in it.
If filename
is None then saves to configuration.output_data.filepath / configuration.yaml.
Will save to GCP, AWS, or local, depending on the protocol suffix of filepath.
config.model
Configuration model for the dataset
General Objects
class General(BaseModel)
General pydantic model
Git Objects
class Git(BaseModel)
Git model
InputData Objects
class InputData(BaseModel)
Input data model
All paths must include the protocol prefix. For local files, it's sufficient to just start with a '/'. For aws, start with 's3://', for gcp start with 'gs://'.
OutputData Objects
class OutputData(BaseModel)
Output data model
Process Objects
class Process(BaseModel)
Pydantic model of how the data is processed
seq_length_30_minutes
@property
def seq_length_30_minutes()
How many steps are there in 30 minute datasets
seq_length_5_minutes
@property
def seq_length_5_minutes()
How many steps are there in 5 minute datasets
history_minutes_divide_by_30
@validator("history_minutes")
def history_minutes_divide_by_30(cls, v)
Validate 'history_minutes'
forecast_minutes_divide_by_30
@validator("forecast_minutes")
def forecast_minutes_divide_by_30(cls, v)
Validate 'forecast_minutes'
Configuration Objects
class Configuration(BaseModel)
Configuration model for the dataset
set_base_path
def set_base_path(base_path: str)
Append base_path to all paths. Mostly used for testing.
set_git_commit
def set_git_commit(configuration: Configuration)
Set the git information in the configuration file
Arguments:
-
configuration
- configuration object -
Returns
- configuration object with git information