filesystem

Cloud functions

filesystem.utils

General utils functions

upload_and_delete_local_files

def upload_and_delete_local_files(dst_path: Union[str, Path],
                                  local_path: Union[str, Path])

Upload an entire folder and delete local files to either AWS or GCP

get_maximum_batch_id

def get_maximum_batch_id(path: Pathy) -> int

Get the last batch ID. Works with GCS, AWS, and local.

Arguments:

  • path - The path folder to look in. Begin with 'gs://' for GCS. Begin with 's3://' for AWS S3. Supports wildcards , *, ?, and [..].

  • Returns - The maximum batch id of data in path. If path exists but contains no files then returns -1.

Raises FileNotFoundError if path does not exist.

delete_all_files_in_temp_path

def delete_all_files_in_temp_path(path: Union[Path, str],
                                  delete_dirs: bool = False)

Delete all the files in a temporary path. Option to delete the folders or not

check_path_exists

def check_path_exists(path: Union[str, Path])

Raises a FileNotFoundError if path does not exist.

path can include wildcards.

rename_file

def rename_file(remote_file: str, new_filename: str)

Rename file within one filesystem

Arguments:

  • remote_file - The current file name
  • new_filename - What the file should be renamed too

get_all_filenames_in_path

def get_all_filenames_in_path(path: Union[str, Path]) -> List[str]

Get all the files names from one folder.

Arguments:

  • path - The path that we should look in.

  • Returns - A list of filenames represented as strings.

download_to_local

def download_to_local(remote_filename: str, local_filename: str)

Download file from gcs.

Arguments:

  • remote_filename - the file name, should start with gs:// or s3://
  • local_filename - the local filename

upload_one_file

def upload_one_file(remote_filename: str,
                    local_filename: str,
                    overwrite: bool = True)

Upload one file to aws or gcp

Arguments:

  • remote_filename - the aws/gcp key name
  • local_filename - the local file name
  • overwrite - overwrite file

makedirs

def makedirs(path: Union[str, Path], exist_ok: bool = True) -> None

Recursively make directories

Creates directory at path and any intervening required directories.

Raises exception if, for instance, the path already exists but is a file.

Arguments:

  • path - The path to create.
  • exist_ok - If False then raise an exception if path already exists.