filesystem
Cloud functions
filesystem.utils
General utils functions
upload_and_delete_local_files
def upload_and_delete_local_files(dst_path: Union[str, Path],
local_path: Union[str, Path])
Upload an entire folder and delete local files to either AWS or GCP
get_maximum_batch_id
def get_maximum_batch_id(path: Pathy) -> int
Get the last batch ID. Works with GCS, AWS, and local.
Arguments:
-
path
- The path folder to look in. Begin with 'gs://' for GCS. Begin with 's3://' for AWS S3. Supports wildcards , *, ?, and [..]. -
Returns
- The maximum batch id of data inpath
. Ifpath
exists but contains no files then returns -1.
Raises FileNotFoundError if path
does not exist.
delete_all_files_in_temp_path
def delete_all_files_in_temp_path(path: Union[Path, str],
delete_dirs: bool = False)
Delete all the files in a temporary path. Option to delete the folders or not
check_path_exists
def check_path_exists(path: Union[str, Path])
Raises a FileNotFoundError if path
does not exist.
path
can include wildcards.
rename_file
def rename_file(remote_file: str, new_filename: str)
Rename file within one filesystem
Arguments:
remote_file
- The current file namenew_filename
- What the file should be renamed too
get_all_filenames_in_path
def get_all_filenames_in_path(path: Union[str, Path]) -> List[str]
Get all the files names from one folder.
Arguments:
-
path
- The path that we should look in. -
Returns
- A list of filenames represented as strings.
download_to_local
def download_to_local(remote_filename: str, local_filename: str)
Download file from gcs.
Arguments:
remote_filename
- the file name, should start with gs:// or s3://local_filename
- the local filename
upload_one_file
def upload_one_file(remote_filename: str,
local_filename: str,
overwrite: bool = True)
Upload one file to aws or gcp
Arguments:
remote_filename
- the aws/gcp key namelocal_filename
- the local file nameoverwrite
- overwrite file
makedirs
def makedirs(path: Union[str, Path], exist_ok: bool = True) -> None
Recursively make directories
Creates directory at path and any intervening required directories.
Raises exception if, for instance, the path already exists but is a file.
Arguments:
path
- The path to create.exist_ok
- If False then raise an exception ifpath
already exists.