DownloadDatasets package

Submodules

LOGS_solutions.DownloadDatasets.DatasetDownloader module

class LOGS_solutions.DownloadDatasets.DatasetDownloader.DatasetDownloader(logs, args)

Bases: object

Downloads all datasets

can_create_symlink()

Check if the system supports creating symbolic links. :rtype: bool :return: True if the system supports creating symbolic links, False otherwise.

download_datasets(project_path, sample_path, project, dataset)

Parse and download datasets. Create symlinks if symlink_path is not None. Create metadata files if include_metadata is True.

Parameters:

• project_path (str) – Path where the project should be saved.

• sample_path (str) – Path where the sample should be saved.

• project (Project) – Project object

• dataset (Dataset) – Dataset object

download_datasets_structured()

Downloads all datasets structured in the given path.

valid_start_from()

Check if last_state.json exists and return the dateAdded if it exists, else return None.

Return type:

Optional[datetime]

Returns:

datetime object of the last dateAdded or None

LOGS_solutions.DownloadDatasets.DatasetHandler module

class LOGS_solutions.DownloadDatasets.DatasetHandler.DatasetHandler(dataset_target_path, dataset, include_metadata=False, duplicate_handling=1, symlink_path=None, original_target_path=None)

Bases: object

The class handles the parsing and downloading of unpacked data records in separate folders.

create_symlink()

Create a symlink to the dataset in the format folder.

download_dataset(filename='')

Downloads the dataset and stores it in the given path.

Parameters:

filename (str) – Name of the store dataset. If empty, the name of the dataset will be used.

download_dataset_unzip(filename='')

Download the unzipped dataset.

Parameters:

filename (str) – Name of the store dataset. If empty, the name of the dataset will be used.

parse_dataset()

Download all files of the dataset and store it in the given path together with a txt file with all important information.

LOGS_solutions.DownloadDatasets.FileHandler module

class LOGS_solutions.DownloadDatasets.FileHandler.FileHandler

Bases: object

static clean_filename(name)

Prepare the name as a file name. Remove unwanted special characters and replace a space with _.

Parameters:

name (str) – The name to be used as the file name.

Return type:

str

Returns:

The prepared filename.

clean_foldername()

Prepare the name as a folder name. Remove unwanted special characters and replace a space with _.

Parameters:

name (str) – The name to be used as the folder name.

Return type:

str

Returns:

The prepared foldername.

LOGS_solutions.DownloadDatasets.ProgressBar module

class LOGS_solutions.DownloadDatasets.ProgressBar.ProgressBar

Bases: object

classmethod progress_bar(message)

Creates a progress bar for the console.

Parameters:

message (str) – The message to display in front of the progress bar.

classmethod start_progressbar(message='')

Starts the progress bar.

Parameters:

message (str) – The message to display in front of the progress bar.

classmethod stop_progressbar()

Stops the progress bar.

classmethod update_processed_files()

Updates the number of processed files.

LOGS_solutions.DownloadDatasets.main module

LOGS_solutions.DownloadDatasets.main.main(args)

Main function to download datasets from a LOGS instance. Important: This classes uses absolute paths. To obtain the correct absolute path, you must ensure that the relative path is correct in relation to the directory in which the script is executed. If you are not sure, use the absolute path for target_path

LOGS_solutions.DownloadDatasets.main.parse_args()

Parses the command line arguments.

Returns:

The parsed arguments.

LOGS_solutions.DownloadDatasets.main.valid_bool(bool_string)

Validates the given boolean string. The string can be ‘true’, ‘yes’, ‘1’, ‘false’, ‘no’, or ‘0’.

Parameters:

bool_string (str) – The boolean string to validate.

Return type:

bool

Returns:

The boolean value of the string.

LOGS_solutions.DownloadDatasets.main.valid_duplicate_handling(dataset_handling)

Validates how datasets with the same name should be handled. Options: 1: rename, 2: overwrite, 3: take first.

Parameters:

duplicate_handling – The duplicate handling to validate.

Return type:

int

Returns:

The duplicate handling as an integer.

LOGS_solutions.DownloadDatasets.main.valid_ids(project_ids)

Validates the project ids

Parameters:

id – ids

Return type:

List[int]

Returns:

[] if project_ids is None or an empty list, otherwise a list of integers

LOGS_solutions.DownloadDatasets.main.valid_path(target_path)

Validates the given paths.

Parameters:

target_path – The target path to validate.

Return type:

Path

Returns:

The absolute path for the target.

Module contents