DownloadDatasets package

Submodules

LOGS_solutions.DownloadDatasets.DatasetDownloader module

class LOGS_solutions.DownloadDatasets.DatasetDownloader.DatasetDownloader(logs, args)

Bases: object

Downloads all datasets, organized as follows: 1) Claimed/Unclaimed 2) Projects 3) Samples 4) Datasets

can_create_symlink()

Check if the system supports creating symbolic links.

Return type:

bool

Returns:

True if the system supports creating symbolic links, False otherwise.

download_dataset(project_path, sample_path, project, dataset, sample)

Parse and download dataset. Create symlinks if symlink_path is not None. Create metadata files if include_metadata is True.

Parameters:

• project_path (str) – Path where the project should be saved.

• sample_path (str) – Path where the sample should be saved.

• project (ProjectMinimal) – Project object

• dataset (Dataset) – Dataset object

• sample (Sample) – Sample object

download_datasets_structured()

Downloads all datasets structured in the given path.

download_datasets_structured_helper(dataset, status, sample=None, count=0)

Helper function to call download_datasets_structured with exception handling. :type dataset: Dataset :param dataset: Dataset object :type status: str :param status: “Claimed” or “Unclaimed” :type sample: Optional[str] :param sample: The sample name of the dataset as a string. If None, it will be set to “Not_mentioned_sample”. :type count: int :param count: The number of samples that have been processed for the current dataset. Used to determine the sample name if the sample parameter is None.

valid_start_from()

Check if last_state.json exists and return the dateAdded if it exists, else return None.

Return type:

Optional[datetime]

Returns:

datetime object of the last dateAdded or None

LOGS_solutions.DownloadDatasets.DatasetHandler module

class LOGS_solutions.DownloadDatasets.DatasetHandler.DatasetHandler(dataset_target_path, dataset, include_metadata=False, duplicate_handling=1, symlink_path=None, original_target_path=None)

Bases: object

The class handles the parsing and downloading of unpacked data records in separate folders.

create_symlink()

Create a symlink to the dataset in the format folder.

download_dataset_unzip()

Download the unzipped dataset.

Parameters:

filename – Name of the store dataset. If empty, the name of the dataset will be used.

parse_dataset()

Download all files of the dataset and store it in the given path together with a txt file with all important information.

LOGS_solutions.DownloadDatasets.main module

LOGS_solutions.DownloadDatasets.main.main(args)

Main function to download datasets from a LOGS instance.

Important: This classes uses absolute paths. To obtain the correct absolute path, you must ensure that the relative path is correct in relation to the directory in which the script is executed. If you are not sure, use the absolute path for target_path

LOGS_solutions.DownloadDatasets.main.parse_args()

Parses the command line arguments.

Returns:

The parsed arguments.

LOGS_solutions.DownloadDatasets.main.valid_bool(bool_string)

Validates the given boolean string. The string can be ‘true’, ‘yes’, ‘1’, ‘false’, ‘no’, or ‘0’.

Parameters:

bool_string (str) – The boolean string to validate.

Return type:

bool

Returns:

The boolean value of the string.

LOGS_solutions.DownloadDatasets.main.valid_duplicate_handling(dataset_handling)

Validates how datasets with the same name should be handled. Options: 1: rename, 2: overwrite, 3: take first.

Parameters:

duplicate_handling – The duplicate handling to validate.

Return type:

int

Returns:

The duplicate handling as an integer.

LOGS_solutions.DownloadDatasets.main.valid_ids(project_ids)

Validates the project ids.

Parameters:

id – ids

Return type:

List[int]

Returns:

[] if project_ids is None or an empty list, otherwise a list of integers

LOGS_solutions.DownloadDatasets.main.valid_path(target_path)

Validates the given paths.

Parameters:

target_path – The target path to validate.

Return type:

Path

Returns:

The absolute path for the target.

LOGS_solutions.DownloadDatasets.main.valid_sample_names(sample_names)

Validates the sample names.

Parameters:

sample_names (str) – The sample names to validate.

Return type:

List[str]

Returns:

A list of sample names.

Module contents