Top-level modules

This module is called when the dataproduct is an unidentified filetype.

ska_dlm_client.minimal_metadata_generator.minimal_metadata_generator(dataproduct_path)

Given the dataproduct_path, file or directory, generate a valid metadata dict.

This function is intended to build a minimal metadata dict from a single file or from a directory of files. The caller is likely calling this as the real file representing the data product metadata could not be found.

No metadata file will be physically created it is just returned as a metadata dict.

Return type:

dict

The fields of the metadata that are filled include:

• execution_block_id based on date and time for uniqueness but readability

• data.files NOTE: a file entry will be created for each file in a directory

  • crc

  • description

  • path of a file

  • size in bytes

  • status current hard coded to “done”

• data.obscore.access_estsize of file or contents of directory

Some internal exceptions are caught and logged with some “defaults” being put in their place so that a valid metadata can always be generated. This will likely need refinement.

Raises

ValueError

When the dataproduct_path is not a file or directory. Not expected during normal operation but problems in this area are being looked at in https://jira.skatelescope.org/browse/YAN-1893

OSError

https://docs.python.org/3/library/exceptions.html#OSError

Constants used throughout the DLM package.

class ska_dlm_client.common_types.ItemType(value)

Data Item on the filesystem.

__format__(format_spec)

Returns format using actual value type unless __str__ has been overridden.

__new__(value)

__repr__()

Return repr(self).

__str__()

Return str(self).

class ska_dlm_client.common_types.LocationCountry(value)

Location country.

__format__(format_spec)

Returns format using actual value type unless __str__ has been overridden.

__new__(value)

__repr__()

Return repr(self).

__str__()

Return str(self).

class ska_dlm_client.common_types.LocationName(value)

Location names.

__format__(format_spec)

Returns format using actual value type unless __str__ has been overridden.

__new__(value)

__repr__()

Return repr(self).

__str__()

Return str(self).

class ska_dlm_client.common_types.LocationType(value)

Location type.

__format__(format_spec)

Returns format using actual value type unless __str__ has been overridden.

__new__(value)

__repr__()

Return repr(self).

__str__()

Return str(self).

class ska_dlm_client.common_types.StorageInterface(value)

Storage interface.

__format__(format_spec)

Returns format using actual value type unless __str__ has been overridden.

__new__(value)

__repr__()

Return repr(self).

__str__()

Return str(self).

class ska_dlm_client.common_types.StorageType(value)

Storage type.

__format__(format_spec)

Returns format using actual value type unless __str__ has been overridden.

__new__(value)

__repr__()

Return repr(self).

__str__()

Return str(self).

Module-level constants used by the DLM client.

class ska_dlm_client.config.ClientConfig(source_name='', target_name='dlm-archive', ingest_url='http://dlm_ingest:8001', storage_url='http://dlm_storage:8003', migration_url='http://dlm_migration:8004', reload_status_file=True, include_existing=False, rclone_access_check_on_register=False, METADATA_FILENAME='ska-data-product.yaml', METADATA_EXECUTION_BLOCK_KEY='execution_block', status_file_filename='.directory_watcher_status.run', DIRECTORY_IS_MEASUREMENT_SET_SUFFIX='.ms')

General configuration class of the SKA DLM clients.

This class holds all configuration parameters needed for the DLM clients. Specific configurations for the individual clients are expanding this class.

__eq__(other)

Return self==value.

__hash__ = None

__init__(source_name='', target_name='dlm-archive', ingest_url='http://dlm_ingest:8001', storage_url='http://dlm_storage:8003', migration_url='http://dlm_migration:8004', reload_status_file=True, include_existing=False, rclone_access_check_on_register=False, METADATA_FILENAME='ska-data-product.yaml', METADATA_EXECUTION_BLOCK_KEY='execution_block', status_file_filename='.directory_watcher_status.run', DIRECTORY_IS_MEASUREMENT_SET_SUFFIX='.ms')

__repr__()

Return repr(self).

__weakref__

list of weak references to the object (if defined)

class ska_dlm_client.config.CmdLineParameters(source_name='', directory_to_watch='', target_name='', storage_url='', ingest_url='', request_url='', migration_url='', readiness_probe_file='', use_status_file=False, reload_status_file=True, parser=<factory>)

Class for common/required command line parameters and helper methods.

__default_args__()

Initialise parameters to include on the command line.

__eq__(other)

Return self==value.

__exit__(exc_type, exc_val, exc_tb)

Delete the readiness file on deletion of the instance.

__hash__ = None

__init__(source_name='', directory_to_watch='', target_name='', storage_url='', ingest_url='', request_url='', migration_url='', readiness_probe_file='', use_status_file=False, reload_status_file=True, parser=<factory>)

__repr__()

Return repr(self).

__weakref__

list of weak references to the object (if defined)

parse_arguments(args=None)

Parse command line arguments and assign to class parameters.

set_application_ready()

Create the required file to indicate the application is ready.

ska_dlm_client.config.Config

alias of ClientConfig