Directory Watcher
Application to watch a directory for changes and send to DLM.
- async ska_dlm_client.directory_watcher.main.amain()
Run the main application logic in an asyncio context.
Creates a DirectoryWatcher, sets up signal handlers for graceful shutdown, and starts the directory watching process.
- ska_dlm_client.directory_watcher.main.create_directory_watcher()
Create a DirectoryWatcher instance from the command line arguments.
Parses command line arguments, creates a Config object, and initializes a DirectoryWatcher (either INotifyDirectoryWatcher or PollingDirectoryWatcher depending on the arguments).
- Return type:
- Returns:
A DirectoryWatcher instance configured with the parsed command line arguments.
- ska_dlm_client.directory_watcher.main.main()
Run the application’s main entry point.
Creates a new asyncio event loop and runs the amain coroutine in it. This function is the entry point when the module is executed directly.
- ska_dlm_client.directory_watcher.main.process_args(args)
Collect all command line parameters and create a Config object.
- Return type:
- Args:
args: The parsed command line arguments from argparse.
- Returns:
A Config object initialized with all the command line parameters.
Class to perform directory watching tasks.
- class ska_dlm_client.directory_watcher.directory_watcher.DirectoryWatcher(config, registration_processor, cmd_line_parameters=None)
Class for the running of the directory_watcher.
- class ska_dlm_client.directory_watcher.directory_watcher.INotifyDirectoryWatcher(config, registration_processor, cmd_line_parameters=None)
Directory watcher using INotify filesystem events.
- async watch()
Watch for changes in the defined directory and process each change found.
- class ska_dlm_client.directory_watcher.directory_watcher.LStatPollingEmitter(event_queue, watch, timeout=1.0, event_filter=None)
Emitter that polls a directory to detect filesystem changes using os.lstat.
NOTE: As of December 2024 a fix for this seems to be in the works but not yet available.
- class ska_dlm_client.directory_watcher.directory_watcher.PollingDirectoryWatcher(config, registration_processor, cmd_line_parameters=None)
DirectoryWatcher using filesystem polling.
- async watch()
Watch for changes in the defined directory and process each change found.
Utilities for monitoring directories and detecting changes.
- ska_dlm_client.directory_watcher.directory_utils.log_changes_with_diff(old_snapshot, new_snapshot)
Log the changes between two directory snapshots using DirectorySnapshotDiff.
This function compares two directory snapshots and logs detailed information about all detected changes, including: - Created files (with size and modification time) - Created directories (with modification time) - Modified files (with old/new size and modification time) - Modified directories (with old/new modification time) - Deleted files - Deleted directories - Moved files - Moved directories
The function uses DirectorySnapshotDiff to efficiently detect changes between the snapshots and handles exceptions that might occur when trying to access file information (such as FileNotFoundError or PermissionError).
- Return type:
- Args:
- old_snapshot: The previous directory snapshot containing information about
files and directories at an earlier point in time
- new_snapshot: The current directory snapshot containing information about
files and directories at the current point in time
- ska_dlm_client.directory_watcher.directory_utils.monitor_directory_with_watchdog(directory, wait_time=5.0, recursive=True)
Monitor a directory using watchdog.observers.polling until no changes are detected.
This implementation uses watchdog’s DirectorySnapshot and DirectorySnapshotDiff to efficiently detect changes.
- Return type:
DirectorySnapshot
- Args:
directory: The directory to monitor wait_time: The time to wait between scans (in seconds) recursive: Whether to scan subdirectories recursively
- Returns:
The final directory snapshot
Look after the entries and the cached file for them.
- class ska_dlm_client.directory_watcher.directory_watcher_entries.DataclassJSONEncoder(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)
JSON encoder for dataclasses.
- default(o)
Override JSON default function.
- class ska_dlm_client.directory_watcher.directory_watcher_entries.DirectoryWatcherEntries(entries_file='/tmp/dlm_entires.json', reload_from_status_file=False, write_directory_entries_file=True)
The list of entries to be managed.
- add(entry)
Override list append.
- read_from_file()
Load the list of DirectoryWatcherEntry from a file in JSON format.
- save_to_file()
Save the list of DirectoryWatcherEntry to a file in JSON format.
- class ska_dlm_client.directory_watcher.directory_watcher_entries.DirectoryWatcherEntry(file_or_directory, dlm_storage_name, dlm_registration_id, time_registered)
An individual entry.
Class to process watcher directory change events.
- class ska_dlm_client.directory_watcher.watcher_event_handler.WatcherEventHandler(config, registration_processor)
Either log or take action for all the possible events.
Originally based on LoggingEventHandler.
- on_closed_no_write(event)
Take action when a closed with no write event is captured.
- Return type:
Class to hold the configuration used by the directory_watcher package.
- class ska_dlm_client.directory_watcher.config.WatcherArgs(source_name='', directory_to_watch='', target_name='', storage_url='', ingest_url='', request_url='', migration_url='', readiness_probe_file='', use_status_file=False, reload_status_file=True, parser=<factory>)
Adding the additional specific command line arguments for the directory_watcher.
- class ska_dlm_client.directory_watcher.config.WatcherConfig(source_name='dir-watcher', target_name='dlm-archive', ingest_url='http://dlm_ingest:8001', storage_url='http://dlm_storage:8003', migration_url='http://dlm_migration:8004', reload_status_file=True, include_existing=False, rclone_access_check_on_register=False, METADATA_FILENAME='ska-data-product.yaml', METADATA_EXECUTION_BLOCK_KEY='execution_block', status_file_filename='.directory_watcher_status.run', DIRECTORY_IS_MEASUREMENT_SET_SUFFIX='.ms', directory_to_watch='/dlm/watch_dir', use_status_file=False)
Configuration parameters for the directory watcher.