Configuration Schema

This is the schema of the execution block (EB) and processing block (PB) in the configuration database, effectively the control plane of the SDP. Both are described in the configuration library using a Pydantic entity model, which sets up the schema for the top level static parameters. Note that both EB and PB entities are accessed using a key, which corresponds to the EB or PB ID.

Execution Block

The EB parameters are set up by the AssignResources command called on the subarray.

Path in configuration database: /eb/[eb_id]

Contents:

{
    "key": "eb-mvp01-20200425-00000",
    "beams": [
        { "beam_id": "vis0", "function": "visibilities", ... }
    ],
    "channels": [
        {
            "channels_id": "vis_channels",
            "spectral_windows": [
                {
                    "spectral_window_id": "fsp_1_channels",
                    "count": 744, "start": 0, "stride": 2,
                    "freq_min": 350000000.0, "freq_max": 368000000.0,
                    "link_map": [[0, 0], [200, 1], [744, 2], [944, 3]]
                }
            ]
        }
    ],
    "context": {},
    "fields": [
        {
            "field_id": "field_a",
            "phase_dir": {
                "ra": [123.0, 0.1],
                "dec": [-60.0, 0.1],
                "reference_time": "...",
                "reference_frame": "ICRF3"
            }
        }
    ],
    "max_length": 21600.0,
    "pb_batch": [ ... ],
    "pb_realtime": [ "pb-mvp01-20200425-00000", ... ],
    "polarisations": [
        {
            "polarisations_id": "all",
            "corr_type": ["XX", "XY", "YY", "YX"]
        }
    ],
    "scan_types": [
        { "scan_type_id": "science", ... },
        { "scan_type_id": "calibration", ... }
    ],
    "subarray_id": None
}

Execution Block State

Path in configuration database: /eb/[eb_id]/state

Dynamic state information of the execution block, consisting of current (and executed) scan parameters and the overall status of the EB.

Contents:

{
    "scan_type": "science",
    "scan_id": 12345,
    "scans": [
        {"scan_id": 12343, "scan_type": "science", "status": "FINISHED"},
        {"scan_id": 12344, "scan_type": "science", "status": "FINISHED"},
    ],
    "status": "ACTIVE",
}

When the execution block is being executed, the status field is set to ACTIVE, and when ended, it is either set to CANCELLED or FINISHED. The scan status can either be set to ABORTED or FINISHED.

Processing Block

Path in configuration database: /pb/[pb_id]

Static definition of processing block information.

Contents:

{
    "key": "pb-mvp01-20200425-00000",
    "eb_id": "eb-mvp01-20200425-00000",
    "script": {
        "kind": "realtime",
        "name": "vis_receive",
        "version": "0.2.0"
    },
    "parameters": { ... }
}

There are two kinds of processing, real-time and batch (offline). Real-time processing starts immediately, as it directly corresponds to an observation that is about to start. Batch processing will be inserted into a scheduling queue managed by the SDP, where it will typically be executed according to resource availability.

Valid kinds are realtime and batch. The script tag identifies the processing script version as well as the required underlying software (e.g. execution engines, processing components). ... stands for arbitrary processing script-defined parameters.

Processing Block State

Path in configuration database: /pb/[pb_id]/state

Dynamic state information of the processing block. If it does not exist, the processing block is still starting up.

Contents:

{
    "resources_available": True,
    "status": "RUNNING",
    "receive_addresses": [
        { "scan_type_id": "science", ... },
        { "scan_type_id": "calibration", ... },
    ],
    "last_updated": "2022-08-01 10:01:12"
}

Tracks the current state of the processing block. This covers both the SDP-internal state (as defined by the Execution Control Data Model) as well as information to publish via Tango for real-time processing, such as the status and receive addresses (for ingest).

Processing Block Owner

Path: /pb/[pb_id]/owner

Identifies the process executing the script. Used for leader election/lock as well as a debugging aid.

Contents:

{
"command": [
    "vis_receive.py",
    "pb-mvp01-20200425-00000"
],
"hostname": "pb-mvp01-20200425-00000-script-2kxfz",
"pid": 1
}