SDP Subarray Device =================== Introduction ------------ The SDP subarray Tango device is the principal means by which processing is initiated in the SDP. State Model ----------- The present implementation is shown in the diagram below. Here the state is the combination of the Tango device state and the observing state (obsState). .. image:: images/sdp-subarray-states.svg :align: center Behaviour --------- The interaction between the Telescope Monitoring and Control (TMC) subsystem and the SDP subarray device is shown below. The SDP subarray device receives commands from the TMC subarray device, and the consequent changes to the state of the SDP are reported in the device attributes. .. image:: images/sdp-subarray-interaction-tango.svg :align: center Schemas ------- The arguments of the subarray commands and some of its attributes are strings containing data in JSON format. The data are described by a schema which is versioned to support evolution of the interfaces. The schema is specified with the ``interface`` keyword in the JSON string: .. code-block:: json { "interface": "https://schema.skao.int/ska-sdp-/", "...": "..." } where ```` identifies the command or attribute: - ``assignres`` - AssignResources command - ``releaseres`` - ReleaseResources command - ``configure`` - Configure command - ``scan`` - Scan command - ``recvaddrs`` - receiveAddresses attribute and ```` is the version of the schema. The JSON string can be validated against the schema using the :doc:`Schemas library `. Its documentation describes the versions of the schemas. The subarray device implements version 2.0 of the schemas, which is used in the examples below. The commands will accept arguments using previous versions of the schemas (0.4, 0.5, 1.0, 1.1 and 1.2) for backwards compatibility. Commands -------- .. list-table:: Commands :header-rows: 1 * - Command - Argument type - Return type - Action * - On - None - None - Sets the device state to ON and the observing state to EMPTY. * - Off - None - None - Sets the device state to OFF. * - AssignResources - String (JSON) - None - :ref:`Assigns resources to the subarray `. * - ReleaseResources - String (JSON) - None - :ref:`Releases resources from the subarray `. * - ReleaseAllResources - None - None - Releases all resources assigned to the subarray. * - Configure - String (JSON) - None - :ref:`Configures scan type for the following scans `. * - Scan - String (JSON) - None - :ref:`Begins a scan of the configured type `. * - EndScan - None - None - Ends the scan. * - End - None - None - Ends the execution block. * - Abort - None - None - Aborts current activity. * - Restart - None - None - Restarts the subarray in the EMPTY observing state Transaction IDs ^^^^^^^^^^^^^^^ Commands that accept an argument will also accept an optional transaction ID. This is used in logging to enable tracing the execution of the commands. If the argument does not contain a transaction ID, then an internal one is generated for the command. This is also done for all commands that do not take an argument. Example of a transaction ID: .. code-block:: json { "transaction_id": "txn-test-20220921-00000000", "...": "..." } .. _subarray_assign_resources: AssignResources ^^^^^^^^^^^^^^^ The argument of the AssignResources command describes externally-managed resources, an execution block (EB) and processing blocks (PBs) to be assigned to the subarray. The EB contains the information that is necessary for the SDP to: - receive visibility data from the correlator beam-former (CBF) and write data to one or more Measurement Sets; - provide reference pointing calibration solutions to the TMC to correct the Dish pointing (SKA-Mid only); - provide calibration solutions to the CBF to form tied-array beams; - receive candidate and timing data from the pulsar search and timing subsystems. The EB defines the scan types that can be selected by the Configure command. .. warning:: If an execution block is in progress (i.e. an EB has been assigned to the subarray and it has not yet been ended), then the AssignResources command will cancel the execution block. The AssignResources command will be rejected if any of the following is true: - An execution block or processing block contains an invalid ID; - An execution block or processing block contains a duplicate ID; - A processing block contains a script not known to the SDP; - The version of processing script requested in any of the processing blocks is incompatible with the version of SDP that is deployed; - If the script contains a JSON schema for the processing block parameters, the parameters will be validated, and the validation returns an error; - The argument contains a processing block, and the Processing Controller or Helm Deployer is offline (which means that processing cannot be started). - For each processing block the required resources are computed. These are based on the resource definitions in the scripts resources. The resource of all of the processing blocks are totaled based on resource type. If any of these total amounts is greater than what is available for this resource type then the configuration is rejected. An example of the argument is below. Note that: - ``resources`` defines the externally-managed resources that are assigned to the subarray: - ``receptors`` - stations / substations (Low) or dishes (Mid) to be added to the subarray - ``execution_block`` is the execution block, which contains: - ``eb_id`` - execution block ID - ``context`` - free-form information from OET defining the context of the observation - ``telmodel`` - telescope model data to use for this EB - ``beams`` - visibility beams and tied-array beams - ``fields`` - fields and pointing information - ``channels`` - frequency channels - ``polarisations`` - polarisation correlation products - ``scan_types`` - scan types - ``processing_blocks`` is a list of processing blocks, each one of which contains: - ``pb_id`` - processing block ID - ``sbi_ids`` - optional list of scheduling block instances to which the processing block belongs - ``script`` - the processing script: its kind (real-time or batch), name and version - ``parameters`` - optional parameters to pass to the script - ``dependencies`` - optional list of dependencies on other processing blocks .. code-block:: json { "interface": "https://schema.skao.int/ska-sdp-assignres/2.0", "resources": { "receptors": ["SKA001", "SKA002", "SKA003", "SKA004"] }, "execution_block" : { "eb_id": "eb-test-20220921-00000", "context": {}, "telmodel": { "source_uris": ["car:ska-mid?main"], "array_layout_path": "instrument/ska1_mid_aa/layout/mid-layout.json" } "beams": [ {"beam_id": "vis0", "function": "visibilities"}, {"beam_id": "pst1", "timing_beam_id": 1, "function": "pulsar timing"} ], "fields": [ { "field_id": "field_a", "phase_dir": { "target_name": "field_a", "attrs": { "c1": 123.0, "c2": -60.0 }, "reference_frame": "icrs" }, "pointing_fqdn": "mid-tmc/telstate/0/pointing" }, { "field_id": "pst_field_0", "phase_dir": { "target_name": "pst_field_0", "attrs": { "c1": 123.05, "c2": -59.95, }, "reference_frame": "icrs" }, }, ], "channels": [ { "channels_id": "vis_channels", "spectral_windows": [ { "spectral_window_id": "fsp_1_channels", "count": 744, "start": 0, "stride": 2, "freq_min": 350000000.0, "freq_max": 368000000.0, "link_map": [[0, 0], [200, 1], [744, 2], [944, 3]] }, { "spectral_window_id": "fsp_2_channels", "count": 744, "start": 2000, "stride": 1, "freq_min": 360000000.0, "freq_max": 368000000.0, "link_map": [[2000, 4], [2200, 5]] }, { "spectral_window_id": "zoom_window_1", "count": 744, "start": 4000, "stride": 1, "freq_min": 360000000.0, "freq_max": 361000000.0, "link_map": [[4000, 6], [4200, 7]] } ] }, { "channels_id": "pulsar_channels", "spectral_windows": [ { "spectral_window_id": "pulsar_fsp_channels", "count": 744, "start": 0, "freq_min": 350000000.0, "freq_max": 368000000.0 } ] } ], "polarisations": [ { "polarisations_id": "all", "corr_type": ["XX", "XY", "YY", "YX"] } ], "scan_types": [ { "scan_type_id": "target:a", "scan_intents": ["science"], "integration_time": 0.125, "averaging_channels": 1, "averaging_samples": 1, "max_length": 21600.0, "beams": { "vis0": {"field_id": "field_a", "channels_id": "vis_channels", "polarisations_id": "all"}, "pst1": {"field_id": "pst_field_0", "channels_id": "pulsar_channels", "polarisations_id": "all"}, } } ], }, "processing_blocks": [ { "pb_id": "pb-test-20220921-00000", "script": {"kind": "realtime", "name": "test-receive-addresses", "version": "0.5.0"}, "parameters": {}, "sbi_ids": ["sbi-test-20220921-00000"] }, { "pb_id": "pb-test-20220921-00001", "script": {"kind": "realtime", "name": "test-realtime", "version": "0.3.0"}, "parameters": {}, "sbi_ids": ["sbi-test-20220921-00000"] }, { "pb_id": "pb-test-20220921-00002", "script": {"kind": "batch", "name": "test-batch", "version": "0.3.0"}, "parameters": {}, "sbi_ids": ["sbi-test-20220921-00000"], "dependencies": [ {"pb_id": "pb-test-20220921-00000", "kind": ["visibilities"]} ] } ] } .. _subarray_release_resources: ReleaseResources ^^^^^^^^^^^^^^^^ The argument of the ReleaseResources command specifies which externally-managed resources to release from the subarray. The external resources are specified in the same format as in the AssignResources command. .. warning:: If an execution block is in progress (i.e. an EB has been assigned to the subarray and it has not yet been ended), then the ReleaseResources command will cancel the execution block. An example of the argument: .. code-block:: json { "interface": "https://schema.skao.int/ska-sdp-releaseres/2.0", "resources": { "receptors": ["SKA001", "SKA002", "SKA003", "SKA004"] } } .. _subarray_configure: Configure ^^^^^^^^^ The argument of the Configure command specifies the type of the following scans from the values that were defined in the execution block. .. warning:: The subarray will not accept the Configure command if there is no execution block in progress (i.e. an EB has not been assigned to the subarray). An example of the argument: .. code-block:: json { "interface": "https://schema.skao.int/ska-sdp-configure/2.0", "scan_type": "target:a" } When executed, the obsState transitions to CONFIGURING. It will only transition to READY once all of the deployments of every real-time processing block in the execution block are up and running. .. _subarray_scan: Scan ^^^^ The argument of the Scan command specifies the scan ID and an optional start time (in ISO 8601 format with UTC timezone). If the start time is not present, the scan starts immediately. An example of the argument: .. code-block:: json { "interface": "https://schema.skao.int/ska-sdp-scan/2.0", "scan_id": 1, "start_time": "2026-02-13T12:34:56.789Z" } Even before the Scan command is executed (or before the start time is reached), data may flow through the system with ID 0. These data are only used for real-time calibration and quality assessment purposes and are not saved in data products. Attributes ---------- .. list-table:: Attributes :header-rows: 1 * - Attribute - Type - Read/Write - Values - Description * - versionId - String - Read - Semantic version - Subarray device server version * - adminMode - Enum - Read/Write - :ref:`subarray_admin_mode` - Subarray admin mode * - healthState - Enum - Read - :ref:`subarray_health_state` - Subarray health state * - healthInfo - Sequence (spectrum) of strings - Read - :ref:`subarray_health_info` - Subarray health information * - obsState - Enum - Read - :ref:`subarray_obs_state` - Subarray observing state * - resources - String - Read - :ref:`subarray_resources` - Externally-managed resources assigned to subarray * - ebID - String - Read - - Execution block ID, or "null" if execution block is not assigned * - receiveAddresses - String - Read - :ref:`subarray_receive_addresses` - Host addresses for receiving data * - scanType - String - Read - - Scan type, or "null" if scan type is not configured * - scanID - Integer - Read - - Scan ID, or 0 if not scanning * - errorMessages - String - Read - - Error messages in processing, or "{}" if there are no error messages to be reported Events ^^^^^^ The subarray device generates both Tango change events and archive events for the following attributes: - State - versionId - adminMode - healthState - healthInfo - obsState - resources - ebID - receiveAddresses - scanType - scanID - errorMessages .. _subarray_admin_mode: Admin mode values ^^^^^^^^^^^^^^^^^ The admin mode is a writeable attribute which is used to indicate the intended use of the subarray. .. list-table:: Admin mode values :header-rows: 1 * - adminMode - Description * - ONLINE (0) - Subarray can be used for normal operations * - OFFLINE (1) - Subarray should not be used * - ENGINEERING (2) - Subarray can be used for engineering purposes such as testing, debugging or commissioning * - NOT_FITTED (3) - Treated as OFFLINE * - RESERVED (4) - Treated as OFFLINE Some of these values are not applicable to a software-only system like the SDP, but they are implemented for compatibility with the standard interface. If the admin mode is set to OFFLINE, NOT_FITTED or RESERVED the device state is set to DISABLE. This prevents commands from being accepted by the subarray, but does not change any other behaviour. .. _subarray_health_state: Health state values ^^^^^^^^^^^^^^^^^^^ .. list-table:: Health state values :header-rows: 1 * - healthState - Description * - OK (0) - Subarray is functioning as expected * - DEGRADED (1) - Subarray can only provide some of its functionality * - FAILED (2) - Subarray is unable to function * - UNKNOWN (3) - Subarray device is unable to determine the health of the subarray The health state is DEGRADED if any of the following scenarios occur: - if the Helm Deployer or the Processing Controller is offline - if the PB state is not FAILED but there are errors in the errorMessages attribute. In any other case the health state is OK. .. _subarray_health_info: Health info attribute ^^^^^^^^^^^^^^^^^^^^^ Reports the reason for the subarray :ref:`Health state ` not being OK. It lists the SDP components (Helm Deployer, Processing Controller) that cannot be reached, and processing blocks that report any errors, and describes what this means for the subarray and processing. .. _subarray_obs_state: Observing state values ^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: Observing state values :header-rows: 1 * - obsState - Description * - EMPTY (0) - No resources are assigned to the subarray * - *RESOURCING (1)* - Resources are being assigned or released * - IDLE (2) - Resources are assigned to the subarray * - *CONFIGURING (3)* - Scan type is being configured (not all ingest processes are ready to receive data yet) * - READY (4) - Scan type is configured and the subarray is ready to scan * - SCANNING (5) - Scanning * - *ABORTING (6)* - Current activity is being aborted * - ABORTED (7) - Most recent activity has been aborted * - FAULT (9) - An error has occurred in observing * - *RESTARTING (10)* - Restarting to return to EMPTY observing state Values in italics indicate transitional states. The subarray may not spend much time in these states, hence there may be no reliable way of observing them using the attribute. .. _subarray_resources: Resources values ^^^^^^^^^^^^^^^^ The ``resources`` attribute is a JSON-formatted string describing the externally-managed resources assigned to the subarray. It has the same format as the ``resources`` value in the arguments of the :ref:`subarray_assign_resources` and :ref:`subarray_release_resources` commands. If no resources are assigned to the subarray, the value is an empty JSON object (``{}``). An example value: .. code-block:: json { "receptors": ["SKA001", "SKA002", "SKA003", "SKA004"] } .. _subarray_receive_addresses: Receive addresses values ^^^^^^^^^^^^^^^^^^^^^^^^ The ``receiveAddresses`` attribute is a JSON-formatted string containing the host addresses to which data from the CSP subsystems (CBF and pulsar search and timing) should be sent. It also contains the names of the Tango attributes or Kafka topics that provide the real-time calibration information to the rest of the system. The data contains one entry per scan type, and each scan type contains one entry per visibility beam or tied-array beam. If there are no receive addresses associated with the subarray, the value is an empty JSON object (``{}``). When pointing calibration is executed, the ``receiveAddresses`` attribute is updated to contain the pointing FQDN, i.e. the Tango attribute address where the calibration results are published. The pointing FQDN is stored in the ``pointing_cal`` key defined by version 0.5 (and newer) of the :doc:`receive addresses schema `. It is available for every scan type but only for visibility beams. An example value: .. code-block:: json { "interface": "https://schema.skao.int/ska-sdp-recvaddrs/2.0", "target:a": { "vis0": { "host": [[0, "192.168.0.1"], [400, "192.168.0.2"], [744, "192.168.0.3"], [1144, "192.168.0.4"]], "port": [[0, 9000, 1], [400, 9000, 1], [744, 9000, 1], [1144, 9000, 1]], "function": "visibilities", "pointing_cal": "tango://mid-sdp/queueconnector/01/pointing_offsets_{dish_id}" }, "pst1": { "host": [[0, "192.168.60.0"]], "port": [[0, 8000]], "function": "pulsar timing", "timing_beam_id": 1, "jones_cal": [[0, "kafka://kafka.cluster.local/pst1-jones-cal"]] "pst_metrics": { "pst_bandpass": "kafka://kafka.cluster.local/pst1-bandpass", "pst_timeseries": "kafka://kafka.cluster.local/pst1-timeseries", "pst_histogram": "kafka://kafka.cluster.local/pst1-histogram" } } } } Error reporting and monitoring ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The processing controller and processing scripts keep track of errors within the processing blocks and execution engines. If an error occurs, the error information and status of the processing block is reported in the processing block state. The subarray monitors the processing block and in turn sets the ``obsState``, ``healthState``, ``healthInfo`` and ``errorMessages`` attributes. If any of the following happen, ``obsState`` will be set to FAULT and error information will be populated in the ``errorMessages`` attribute: - Processing script pod fails or fails to start - Execution engine pod fails to start - An unhandled error occurs in the processing script An example of what an error in ``errorMessages`` might look like is shown below. In this case, the Measurement Sets were not found in the expected directory: .. code-block:: json { "pb-test-20211111-00000": { "data-product": { "error_message": [{ "code": { "file": "/usr/local/lib/python3.10/site-packages/ska_sdp_wflow_pointing_offset/utils.py", "function": "construct_ms_file_name", "line": 119 }, "message": "FileNotFoundError: Missing data: The following scan IDs do not have associated MeasurementSets in the directory: [1, 2, 3, 4, 5]", "time": "2024-11-15 11:30:57.524841", "type": "error" }], "key": { "kind": "data-product", "name": "pointing-offset", "pb_id": "pb-test-20211111-00000" } }, }