Flows created

This section lists the data flows created in the configuration database by the vis-receive script.

TBA

Individual processor flows

MSWriter

Flows created for the MSWriter Processor

Name

Data model

Source

Sink

vis-receive-mswriter-processor-beam-{beam_id} (one created per visibilities beam)

Visibilities

Data Product (Measurement Set at PVC subpath product/{eb_id}/ska-sdp/{pb_id}/beam-{beam_id})

RCal (Real-time Calibration)

Flows created for RCal gains output

Name

Data model

Source

Sink

rcal-gains-{scan_type_id}-{vis_beam_id}-{pst_beam_id} (one per scan type, visibility beam and pulsar timing beam)

Gains

  • FlowSource with function="vis-receive:rcal-processor" and parameters: {scan_type_id, beam_id}.

  • LSM request key (see below) with function="vis-receive:rcal-processor"` and parameters: ``field_id.

Kafka DataQueue (topic: rcal-gains-{pb_id}-{scan_type_id}-{pst_beam_id}, format: npy)

RCal gains are published to Kafka topics for downstream consumption by beamformers and monitoring systems. Each combination of scan type and pulsar timing beam gets a dedicated topic. The gains are complex vectors (Jones matrices) serialized in numpy format.

The RCal processor discovers these flows by searching for flows with: - data_model == "Gains" - A single source with function == "vis-receive:rcal-processor" - Source parameters containing both scan_type_id, visibility_beam_id and pst_beam_id.

Downstream processes can discover RCal gains topics via the SDP receive addresses: receive_addresses[scan_type_id][beam_id]["jones_cal"] which provides the Kafka endpoint in the format kafka://host:port/topic.

Flows created for RCal QA bandpass output

Name

Data model

Source

Sink

rcal-bandpass-qa (one per processing block)

Gains

FlowSource with function="vis-receive:rcal-processor:bandpass-calibration-generation"

Kafka DataQueue (topic: rcal-bandpass-qa-{pb_id}, format: msgpack_numpy)

A single QA bandpass flow is created per processing block, shared across all visibility beams. The RCal processor publishes direction-independent bandpass calibration solutions (a time-squeezed GainTable serialized as msgpack_numpy) to this topic after each solve interval.

The RCal processor discovers this flow by searching for flows with:

  • data_model == "Gains"

  • A DataQueue sink

  • A source with function == "vis-receive:rcal-processor:bandpass-calibration-generation"

Flows created for RCal LSM requests

Name

Data model

Source

Sink

rcal-lsm-{field_id} (one for each visibility beam’s field)

CsvNamedColumns

FlowSource with function="GlobalSkyModel.RequestLocalSkyModel" and parameters: {ra, dec, fov}

DataProduct (pv_subpath: product/{pb._eb_id}/ska-sdp/{pb._pb_id}/ska-sdm/sky/{field_id})
Flows created for RCal gains solutions QA Metrics

Name

Data model

Source

Sink

rcal-qa-metrics-{vis_beam_id} (one per visibility beam)

GainsMetrics

FlowSource with function="vis-receive:rcal-processor:qa-metrics" and parameters: {visibility_beam_id}

DataQueue (topic: rcal-qa-metrics-{pb_id}-{vis_beam_id}, format: json)

rcal-qa-metrics-tango-{vis_beam_id} (one per visibility beam)

GainsMetrics

QA metrics dataqueue flow (see above)

Tango (attributes bandpassConverged_{vis_beam_id} and visibilityChisq_{vis_beam_id})

Two QA metrics for assessing gains solution quality, bandpass_converged (boolean) and visibility_chisq (float) are produced per visibility beam. These are sent to Kafka in json format, from where they are picked up and exposed as Tango attributes, bandpassConverged_{vis_beam_id} and visibilityChisq_{vis_beam_id}.