Changelog

Development

  • BREAKING updated launch parameters for RCal.ddal.

9.0.0

This is the changelog for the vis-receive script, which highlights only changes to the script itself, and important changes for each internal component. See the individual release notes for receive-core 7.8.0/7.9.0, receive-modules 9.0.0/9.0.1/9.0.2, receive-processors 6.0.0/6.0.1 and realtime-calibration 5.0.0 for more details on changes within specific components.

Added

  • Receivers now implement ADR-111, accepting data for scan 0. Data for such scans is not written into a Measurement Set.

  • Added phase and expires_at parameter options to MSWriter.

  • Added DataProductPersist flow per output DataProduct.

  • Added multi-beam reception support. Individual receiver processes within a single node are setup to receive the same amount of frequency channels for each scan type.

  • rcal now creates flows for sending gains solution QA metrics. Two flows (one to Kafka, one forwarding to Tango) and corresponding Kafka topic are created per vis beam; configured to send two metrics (bandpassConverged and visibilityChisq)

  • Logs related to vis-receive deployment IP assignment are marked with redact:false so Kibana does not redact their content, allowing debugging of ongoing IP clashing issues.

  • New rcal.sky_model parameter for users to point to a pre-existing sky model on disk for RCal to use.

  • If rcal.sky_model is not given, a request for a local sky model for RCal is created for the fields used by all visibility beam ID. The resulting Flows are added as a source of the corresponding RCal’s gains flows.

  • Added QA bandpass calibration flow registration. The vis-receive script now creates a flow and Kafka topic for the RCal processor to publish direction-independent bandpass solutions to the QA display infrastructure (YAB-380).

Fixed

  • A number of metadata variables are now correct in the output Measurement Set, and the in-memory Visibility data model. This includes:

    • time is now at the centre of the interval (used to be at the end of the interval, similar to how CBF sends timestamps).

    • interval was hardcoded to 1, now is provided externally to SDP via AssignResources.

    • exposure was previously missing in memory, and set to == time in the Measurement Set. It is now set to interval * (FD / 255)^2.

    • time_centroid was previously missing in memory, and set to == time in the Measurement Set. It is now calculated from the TCI values sent by CBF.

    • weight was previously directly set to the FD values sent by CBF. It is now set to exposure * bandwidth_in_khz.

  • Fixed invocation of QA display processors to use --num-receivers command line parameter instead of --num-subbands.

  • Corrected vis/PST beam pairing to use the calibration_beam field from AssignResources v2.0 schema (YAB-363).

  • Invalid visibility data (Nan or Inf) is automatically flagged. This addresses SKB-1252.

  • The uvw data variable of the Visibility datasets now always has 3 dimensions. This addresses SKB-1265.

  • H5Parm files written by RCal are flushed to disk after every write, making them usable even in situations when RCal doesn’t exit cleanly and therefore doesn’t close them as expected. This addresses SKB-1315.

  • The average weight value for each time/baseline is written to the WEIGHT column when per-frequency weights are written to WEIGHT_SPECTRUM (previously the column stored 0s). This addresses SKB-1358.

Removed

  • BREAKING Removed max_ports_per_node parameter. Use num_nodes to deploy multi-node vis-receive.

Changed

  • BREAKING Receivers associate incoming data with scans by matching timestamps inside each payload against the scan schedule published by SDP LMC.

  • BREAKING The rcal.flux_limit and rcal.uv_min parameters now default to None instead of 0.0 and 100.0, respectively. This allows the underlying RCal defaults to be used, whatever they might be, without having to match those defaults in the vis-receive script.

  • Bumped versions of:

    • ska-sdp-realtime-receive-modules OCI image to ``9.0.2`.

    • ska-sdp-realtime-receive-processors OCI image to 6.0.1.

    • ska-sdp-realtime-calibration OCI image to 5.0.0.

    • ska-sdp-scripting to ^3.0.1.

    • ska-sdp-config to ^1.6.1

  • Removed dependency on ska-telmodel and replaced with ska-telmodel-client at ^1.30.1

  • The buffer request calculation for mswriter now uses calculates visibility counts on a per scan_type basis, including using the new per-scan type integration_time and max_length (if available), for a more accurate estimate.

9.0.0-alpha1

BREAKING Alpha release of several additions for early testing in SDP integration test suite.

8.1.1

Changed

  • Bumped version of ska-sdp-realtime-calibration OCI image to 4.0.1.

  • Default receive and plasma containers imagePullPolicy to IfNotPresent. The Helm Chart defaults to Always following advice given in YAN-1275 (see recording @ 34:00), but this introduces slowness to the startup of the script (see SKB-771), as well as pressure on the Kubernetes cluster (see STS-2647 and related Slack thread). In particular this message indicates IfNotPresent is an appropriate default. This partly addresses SKB-771.

Fixed

  • RCal processor works correctly when running as non-root users. Previous versions did not have a home directory for the default receive user shipped with the OCI image, resulting in permission errors, in particular when ska-telmodel attempted to create and populate its cache.

8.1.0

Added

  • Added RCal configuration support using vis-receive script rcal parameter.

  • Added RCal configuration for Kafka-based gains solution publishing. Flows are written to the SDP Config DB, topics are created in Kafka, and published in the SDP receive addresses for external consumption under the jones_cal key.

  • Added default Stats configuration settings to vis-receive deployment.

  • Added support for populating the obscore section of ska-data-product.yaml files produced by mswriter.

Changed

  • RCal processors are launched per visibility beam instead of a single instance. Visibility data is first accumulated by frequency, then by time, before given to RCal.

  • Bumped versions of:

    • ska-sdp-realtime-receive-modules OCI image to 8.0.0.

    • ska-sdp-realtime-receive-processors OCI image to 5.0.0.

    • ska-sdp-realtime-calibration OCI image to 4.0.0.

    • ska-sdp-helmdeploy-vis-receive Helm Chart to 1.3.0.

  • Bumped Docker base images to ska-build-python 0.3.3 and ska-python 0.2.5.

8.0.0 (Alpha)

Note

It was believed at the time of releasing that scripts did not support pre-release versioning.

Changed

  • Dataproduct flows created by mswriter now include a FlowSource with a beam_id parameter to make them more easily searchable

  • Bumped versions of:

    • ska-sdp-realtime-receive-modules to 8.0.0-rc1.

    • ska-sdp-realtime-receive-processors to 5.0.0-rc1.

    • ska-sdp-realtime-calibration to 3.0.0-dev.cba0ea060.

  • BREAKING Measurement Sets written by the mswriter processor are now output into beam-<beam_id>/scan-<scan_id>/<output_basename>.<scan_type_id>.<subband>.<output_extension>.

Fixed

  • Fixed mswriter flow lookup logic to match flows created by the vis-receive script. Data product flows are correctly updated and contain the paths and statuses of the Measurement Sets as they are closed. This was broken in version 7.0.0.

7.0.4

Changed

  • Bumped ska-sdp-realtime-receive-modules to 7.0.2.

Fixed

  • Fixed regression introduced in vis-receive 7.0.3 where ITF station names (e.g., itf1) were not accepted as valid anymore by the receiver code. This addresses SKB-1189.

7.0.3

Changed

  • Bumped ska-sdp-scripting to ^2.1.0.

  • Bumped ska-sdp-realtime-receive-processors to 4.2.0.

Fixed

  • Fixed parsing of scan types from AssignResources execution blocks in the vis-receive script itself after clarifications made to ADR-54. This addresses SKB-1147.

  • PHASED_ARRAY table written by the mswriter processor is now correctly filled and compatible with the Everybeam software. This addresses SKB-1060.

7.0.2

Fixed

  • Fixed schema for telstate TRL dictionary values.

7.0.1

Fixed

  • Fixed telstate BeamID validation to support any string. This matches ska-sdp-assignres 1.1.

Updated

  • Updated dependency on realtime-receive-modules to ^7.0.1. This version fixes the parsing of scan types from AssignResources execution blocks after clarifications made to ADR-54. This addresses SKB-1147.

7.0.0

Added

  • BREAKING Added schema for telstate script parameter.

  • Added new telescope_model script parameter for configuring the location of the telescope model and the path in the telescope model where individual station documents can be loaded from.

  • Added a SKA_TELMODEL_SOURCES environment variable to all containers in the resulting vis-receive deployment. The variable is set to the telescope_model.sources script parameter value to provide a consistent reference to Telescope Model data.

  • Added new rcal.enabled script parameter to indicate whether to launch RCal in the vis-receive pipeline.

Changed

  • BREAKING Changed pointing fields in telstate script parameter to a 2-layer mapping of beam ID, receptor name, and TRL.

  • mswriter now creates one Data Product flow per (visibiltiies) beam, instead of a single flow.

  • Changed arguments provided to mswriter to include an explicit telescope name.

Removed

  • BREAKING Removed deprecated parameter use_network_definition.

6.3.0

Added

  • Add buffer request for Measurement Set data product. Its size is calculated as a worst-case scenario in the frequency, polarisation and time dimensions, and is based on the information in the Execution Block.

  • mswriter performs shutdown sequence when End command is sent to the SDP Subarray and transitions the Measurement Set data product flow to COMPLETED at the end of said sequence.

  • mswriter writes CBF FD values as weights into output Measurement Set. This addresses SKB-1042 fully and SKB-1081 partially.

Changed

  • Bumped Docker SKA python base images to ska-build-python 0.3.2 and ska-python 0.2.4.

  • Bumped ska-sdp-realtime-receive-processors to 4.1.0.

  • Bumped ska-sdp-realtime-receive-modules to 7.0.0.

  • Restructured source files into a python package structure.

  • The runtime capabilities NET_ADMIN and NET_RAW now correctly enable permissions for the ip, netstat, ping, tcpdump, ifconfig, and arp executables in the receiver containers.

  • Use ska-sdp-helmdeploy-vis-receive chart version 1.2.0. Notably this new version includes new environment variables so pods and receivers know their intra- and inter-node rank, and also shortens the readinessProbe period of containers to one second for faster startup times.

Fixed

  • Several Measurement Set metadata fields that were previously wrongly filled have now the correct value. This addresses SKB-671.

  • Receivers now correctly react to Configure commands on the SDP Subarray, and the reception loop is promptly stopped when necessary. This addresses SKB-1104.

6.2.0

Changed

  • Revamped script parameter schema descriptions and documentation.

  • Default QA metric list changed from ‘stats’ to ‘all’.

  • Organisation of QA metric generation per container updated.

  • Bumped ska-sdp-realtime-receive-processors to 4.0.2.

Fixed

  • mswriter correctly counts the number of receivers, correctly closing Measurement Sets at EndScan time. This addresses SKB-1099.

  • OBSERVATION/TELESCOPE_NAME column in Measurement Set contains SKA-Low/SKA-Mid instead of Unknown, depending on the telescope data is being received for. This addresses SKB-1062.

  • Fixed validation of the reception_network vis-receive parameter, which could not be successfully provided by users.

6.1.2

Fixed

  • Bumped ska-sdp-realtime-receive-modules to 7.0.0-alpha.5, which fixes an issue where receptors were not sorted according to the CBF SDP ICDs. This addresses SKB-986.

6.1.1

Changed

  • Bumped ska-sdp-realtime-receive-modules to 7.0.0-alpha.4 to fix a problem where the Beam ID sent by Low CBF was ignored.

6.1.0

Changed

  • Added new QA Display metrics: averagedamplitude, amplitudeandphasevariance.

  • QA Display metrics are now beam aware.

6.0.0

Changed

  • BREAKING vis-receive defaults to always publishing IPs into the SDP receive addresses. This is to better align with SDP interface definitions, since other subsystems rely on IPs, and not hostnames, being published. Note that this affects mostly software-only environments; in platforms where vis-receive binds to the host’s NIC SDP already publishes IPs. When dry_run is set, IP addresses in the 127.0.0.0/8 link-local network are used. This addresses SKB-532.

Added

  • New allow_hostnames_in_receive_addresses parameter allows users to continue publishing Kubernetes hostnames into the SDP receive addresses when vis-receive is bound to the Kubernetes network.

5.5.0

Added

  • Added additional terms for pointing FQDN templates: "dish_name", "station_name", "beam_id".

Changed

  • Bumped ska-sdp-realtime-receive-processors to 3.5.3 to handle station names with hyphens. This addresses SKB-675.

Fixed

  • Corrected default beam ID for Low observations from 0 to 1. This addresses SKB-675.

5.4.4

Fixed

  • Bumped ska-sdp-realtime-receive-modules to 6.1.2. This addresses SKB-944.

Changed

  • Bumped ska-sdp-realtime-receive-processrs to 3.5.2 for the mswriter processor, reducing the amount of images required by a vis-receive deployment.

5.4.3

Fixed

  • Increased the default max message size for QA Metrics through Kafka, default is now 50MiB, this will allow ~14k channels with no channel averaging enabled to be used.

  • Bumped ska-sdp-realtime-receive-processors to 3.5.2 for the QA metrics. This addresses SKB-950 and SKB-951.

5.4.2

Changed

  • Bumped ska-sdp-realtime-receive-processors to 3.5.1

  • Include the ability to change the max message size for QA Metrics through Kafka, and set the default to 10MiB.

Fixed

  • Corrected scaling of x-axis for lag plot metrics. This addresses SKB-910.

5.4.1

Changed

  • Bumped ska-sdp-scripting to ^1.2.0.

Fixed

  • Receive addresses are published only once regardless of whether they contain IPs or hostnames. This addresses SKB-840.

5.4.0

Changed

  • Update Signal Display Metrics topics to be processing block aware. As well as made the receiver stats have (unused) parameters, so that it looks the same as the other metrics.

  • Updated Signal Display Receiver Metrics flow to include configuration, for uniformality with the other metrics

5.3.2

Fixed

  • Bump receive-modules and receive-processors to 6.1.1 and 3.4.0 respectively. These new component versions detect if the Plasma Store crashes, automatically reconnecting as needed. This addresses SKB-856.

5.3.1

Fixed

  • Put the --metrics ... arguments of the signal display metrics generator processors so they appear at the end of the command line. This is currently required by the Signal Displays UI.

5.3.0

Added

  • Added Low specific pointings flow definitions for target and tracking from MCCS station beam devices.

Changed

  • Bump receive-processors image to 3.3.1. This new version includes fixes in the frequency accumulation processor logic, and changes the mswriter to continuously flush data to disk. This addresses SKB-895.

  • Changed the processes_per_node PB parameter defaults from 1 to None. When None, the script calculates an appropriate number of processes per node, setting it to a value that allows full bandwidth observations to be carried out correctly. The plasma store is automatically sized accordingly. This addresses SKB-787.

  • Receivers generate Visibility datasets of fixed size in the time axis. This ensures that in multi-receive deployments, the different Visibilities are aligned in time.

Fixed

  • Signal Display processors work correctly against multiple receiver instances by first combining data for different frequency slices. This addresses SKB-787.

5.2.1

Changed

  • Plasma container uses the same OCI image than the receiver, reducing the deployment cost of the vis-receive pipeline to help address SKB-771.

5.2.0

Added

  • New mswriter.upper_triangular_baselines option (defaults to False) to ensure MSs have baselines written in upper-triangular order as expected by most 3rd-party software.

Changed

  • Update receive-modules and receive-processors image to 6.1.0 and 3.1.1 respectively. These new versions includes fixes for the receiver, the processor infrastructure, and to the mswriter processor. Addresses SKB-695, SKB-814 and many parts of SKB-787 (but not fully).

  • Default UVW engine is measures (it was implicitly katpoint). The latter is an order of magnitude slower. They are compatible to within 1mm and 40 arcsec.

5.1.1

Fixed

  • Internal validation of receive addresses correctly accepts 3-tuples in the port array.

  • Data queue topic creation works regardless of how many partitions an existing topic already has.

5.1.0

Added

  • reception_network setting added to indicate the network the receiver should bind to. Default behaviour is to automatically determine if hardware support is available (i.e., there is a network definition found in the config DB).

Changed

  • use_network_definition is deprecated in favour of reception_network, If both are given, reception_network takes precedence.

Fixed

  • dry_run parameter has been brought back, it was accidentally removed in 5.0.0.

5.0.0

  • Bumped ska-sdp-scripting to ^1.0.0 to support SDP v1.

  • Bumped default image versions for receiver, processors, integration and rcal to 6.0.0, 3.0.0, 1.0.0 and 3.0.0 respectively, all of which support SDP v1.

  • Added pod settings custom IP reading unit tests.

  • Added writing SharedMem flow configuration for receiver.

  • Added writing DataProduct flow configuration for mswriter processor.

  • Changed vis-receive to validate script inputs and chart value outputs using pydantic models.

  • Sub-networks assigned to vis-receive deployments are now allocated starting from the sub-network that comes after the last allocated one. This reduces the chances of a vis-receive deployment that takes a long time to terminate (or gets stuck in error) from clashing with newly-minted deployments.

  • Removed custom SDP Config DB operations code, replaced with operations classes from ska-sdp-config.

  • Update Dockerfile to use SKA python base image.

  • BREAKING processors parameter doesn’t accept a list of strings anymore (this had been deprecated since vis-receive 2.2.0).

4.5.0

  • Signal Metrics will now create the topics if needed, and update the amount of partitions as needed.

  • Signal Metrics adds new parameters: + window_count: The amount of extra windows to allow. + rounding_sensitivity: The precision to round to.

  • Upgrade Signal Metrics Generator to latest version (0.24.0)

  • Updated to using flows to configure the queue connector device for pointings and signal display.

4.4.1

  • Updated the patch version of the averagetime-mswriter image to 0.3.1 This picks up a minor change in how the data is accumulated and averaged.

4.4.0

  • Signal Display Metrics now configures down-scaled processing by default.

  • Signal Display Metrics configuration is now validated using a pydantic model.

  • Signal Display Metrics processor is updated to the latest version. It is now defaulted to 0.23.0

  • Added a new processor that integrates the visibilities in time before writing out the Measurement Sets. This processor is called averagetime-mswriter.

4.3.0

  • Use poetry for managing dependencies

  • Add new Signal Metrics Display configuration

  • BREAKING the previous Signal Metrics processors should no longer be used, only use the new configuration.

  • Bumped the default version of the mswriter processor to 2.3.0

  • Bumped the default version of the receiver to 5.2.2

  • Use ska-sdp-scripting 0.9.0, which uses ska-sdp-config 0.9.0 with new pydantic-based interface.

  • Bumped the default version of the RCAL processor to 2.0.1. Also to pick up changes to the configuration database.

4.2.1

  • Use ska-sdp-scripting 0.7.1, which removes cancelled deployments.

4.2.0

  • Moved receive_addresses code to scripting library (part of ska-sdp-scripting 0.7.0)

  • Use ska-sdp-scripting 0.7.0, which uses ska-sdp-config 0.6.0 with new pydantic-based interface. It also addresses a bug in pb.receive_addresses.

  • BREAKING This and newer versions are not compatible with SDP version < 0.21.0

4.1.0

  • mswriter-processor version updated to 2.0.2

  • New function added to configure queue connector for dish pointing in case vis-receive is deployed together with pointing-offset script

  • Upgraded the QA Metrics Processor to 0.20.0

4.0.0

  • New Signal Display Metrics processors added, using prefix signal-display-metrics-* Four have been added -all, -basic, -phase, and -amplitude. The first can be used for small datasets/testing, but the other 3 should be used in almost all other cases.

  • BREAKING the QA Metrics processor (qa-metric-generator) has been removed and replaced with signal-display-metrics-* processors.

3.0.0

  • BREAKING Bump latest version of mswriter processor to 2.0.0, rcal processor to 2.0.0, and receive-modules to 5.0.0. This new combination of components implements the required logic to close the output Measurement Sets at EndScan time, as well as updating the SDP Config DB with the list of Measurement Sets that have been written. This change is breaking because this new version of the receiver will not work against processors based on versions of the receive-processors package lower than 2.0.0.

2.3.0

  • Bump latest version of rcal processor to 1.0.0. This version of the rcal processor can deal with the latest versions of the receiver that implement data aggregation across time and frequency.

2.2.1

  • Use ska-sdp-scripting 0.6.3, which implements configuring the QueueConnector device in Phase enter and exit methods

2.2.0

  • The processors PB parameter can now be a dictionary instead of a list, allowing for partial built-in overrides to be given. Passing a list is still supported, but deprecated.

  • Bump default mswriter processor version to 1.1.1.

  • Add initial qa-metrics-generator and rcal processor definitions.

2.1.0

  • Updated to new LMC Queue Connector 3.0.1 configuration database layout and schema.

  • Added waiting for LMC Queue Connector to reach ON state before transitioning PB to ready.

  • Updated to scripting library ~= 0.6.2.

2.0.0

  • Changed how the network configuration is read from the SDP configuration database. Instead of reading it from /network-definition, it is now read from /resource/<resource_type>:<resource_name>.

  • Changed how networks are allocated within SDP. Instead of updating a central /network-definintion value in the SDP configuration database, we now append/remove entries of the form <pb_id>:<resource_type>:<resource_name> for each allocation under the /allocation prefix, and leave the /network-definition value unedited.

  • BREAKING Extra PB parameters are not forwarded anymore as Helm values for the vis-receive Helm Chart. Use the new extra_helm_values PB parameter instead.

  • Added top-level extra_helm_values PB parameter (defaults to {}) where users can specify any arbitrary extra set of Helm values that should be directly forwarded to the vis-receive Helm Chart.

  • Added top-level port_start PB parameter (defaults to 21000), which was previously hidden within the implicit set of Helm values that the vis-receive chart received.

  • Added convenience top-level transport_protocol PB parameter (defaults to "udp"), which previously had to be given via the extra Helm values forwarded to the vis-receive Chart.

1.5.0

  • Updated default major versions of the receiver and mswriter processor images to 4 and 1 respectively.

  • The SDP_KAFKA_HOST environment variable pointing to the SDP Kafka instance is now exposed to all pods.

1.4.0

  • Added use_network_definition as a PB parameter. When True this will use the /network-definition stored in SDP etcd to override the extraMetadata field of pod_settings to use the host network port defined via a “network attachment definition” in Kubernetes along with an allocated IP address with its cidr. The used subnet range will be added to the allocted-subnets of the /network-definition and update the SDP etcd entry. On completion the allocated subnet will be released.

  • Update to scripting library ~= 0.6.0.

  • Stop forcing a backend for the ska-sdp-config library in the pods launched by this script.

1.3.0

  • Added writing to LMC Queue Connector database configuration.

1.2.0

  • Update version of built-in mswriter processor to 0.5.1 and of the receiver to 3.10.1.

1.1.1

  • Update to latest scripting library (0.5.2)

1.1.0

  • Update to latest scripting library (0.5.0). This required the update of how phase.wait_loop is used in the script.

1.0.1

  • Fix issue with custom IPs causing the script to crash due to a wrong variable name being referenced while updating the PB’s receive addresses.

1.0.0

  • Complete rewrite to use the new vis-receive Helm Chart, which is itself an almost complete rewrite of the receive Helm Chart.

0.8.1

  • Update to latest scripting library (0.4.2)

0.8.0

  • Read name of PVC to be used by the receiver from the SDP_DATA_PVC_NAME environment variable as communicated by the SDP Processing Controller.

  • Communicate more environment variables to extra containers. This is required to use the new SDP metadata package.

0.7.0

  • Update to latest receiver image (3.6.0).

  • Add verbosity PB parameter.

  • Add PB parameters to map a native network device, assign an IP to it, and indicate a node selector for the receiver pod.

0.6.0

  • Update to latest receiver image (3.3.0).

  • Update to latest scripting library (0.4.1)

  • Point the receiver to the correct SDP Configuration Database and Execution Block so it can read Scan metadata associated to each incoming payload and the list of antennas making up the subarray.

  • Made Docker image slimmer by removing unnecessary software installations.

0.5.1

  • Update to latest receiver image (2.1.0).

  • Use new readinessProbe available on the receiver chart.

  • Removed transmission.* option handling as they are not needed anymore and added confusion.

0.5.0

  • Port to use SDP scripting library (formerly known as the workflow library).

0.4.0

  • Removed reception.num_ports configuration value, replaced it with reception.channels_per_stream, which is what the receiver program uses.

  • Improved default values used to launch receiver chart.

0.3.8

  • Refactored code and added workflow pytest code

0.3.7

  • Updated to add flexibility for the command parameter and updated default parameters

0.3.6

  • Ported to work with the correct version of the workflow library (0.2.6)

0.3.5

  • Capable to deploy a receive process, plasma store, and a number of other processes to consume the data from the plasma store.

0.3.4

  • Use dependencies from the central artefact repository and publish the workflow image there.

0.3.3

  • Ported to use the latest version of workflow library (0.2.4). Capable to deploy multiple receive processes. Ports published in the receive addresses match with the actual ports of the receive process(es)

0.3.2

  • use python:3.9-slim as the base docker image