Step-by-step testing in SDP

This section gives an example of how one can run the pointing processing script, together with the vis-receive script, in SDP. The example uses the Mock Dish devices to send pointing data and the CBF emulator to send visibility data.

This procedure has been written into a pointing calibration Jupyter notebook in the SDP Notebooks repository, which demonstrates the pointing offset script deployment using test data.

1. Set up data pods

We will need two pods to interact with input and output data of the test. Both of thme mount a Persistent Volume Claim (PVC) to hold data. The sender-data pod is needed to provide the input data for the Mock Dishes and the CBF emulator. The receive-data pod is needed to inspect the outputs of the receiver and the pointing pipeline.

The testing suite of SDP provides a pod definition, which can be used for for both pods via the data pod configuration yaml.

The sender-data pod has to be deployed in the SDP control-system namespace, while the other needs to be deployed in the SDP processing namespace. Make sure you update the PVC claim name to the PVC used by SDP in the above files (usually test-pvc):

volumes:
- name: data
  persistentVolumeClaim:
    claimName: <sdp-pvc-name>

To make it easy to distinguish between the two pods, also update the name of the pod that you will deploy with to either sender-data or receive-data:

metadata:
  name: <pod-name>

To install them, run these kubectl commands (use the appropriate file and namespace):

kubectl create -f <sender-pod-yaml> -n <control-system-namespace>
kubectl create -f <receive-pod-yaml> -n <processing-namespace>

2. Download the data for testing

Warning

Only download the data, if you are working with a PVC installed by SDP and not with a shared, already existing PVC! Else, make sure you download to a path unique to you and clean it up after your testing.

The Measurement Sets (MS) used for testing the pointing pipeline can be found on Google Cloud Platform (GCP). These MS are simulated 5 scans with the SKA AA0.5 configuration at Band 2. The pointing HDF files, used by the Mock Dishes, on GCP can be found here.

Download the data in to the sender-data pod. Exec into the pod using kubectl or k9s (see k9s docs here):

kubectl exec -it sender-data -n <control-system-namespace> -- bash

Install gsutil :

pip install gsutil

Download the MS data and pointing data from the GCP link above into the mounted PVC (usually /mnt/data/) by running the command below. If you are connected to a shared PVC, make sure you use a directory name unique to yourself and not test_data. Don’t forget to remove the data once you are finished testing, if you used a shared PVC.

mkdir -p /mnt/data/test_data/scans
gsutil -m cp -r \
    "gs://ska1-simulation-data/simulations/pointing-offset/orc-2052/reordered/scan-1.ms" \
    "gs://ska1-simulation-data/simulations/pointing-offset/orc-2052/reordered/scan-2.ms" \
    "gs://ska1-simulation-data/simulations/pointing-offset/orc-2052/reordered/scan-3.ms" \
    "gs://ska1-simulation-data/simulations/pointing-offset/orc-2052/reordered/scan-4.ms" \
    "gs://ska1-simulation-data/simulations/pointing-offset/orc-2052/reordered/scan-5.ms" \
    /mnt/data/test_data/scans

Later the CBF emulator needs to be pointed to these data. Next, download the pointing HDF5 files that will be used to simulate the incoming data from the mock dishes:

mkdir -p /mnt/data/test_data/pointing-hdfs
gsutil -m cp \
    "gs://ska1-simulation-data/simulations/pointing-offset/orc-2052/pointing-hdfs/actual_pointing_scan_scan-1.hdf" \
    "gs://ska1-simulation-data/simulations/pointing-offset/orc-2052/pointing-hdfs/actual_pointing_scan_scan-2.hdf" \
    "gs://ska1-simulation-data/simulations/pointing-offset/orc-2052/pointing-hdfs/actual_pointing_scan_scan-3.hdf" \
    "gs://ska1-simulation-data/simulations/pointing-offset/orc-2052/pointing-hdfs/actual_pointing_scan_scan-4.hdf" \
    "gs://ska1-simulation-data/simulations/pointing-offset/orc-2052/pointing-hdfs/actual_pointing_scan_scan-5.hdf" \
    "gs://ska1-simulation-data/simulations/pointing-offset/orc-2052/pointing-hdfs/requested_pointing_scan_scan-1.hdf" \
    "gs://ska1-simulation-data/simulations/pointing-offset/orc-2052/pointing-hdfs/requested_pointing_scan_scan-2.hdf" \
    "gs://ska1-simulation-data/simulations/pointing-offset/orc-2052/pointing-hdfs/requested_pointing_scan_scan-3.hdf" \
    "gs://ska1-simulation-data/simulations/pointing-offset/orc-2052/pointing-hdfs/requested_pointing_scan_scan-4.hdf" \
    "gs://ska1-simulation-data/simulations/pointing-offset/orc-2052/pointing-hdfs/requested_pointing_scan_scan-5.hdf" \
    "gs://ska1-simulation-data/simulations/pointing-offset/orc-2052/pointing-hdfs/source_offset_scan_scan-1.hdf" \
    "gs://ska1-simulation-data/simulations/pointing-offset/orc-2052/pointing-hdfs/source_offset_scan_scan-2.hdf" \
    "gs://ska1-simulation-data/simulations/pointing-offset/orc-2052/pointing-hdfs/source_offset_scan_scan-3.hdf" \
    "gs://ska1-simulation-data/simulations/pointing-offset/orc-2052/pointing-hdfs/source_offset_scan_scan-4.hdf" \
    "gs://ska1-simulation-data/simulations/pointing-offset/orc-2052/pointing-hdfs/source_offset_scan_scan-5.hdf" \
    /mnt/data/test_data/pointing-hdfs

3. Install the Mock Dish helm chart

The chart can be found in the SDP integration repository:

Inspect the values.yaml file. Based on this, you need to create a custom values file and add the following data:

mock_dish:
  receptors: <receptors>
  dishMaster:
    achievedPointing:
      1: <hdf_data_dir>/actual_pointing_scan_scan-1.hdf
      2: <hdf_data_dir>/actual_pointing_scan_scan-2.hdf
      3: <hdf_data_dir>/actual_pointing_scan_scan-3.hdf
      4: <hdf_data_dir>/actual_pointing_scan_scan-4.hdf
      5: <hdf_data_dir>/actual_pointing_scan_scan-5.hdf
  dishLeafnode:
    desiredPointing:
      1: <hdf_data_dir>/requested_pointing_scan_scan-1.hdf
      2: <hdf_data_dir>/requested_pointing_scan_scan-2.hdf
      3: <hdf_data_dir>/requested_pointing_scan_scan-3.hdf
      4: <hdf_data_dir>/requested_pointing_scan_scan-4.hdf
      5: <hdf_data_dir>/requested_pointing_scan_scan-5.hdf
    sourceOffset:
      1: <hdf_data_dir>/source_offset_scan_scan-1.hdf
      2: <hdf_data_dir>/source_offset_scan_scan-2.hdf
      3: <hdf_data_dir>/source_offset_scan_scan-3.hdf
      4: <hdf_data_dir>/source_offset_scan_scan-4.hdf
      5: <hdf_data_dir>/source_offset_scan_scan-5.hdf

pvc:
  name: <pvc_name>
  path: /mnt/data/

Replace the following:

  • <receptors>: needs to match what is later added to the AssignResources configuration string.

  • <hdf_data_dir>: where you downloaded the HDF files to in your PVS: /mnt/data/test_data/pointing-hdfs

    or your custom path

  • <pvc_name>: the PersistentVolumeClaim that contains the data (usually test-pcv or shared)

Note that the numbers associated with the pointing files are the Scan IDs. Save the file as a yaml file.

To install, run:

helm install mock-dish ska-sdp-integration/tests/resources/charts/mock-dish -f <custom-yaml>.yaml -n <control-system-namespace>

(update path and file name as needed).

4. Run AssignResources on the subarray device

Exec into the itango-console pod (with kubectl as below, or by selecting the pod in k9s and pressing s to drop into the shell, then typing itango3):

kubectl exec -it ska-tango-base-itango-console -n <control-system-namespace> -- itango3

This only needs to be done once.

Connect to the subarray device:

d = DeviceProxy('test-sdp/subarray/01')
d.On()

Load the configuration string into a variable:

import json
config = json.dumps(<example_configuration_string>)

The configuration string used for this test can be found here.

Note the following about this string and update as instructed:

  • It runs two processing scripts in two different processing blocks (PB):

    • The pointing-offset script runs with PB ID pb-test-20211111-00000

    • The vis-receive script runs with PB ID pb-test-20211111-00001

  • The pb_id in the pointing-offset script’s dependencies key needs to match the pb_id of the vis-receive script.

  • Similarly, the pb_id in the vis-receive script’s dependencies key needs to contain the pb_id of the pointing-offset script. If this is omitted, the QueueConnector device will not load pointing data from the dish tango devices.

  • If you rerun the script, either delete the PB information from the configuration DB, or update the IDs in the configuration string.

  • resources.receptors refer to SKA Mid dish names.

Run AssignResources:

d.assignResources(config)

This will start the two processing blocks in the processing namespace, which, in return, will start the pod for the pointing pipeline and the pod for the receiver, respectively.

Once the obsState reached IDLE (check with d.obsstate), run Configure for pointing scans:

d.Configure(json.dumps({
    "interface": "https://schema.skao.int/ska-sdp-configure/1.0",
    "scan_type": "pointing-xyz"
}))

5. Running Scans

From this point, the below commands need to be run for every scan. Once the obsState of the subarray device is READY (you can check by printing d.ObsState), turn on the dishes (both Dish LeafNodes and DishMaster devices), so that the pointing data can be accessed from their attributes:

devs = [dev for dev in get_device_list() if 'mockdish' in dev]

for dev in devs:
    dish = DeviceProxy(dev)
    dish.Scan(<scan_id>)

Next, run the first scan:

d.Scan(json.dumps({
    "interface": "https://schema.skao.int/ska-sdp-scan/1.0",
    "scan_id": <scan_id>
}))

Replace <scan_id> with the current scan ID (integer).

Set up the CBF emulator to send the first MS (in the command line, not in tango). The helm chart can be found in the SDP integration repository:

Inspect and update the values.yaml file, including the name of the pvc (pvc.name). Create a custom values file in yaml format, which you will be modifying for each scan. In that, have the following:

receiver:
  hostname: "proc-pb-test-20211111-00001-vis-receive-00-0.proc-pb-test-20211111-00001-vis-receive.<processing-namespace>"

command:
  - "emu-send"
  - "/mnt/data/test_data/scans/scan-<scan_id>.ms"
  - "-o"
  - "transmission.transport_protocol=tcp"
  - "-o"
  - "transmission.method=spead2_transmitters"
  - "-o"
  - "transmission.num_streams=1"
  - "-o"
  - "transmission.rate=10416667"
  - "-o"
  - "reader.num_repeats=1"
  - "-o"
  - "transmission.target_host=proc-pb-test-20211111-00001-vis-receive-00-0.proc-pb-test-20211111-00001-vis-receive.<processing-namespace>"
  - "-o"
  - "transmission.target_port_start=21000"
  - "-o"
  - "transmission.scan_id=<scan_id>"
  - "-o"
  - "transmission.telescope=mid"

pvc:
  name: <pvc-name>

Replace the following:

  • <processing_namespace>: namespace where the processing pods are running (in 2 places)

  • <scan_id>: the current scan ID (in 2 places)

  • <pvc_name>: the PersistentVolumeClaim that contains the data (usually test-pcv or shared)

In addition, pay attention to the following:

  • receiver.hostname and transmission.target_host: If you changed the processing block ID for the vis-receive, then update the relevant part of the string with the correct ID.

  • "/mnt/data/test_data/scans/scan-{scan_id}.ms", which needs to point to the file you are sending, in the sender-data pod.

Install the chart:

helm install cbf-send-1 ska-sdp-integration/tests/resources/charts/cbf-sender -f <custom-yaml>.yaml

(change the path and file name as needed)

Once the emulator finished running (the pod status shown as completed), back in itango, end the scan:

d.endScan()

Turn off the mock dishes:

for dev in devs:
    dish = DeviceProxy(dev)
    dish.EndScan()

6. Repeat for all scans

Repeat four more times steps 5:

  • Turn on the mock dishes, using the next scan_id

  • Run d.Scan with the right scan_id

  • Install the CBF emulator with an updated command: scan_id and MS name

  • End the scan and turn off the dishes

The pointing offset pipeline is triggered when the fifth endScan command is executed. It will read the MS data sets that can now be accessed in the receive-data pod, run its processing, then send the output offsets to an attribute of the QueueConnector Device. You can monitor the progress of the pipeline by looking at the logs of the pod in the processing namespace, e.g. pod name proc-pb-test-20211111-00000-pointing-offset*.

7. Retrieving data from the QueueConnector Device

The SDP QueueConnector device is configured via a data flow to read data from a kafka topic and save it to an attribute. The pointing-offset script sets this up (see Configuring data flows), defaulting to msgpack-numpy encoding and pointing_offset tango attribute prefix.

The actual attribute name will contain the dish ID, e.g. pointing_offset_SKA001. There is one attribute per dish. Each attribute contains spectrum data of three floating point values: [scan_id, xel_offset, el_offset]. The scan_id is the ID of the last scan of the given pointing observation. Cross-elevation (xel) and elevation (el) offsets are given in degrees.

After the pipeline run is complete, in itango3, run the following code to display the data (replace the dish ID as required):

q = DeviceProxy("test-sdp/queueconnector/01")
q.pointing_offset_SKA001

8. Finishing and clean up

Once the pointing pipeline finished, it will wait for the next set of pointing scans. To remove the deployment, you need to end the observation by returning the subarray to an EMPTY state (in itango):

d.end()
d.releaseAllResources()

If you want to remove all the resources you deployed, uninstall the following helm charts (run in the command line): five cbf-send-<id> charts, mock dish chart, SDP chart:

helm uninstall cbf-send-1  # repeat for 2, 3, 4, 5
helm uninstall mock-dish
make uninstall-sdp  # if you used the make targets to install

To remove the sender and receive pods, run:

kubectl delete -f <sender-pod-yaml> -n <control-system-namespace>
kubectl delete -f <receive-pod-yaml> -n <processing-namespace>

Additional: Capturing errors

The steps above assume the pointing pipeline executes successfully and outputs valid offset values for each antenna. However, sometimes the pipeline will output an error - see the pipeline documentation for details of how to retrieve the error message(s) in SDP.