Measurement Set Utilities and Format๏ƒ

This package provides core functionality for reading, writing, and managing Measurement Sets (MS). This document summarises the Measurement Set I/O capabilities of the package and details the supported table specifications of datasets.

Note

This document does not describe on how many of the values that are stored into the Measurement Set are obtained. For further details on data flows from CBF and SDP into Measurement Sets, read the vis-receive pipeline documentation.

Measurement Set Standard Schemas๏ƒ

The complete set of recognised Measurement Set schema specifications to this package is:

Warning

MSv4 is not yet supported and is planned for future releases

Writing a Measurement Set๏ƒ

Creation๏ƒ

Measurement Sets are written for a specific SKA subarray, scan, beam and spectral window configuration. Therefore, at creation time the following information is required:

  • A list of receptors making up the subarray.

  • Information about the scan for which the data is being observed.

  • Information about the beam to which the data belongs.

  • Information about the spectral window associated with the data.

In addition, the following optional information can be provided:

  • Baseline ordering information. If not provided, one is generated from the provided receptors, assuming auto-correlations and upper-triangular baselines.

  • Further observation metadata, including telescope name, observer and various identifiers.

  • The standard used to create a Measurement Set, defaults to MSv3๐›ฝ.

Note

Although this package in principle supports writing data for multiple beams and spectral window configurations into a single Measurement Set, this support was based on simplistic assumptions, and is not how Measurement Sets are written in production systems. This functionality is therefore considered deprecated, and not covered by this document. Future versions of this package will drop this support.

Creating Measurement Sets is done by instantiating the MSWriter class.

Adding data๏ƒ

This package allows users to continuously write visibility data into a Measurement Set. Visibility data has four dimensions: time, baseline, frequency and polarisation. When writing visibility data, it needs to be provided for a single time, all the baselines and all polarisations. Frequencies can be provided in chunks, or all at once. Values must also be provided for the corresponding exposure, interval, time, and UVW. Optionally user can also provide flags, weights, sigma and time centroid values. See MAIN table for details.

Pointing data can be provided by users at any point in time, and is appended at the end of the POINTING table. See POINTING table for details.

Output reference๏ƒ

All Measurement Set tables created and written by this package are detailed below. Because Measurement Sets are created for a specific scan, beam and spectra window, many of the values in the *_ID columns (e.g., MAIN/OBSERVATION_ID) are set to 0.

MAIN table๏ƒ

(MSv2 reference)

(MSv3 reference)

The MAIN table in the Measurement Set holds the bulk of the visibility data. Each row corresponds to a time and baseline combination, for which visibilities and their corresponding metadata are stored. \(N_c\) represents number of channels, \(N_p\) represents number of polarisations.

Column

Data type

Units

Measure

Value

ANTENNA1

int

Indices of first receptor according to baseline information provided at creation time, see Creation.

ANTENNA2

int

Indices of second receptor according to baseline information provided at creation time, see Creation.

ARRAY_ID

int

0

DATA

complex(\(N_c\), \(N_p\))

Jy

Written as provided

DATA_DESC_ID

int

0

EXPOSURE

double

s

Provided scalar value is repeated for all baseline rows

FEED1

int

0

FEED2

int

0

FIELD_ID

int

0

FLAG

bool(\(N_c\), \(N_p\))

Written as provided, defaulting to False. For missing data (i.e. frequencies/baseline/time combinations for which no data ends up written to the Measurement Set), will be set to True.

FLAG_CATEGORY

bool(*)

Unset

FLAG_ROW

bool

False

INTERVAL

double

s

Provided scalar value is repeated for all baseline rows

OBSERVATION_ID

int

0

PROCESSOR_ID

int

0

SCAN_NUMBER

int

Written as provided

SIGMA

float(\(N_c\))

  • Defaults to 1s

  • Otherwise written as provided

STATE_ID

int

0

TIME

double

s

epoch, ref: UTC

Provided scalar value is repeated for all baseline rows

TIME_CENTROID

double

s

epoch, ref: UTC

  • Defaults to same as TIME

  • Otherwise written as provided

UVW

double(3)

m

uvw, ref: ITRF

Written as provided

WEIGHT

float(\(N_c\), \(N_p\)) or float(\(N_p\))

  • Optional, defaults to 1s with frequency dimension

  • If MSV2 and input has frequency dimension, average across frequency before storing

  • If MSV3B and input has no frequency dimension, broadcast along the frequency dimension before storing

  • Otherwise written as provided

WEIGHT_SPECTRUM

float(\(N_c\), \(N_p\))

Only if MSV2 and input has frequency dimension, in which case it is written as provided

ANTENNA table๏ƒ

(MSv2 reference)

The ANTENNA table is populated with the contents of the list of receptors provided at creation time (see Creation). One row is created per receptor as follows:

Column

Data type

Units

Measure

Value

DISH_DIAMETER

double(3)

m

Receptor dish diameter

FLAG_ROW

bool

False

MOUNT

string

Receptor mount type

NAME

string

Receptor name

OFFSET

double(3)

m

position, ref: ITRF

[0, 0, 0]

POSITION

double(3)

m

position, ref: ITRF

Receptor X/Y/Z position

STATION

string

Same as NAME

TYPE

string

"GROUND-BASED"

DATA_DESCRIPTION table๏ƒ

(MSv2 reference)

A single DATA_DESCRIPTION row is created:

Column

Data Type

Units

Measure

Value

FLAG_ROW

bool

False

POLARIZATION_ID

int

0

SPECTRAL_WINDOW_ID

int

0

FEED table๏ƒ

(MSv2 reference)

The FEED table is populated using the receptors provided at creation time (see Creation). One row is created per receptor as follows:

Column

Data Type

Units

Measure

Value

ANTENNA_ID

int

Number of receptor per ANTENNA table

BEAM_ID

int

-1

BEAM_OFFSET

double(2, 2)

rad

direction, ref: J2000

[[0, 0], [0, 0]]

FEED_ID

int

0

INTERVAL

double

s

Observing period spanned by the data recorded in the MAIN table

NUM_RECEPTORS

int

2

POLARIZATION_TYPE

string(2)

["X", "Y"]

POL_RESPONSE

complex(2, 2)

[[1 + 0j, 0 + 0j], [0 + 0j, 1 + 0j]]

POSITION

double(3)

m

position, ref: ITRF

[0, 0, 0]

RECEPTOR_ANGLE

double(2)

rad

[0, 0]

SPECTRAL_WINDOW_ID

int

0

TIME

double

s

epoch, ref: UTC

Mid-point of the observing period spanned by the data recorded in the MAIN table

FIELD table๏ƒ

(MSv2 reference)

A single FIELD row is created with the contents of the field associated to the beam provided at creation time (see Creation).

Column

Data Type

Units

Measure

Value

CODE

string

""

DELAY_DIR

double(2, 1)

rad

direction, ref: J2000

Same as PHASE_DIR

FLAG_ROW

bool

False

NAME

string

Fieldโ€™s ID

NUM_POLY

int

0

PHASE_DIR

double(2, 1)

rad

direction, ref: J2000

RA/Dec of the fieldโ€™s phase direction

REFERENCE_DIR

double(2, 1)

rad

direction, ref: J2000

Same as PHASE_DIR

SOURCE_ID

int

-1

TIME

double

s

epoch, ref: UTC

Time at which the row is written

FLAG_CMD table๏ƒ

(MSv2 reference)

The FLAG_CMD table is created but it is always empty.

HISTORY table๏ƒ

(MSv2 reference)

A single HISTORY row is created. now is calculated when the Measurement Set is created:

Column

Data Type

Units

Measure

Value

APPLICATION

string

"cbf-sdp-emulator" (for historical reasons, subject to change)

APP_PARAMS

string(*)

sys.argv[1:]

CLI_COMMAND

string(*)

sys.argv

MESSAGE

string

"Measurement Set created at <TIME-in-ISO-format>"

OBJECT_ID

int

0

OBSERVATION_ID

int

-1

ORIGIN

string

"cbf-sdp-emulator" (as above)

PRIORITY

string

"NORMAL"

TIME

double

s

epoch, ref: UTC

Time at which the row is written

OBSERVATION table๏ƒ

(MSv2 reference)

A single OBSERVATION row is created with the metadata provided at Measurement Set creation time (see Creation).

Column

Data Type

Units

Measure

Value

FLAG_ROW

bool

False

LOG

string(*)

Unset

OBSERVER

string

Written as provided

PROJECT

string

Written as provided

RELEASE_DATE

double

s

epoch, ref: UTC

0

SCHEDULE

string(*)

Written as provided

SCHEDULE_TYPE

string

""

SKA_EXECUTION_BLOCK_ID

string

The SKA SDP Execution Block ID, written as provided

SKA_PROCESSING_BLOCK_ID

string

The SKA SDP Processing Block ID, written as provided

SKA_SCAN_ID

int64

The SKA Scan ID, written as provided

TELESCOPE_NAME

string

Written as provided

TIME_RANGE

double(2)

s

epoch, ref: UTC

Start and end time of the observing period spanned by the actual recorded data in MAIN

POINTING table๏ƒ

(MSv2 reference)

The POINTING table is populated from the pointing information provided by users. When pointing information is provided, each row is populated as follows:

Column

Data Type

Units

Measure

Value

ANTENNA_ID

int

Written as provided

DIRECTION

double(1, 2)

rad

direction, ref: AZEL

Written as provided

INTERVAL

double

s

0

NAME

string

Written as provided

NUM_POLY

int

0

POINTING_OFFSET

double(1, 2)

rad

direction

Written as provided

SOURCE_OFFSET

double(1, 2)

rad

direction

Written as provided

TARGET

double(1, 2)

rad

direction, ref: AZEL

Written as provided

TIME

double

s

epoch, ref: UTC

Written as provided

TIME_ORIGIN

double

s

epoch, ref: UTC

Same as TIME

TRACKING

bool

Written as provided

POLARIZATIONS table๏ƒ

(MSv2 reference)

A single POLARIZATIONS row is creeted with the contents of the polarisation configuration associated to the beam for which data is stored (see Creation).

Column

Data Type

Units

Measure

Value

CORR_PRODUCT

int(2, 4)

[[0, 0], [0, 1], [1, 0], [1, 1]]

CORR_TYPE

int(4)

[9, 10, 11, 12] (corresponding to XX,XY,YX,YY)

FLAG_ROW

bool

False

NUM_CORR

int

4

PROCESSOR table๏ƒ

(MSv2 reference)

The PROCESSOR table has a single row with the following entries:

Column

Data Type

Units

Measure

Value

FLAG_ROW

bool

False

MODE_ID

int

-1

SUB_TYPE

string

"<telescope_name>-CBF", where telescope_name is part of the metadata provided at creation time (see Creation).

TYPE

string

CORRELATOR

TYPE_ID

int

-1

SPECTRAL_WINDOW table๏ƒ

(MSv2 reference)

A single SPECTRAL_WINDOW row is created with the contents of the spectral window information for which data is stored (see Creation).

Column

Data Type

Units

Measure

Value

CHAN_FREQ

double(NUM_CHAN)

Hz

frequency

Centre frequencies of the spectral window channels

CHAN_WIDTH

double(NUM_CHAN)

Hz

Width of the spectral window channels

EFFECTIVE_BW

double(NUM_CHAN)

Hz

Effective bandwidth of the spectral window channels

FLAG_ROW

bool

False

FREQ_GROUP

int

0

FREQ_GROUP_NAME

string

""

IF_CONV_CHAIN

int

0

MEAS_FREQ_REF

int

5 (i.e., TOPO)

NAME

string

Spectral Window ID

NET_SIDEBAND

int

0

NUM_CHAN

int

Number of channels in the spectral window

REF_FREQUENCY

double

Hz

frequency

Lower bound of the frequency of the first channel in the spectral window

RESOLUTION

double(NUM_CHAN)

Hz

Same as EFFECTIVE_BW

TOTAL_BANDWIDTH

double

Hz

Difference between the upper bound of the frequency of the last channel in the spectral window and REF_FREQUENCY

STATE table๏ƒ

(MSv2 reference)

A single STATE row is created with the following contents:

Column

Data Type

Units

Measure

Value

CAL

double

K

0

FLAG_ROW

bool

False

LOAD

double

K

0

OBS_MODE

string

Comma-separated list of scan intents of the scan provided at creation time (see Creation).

REF

bool

False

SIG

bool

False

SUB_SCAN

int

0