Usage Guide
Data
Telescope model data is stored as objects identified as keys. Each key takes the following form:
[domain]/([sub-domain]/)*[name].[type]
Where
[domain]/
specifies the coarse top-level telescope model data domain.[sub-domain]/
provides further hierarchical data sub-categories[name]
associates a name with the telescope model data.[type]
identifies the file type, which is used to identify the kind of file contents. The library currently supportsjson
andyaml
.
Names should be chosen for being self-describing and stable long-term. Top-level domains:
environment/... # Environment telescopes are deployed in
instrument/... # Telescopes and their equipment
software/... # Software deployed to the telescopes
To retrieve a particular piece of data from the telescope model,
csimply construct an ska_telmodel.data.TMData
object and
use the []
operator to access:
from ska_telmodel.data import TMData
tmdata = TMData()
print(tmdata['instrument/ska1_low/layout/low-layout.json'].get_dict())
This works because the telescope model comes with a number of default
sources that will be checked for matching telescope model
data. ska_telmodel.data.TMObject.get_dict()
automatically
parses and converts JSON and YAML documents, but you can also get the
raw data using ska_telmodel.data.TMObject.get()
, or open or
copy the contents as a file using
ska_telmodel.data.TMObject.open()
or
ska_telmodel.data.TMObject.copy()
respectively.
To get an idea what is contained in a particumlar part of the telescope
model data tree, simply iterate over it (equivalent to
ska_telmodel.cli.cmd_ls()
):
from ska_telmodel.data import TMData
tmdata = TMData()
for key in tmdata['instrument']:
print(key)
Logically, the []
operator constructs a sub-set of all
telescope model data. If the key is a valid object name (i.e. has an
extension, so contains a '.'
) this subset is assumed to contain
only a single object, and the []
operator will return a
ska_telmodel.data.TMObject
instance.
Data Sources
Telescope model data can be retrieved from a list of sources, which can
be specified to the ska_telmodel.data.TMData
constructor,
using the SKA_TELMODEL_SOURCES
environment variable or left to
in-built ska_telmodel.data.sources.DEFAULT_SOURCES
. Each
source is represented as an URI that specifies the source of truth for
some portion of telescope model data.
The following telescope model data backends are currently supported:
mem://?[key1]=[value1]&[key2]=[value2]
(seeska_telmodel.data.backend.MemoryBackend
)
file://[absolute path]
(seeska_telmodel.data.backend.FilesystemBackend
)
gitlab://[gitlab server]/[project name]?[branch]#[directory]
(seeska_telmodel.data.backend.GitlabBackend
)
car://[gitlab server]/[project name]?[branch]#[directory]
(seeska_telmodel.data.backend.CARBackend
)
The simplest example would be to utilise
ska_telmodel.data.backend.MemoryBackend
to set a key
directly:
from ska_telmodel.data import TMData
tmdata = TMData(['mem://?test.txt=test_data'])
print(tmdata['test.txt']) # -> b"test_data\n"
This can also be configured using environment variables:
import os
from ska_telmodel.data import TMData
os.environ['SKA_TELMODEL_SOURCES'] = 'mem://?test.txt=test_data'
tmdata = TMData()
print(tmdata['test.txt']) # -> b"test_data\n"
You would typically do this from outside your program, see the
documentation for ska_telmodel.cli.cmd_pin()
and
ska_telmodel.cli.cmd_cp()
for examples.
A more complex example would be to retrieve data from Gitlab using
ska_telmodel.data.backend.GitlabBackend
:
from ska_telmodel.data import TMData
gl_uri = 'car://gitlab.com/ska-telescope/ska-telmodel?master#tmdata'
tmdata = TMData([gl_uri])
print(tmdata['software/tango/dsh/DishManager.yaml'])
This will retrieve data directly from the telescope model library repository.
Note that external telescope model data sources using
ska_telmodel.data.backend.GitlabBackend
or
ska_telmodel.data.backend.CARBackend
will cache data locally in
order to prevent repeated requests to servers. This means that if we reference
a Gitlab branch (like master
in the example), the telescope model data in
the cache might go out of sync with the server.
This is intentional, as it means that we provide a consistent view of telescope
model data as long as possible. It is generally best to use “pinned” sources
(see ska_telmodel.cli.cmd_pin()
), but in day-to-day usage, you can
simply use the -U
flag as documented in Command Line Usage or (less preferably) the
update
option to ska_telmodel.data.TMData
to occassionally
refresh the cache as needed. The library will occassionally check for and warn
about stale caches.
Adding a New Gitlab Data Source
If you want others to be able to view data in your GitLab repository
using ska_telmodel.data.backend.GitlabBackend
or
ska_telmodel.data.backend.CARBackend
, first you
will need to place the data you wish to export in a top level
dictionary in your repo named tmdata
. For example:
/tmdata/instrument/mccs_configuration/config_file_low.json
/tmdata/instrument/mccs_configuration/config_file_mid.json
Important to note:
Try to use a directory structure that is compatible with domains (see above) and is reasonably likely to remain stable.
Currently only .json and .yaml files are accepted, and you should have schemas associated with them.
Next add telescope model data support to your top-level Makefile
as documented in https://developer.skao.int/projects/ska-cicd-makefile/en/latest/README.html :
include .make/tmdata.mk
At this point you should be able to verify that make tmdata-package
will
result in both a tmtree.json
and a tmdata.tar.gz
file getting created
in build/tmdata
. Next add the packaging and publishing stage to your Gitlab
pipeline by adding the following lines to the .gitlab-ci.yml
file as documented in
https://developer.skao.int/projects/templates-repository/en/latest/README.html
:
- project: 'ska-telescope/templates-repository'
file: 'gitlab-ci/includes/tmdata.gitlab-ci.yml'
Now once you merge these changes into the main
branch, others will be able
to access this data by specifying your repository as the source:
$ ska-telmodel --sources=car:mccs/ska-low-mccs?main ls
instrument/mccs_configuration/config_file_low.json
instrument/mccs_configuration/config_file_mid.json
Branches other than main
will also work, just adjust the URL
accordingly. However by default the GitLab pipeline will only
upload the TMData package to the artefact repository on the main branch
as well as tags.
If you want the data to be accessible without passing command line
parameters, make a merge request to the ska-telmodel repository
( https://gitlab.com/ska-telescope/ska-telmodel ) that adds your
repository address the src/ska_telmodel/data/source.py
file.
This makes your telescope model data available “globally”:
$ ska_telmodel ls
[...]
instrument/mccs_configuration/config_file_low.json
instrument/mccs_configuration/config_file_mid.json
Schemas
Schemas check JSON-like objects for conformance, e.g. nested
dictionaries containing primitives and lists. They especially have a
JSON schema representation - though
ska_telmodel.schema.validate()
will generally implement more
thorough checks.
All schemas are identified by an URI of the form:
https://schema.skao.int/ska-[subsystem]-[interface]/[major].[minor]
The entire URI should be lower-case alphanumerical. The
[subsystem]
identifies the leading party for maintaining the
schema, and [interface]
the concrete interface
implemented. Depending on context, this might either be data produced
or consumed by the sub-system in question.
Versioning should follow semantic versions: Changes in minor version indicate backwards-compatible changes such as adding new fields or otherwise introducing additional accepted schemas. Changes that break backwards compatibility should change the major version.
You can use the URIs with ska_telmodel.schema.validate()
to
validate data:
from ska_telmodel.data import TMData
from ska_telmodel.schema import validate
uri = "https://schema.skao.int/ska-telmodel-layout-location/0.0"
layout_dict = TMData()['instrument/ska1_low/layout/low-layout.json'].get_dict()
validate(uri, layout_dict)
Furthermore you can use ska_telmodel.schema.example_by_uri()
to retrieve examples of certain schemas (which are replicated in the
schema section of this documentation).