Entities and Operations

Note

All code examples below work on a txn Transaction object created like this:

for txn in Config(backend="memory").txn():
    # example code

Overview

The SDP Configuration Database manages entities of different types. Each type of entity is stored under a well-determined prefix or path. Individual entities within a certain type are identified by a key, which is composed by one or more key parts. Each key part accepts values that are constrained by a pattern. Additionally, entities of certain types have an associated owner and state. An entity is said to be alive if its owner entry exists in the database.

The Transaction class provides a series of attributes (e.g., Transaction.deployment) through which users are meant to operate on the different entities that are stored in the database. This section describes how this interaction takes place.

Note that Transaction also offers a set of methods for some of these operations (e.g., Transaction.create_deployment()). Those methods are deprecated, and will be shortly after removed, so using the attributes should be preferred.

Entity models

Note

Support for entity model classes has been added, but not all entities have been modelled. Check the Supported entities section for information of which entity types are currently modelled. Long-term the aim is to model all of them.

Historically the SDP Configuration Database has dealt with plain dict objects as entity values. On top of that, two classes modelling ProcessingBlock and Deployment had been provided to users allowing for conversion from/to dictionaries for operating with the (old, dict-based) Transaction API.

Entities now are expected to be modelled as individual classes using pydantic. This has several benefits:

  • Provides users with a well-defined in-memory representation for each entity type.

  • Makes it possible to easily distinguish between entities of different types.

  • Allows to model restrictions on each field, triggering validation errors when invalid content is provided.

When an entity is modelled using a pydantic model class, the corresponding operations described below are defined in terms of instances of that class rather than plain dictionaries. The model class also defines the entity key as a field that should be modelled itself with pydantic.

Operating on entities

To access entities of a particular type users need to operate on the corresponding attribute of Transaction. See Supported entities for a list of all supported entities and their corresponding attributes. For example, to operate on subarray entities one selects the subarray attribute:

>>> txn.subarray  
<ska_sdp_config.operations.subarray.SubarrayOperations object at ...>

These attributes provide operations on entities of that type. There are two types of operations: those that act over individual entities, and those that operate over the whole set of entities for a given entity type.

The self attribute

The Transaction class has a special self attribute pointing to the operations for the entity pointed to by the owned_entity argument of the Config constructor. The argument is a two-tuple with:

  • The entity type as a Transaction attribute name

  • The full key of the entity in question.

See Supported entities for references for both values.

For example, the following Config class is configured to get convenient access to the test component:

cfg2 = Config(backend="memory", owned_entity=("component", "test"))
for txn in cfg2.txn():
   print(txn.self.path)

/component/test

If no identity is provided (like in the case of the Config object used in these examples), self will be None.

>>> txn.self is None
True

dict entities

Indexing

To operate on an individual entity modelled by a plain dictionary users need to “index” into the entity type to identify the entity of interest.

Note

The main exception to this is the controller attribute. Since there is a single LMC Controller in the system, this attribute gives direct access to the operations over the (single) entity in the Database without the need to “index” into it.

This indexing is done by invoking the corresponding attribute with the key information that identifies an individual entity. For example, to operate over the subarray identified by "01" one calls the subarray attribute with the key:

>>> txn.subarray(key="01")
<EntityOperations path="/lmc/subarray/01">
>>> txn.subarray(subarray_id="01")
<EntityOperations path="/lmc/subarray/01">
>>> txn.subarray("01")
<EntityOperations path="/lmc/subarray/01">

Note how here we can provide either:

  • The special key keyword argument. This specifies the full key as a single value (see Overview for the distinction between key and key parts). For entities with a single key part (like subarrays) this is no different than giving the key part value, but for entities with multiple key parts it means that a single value can be used to index. If key is given, no other keyword arguments are accepted.

  • The subarray_id keyword argument. This keyword argument is accepted because its name matches one of the key parts (and the only one).

  • Since subarrays have only a single key part, the value for this single key part can also be given as a single positional parameter.

Incorrect indexing results in errors raised even before trying to hit the database:

>>> txn.subarray(key="01", subarray_id="02")
Traceback (most recent call last):
ValueError: 'key' cannot be combined with other keyword arguments
>>> txn.subarray(not_the_subarray_id="01")
Traceback (most recent call last):
ValueError: "not_the_subarray_id" is not a valid key part under /lmc/subarray
>>> txn.subarray("01", "02")
Traceback (most recent call last):
ValueError: Only single positional argument can be given

Note also that providing key or key part values that don’t adhere to the key part patterns supported by the entity also results in errors:

>>> txn.subarray(key="not_a_number")
Traceback (most recent call last):
ska_sdp_config.operations.entity_operations.InvalidKey: Invalid key='not_a_number' for pattern=re.compile('^(?P<subarray_id>[0-9-_]+)$')
>>> txn.subarray(subarray_id="not_a_number")
Traceback (most recent call last):
ValueError: "not_a_number" is not a valid value for key part "subarray_id" under /lmc/subarray
>>> txn.subarray("not_a_number")
Traceback (most recent call last):
ValueError: "not_a_number" is not a valid value for key part "subarray_id" under /lmc/subarray

See the API documentation to see all patterns in detail.

Entity operations

Once an entity has been indexed into, all its operations become available:

  • The key, key_parts and path attributes inform users of the key, key parts (names and values) and database path for the particular entity.

  • The create(), get(), update() and delete() methods allow users to perform basic CRUD operations.

  • The create_or_update() and exists() methods are small utilities built on top of the basic methods.

Additionally, if an entity type might give access to two extra attributes:

pydantic entities

Indexing

For entities that are modelled via pydantic, manual indexing with strings as shown above is still possible, but not encouraged. Instead, high-level functions are provided that take a key, or a value, or either, and they internally perform the necessary indexing. See Entity operations for more information.

To enforce this expected API usage, users must use the index_by_key_parts() method explicitly instead of using the magic __call__ function when manual access is required:

>>> from ska_sdp_config.entity import Deployment
>>> txn.deployment.index_by_key_parts(key="my-deployment")
<EntityOperations path="/deploy/my-deployment">

Entity operations

If an entity is modelled using pydantic then all the operations above are also available directly at the corresponding Transaction attribute, and receiving a entity value as the only argument. For writing operations this is convenient, since the full value needs to be given anyway. For the rest, only the key parts from the entity are extracted and used, with the rest ignored.

For example, the following are equivalent:

>>> from ska_sdp_config.entity import Deployment
>>> deployment1 = Deployment(key="my-deployment1", kind="helm", args={})
>>> deployment2 = Deployment(key="my-deployment2", kind="helm", args={})
>>> deployment3 = Deployment(key="my-deployment3", kind="helm", args={})

# Indexing happens as part of the operation
# Here we can index by key or by value
>>> txn.deployment.exists(deployment1)
False
>>> txn.deployment.exists("my-deployment3")
False

# Some operations require a full value; indexing happens still by key only
>>> txn.deployment.create(deployment1)
>>> txn.deployment.create(deployment2)
>>> txn.deployment.create(deployment3)

# They all worked
>>> len(txn.deployment.list_keys())
3

# Reading via top-level get()
>>> all(txn.deployment.get(dpl) for dpl in (deployment1, deployment2, deployment3))
True

Collective operations

Other operations are performed over the entire entity type. As such, these operations are directly available on the corresponding Transaction attribute. The only operations currently supported over a whole entity type is querying.

Note

Again, the main exception to this is the controller attribute. Since there is a single LMC Controller in the system, no querying operations are supported.

To query entities of a given type, users invoke the query_keys(), list_keys(), query_values() or list_values() methods on the corresponding attribute. By default they all return all keys/values for the given entity type. Constrains can be given in the form of keyword arguments that must match the entity’s key parts.

For entities that are modelled with pydantic, the returned keys and values are instances of the corresponding model. If entities are modelled with plain dictionaries, keys are :-delimited strings and values are dictionaries.

For example, this code queries all subarray keys:

>>> txn.subarray.list_keys()
[]
>>> txn.subarray("01").create({"value": 1})
>>> txn.subarray("02").create({"value": 2})
>>> sorted(txn.subarray.list_keys())
['01', '02']

If we wanted keys and values for all subarrays whose IDs start with 0 we’d write:

>>> txn.subarray.list_values(subarray_id_prefix="0")
[('01', {'value': 1}), ('02', {'value': 2})]

The next example creates a whole set of scripts, which are modelled with pydantic, then iterates over the first 5 vis-receive ones scripts with version 1.3.*:

from ska_sdp_config.entity import Script
from itertools import islice, product

# Create loads of scripts
names = ("vis-receive", "test-recv-addresses")
for name, major, minor, patch in product(names, range(10), range(10), range(10)):
   full_version = f"{major}.{minor}.{patch}"
   key = Script.Key(name=name, version=full_version, kind="realtime")
   image = f"{name}:{full_version}"
   script = Script(key=key, image=image)
   txn.script.create(script)
print(f"There are {len(txn.script.list_keys())} scripts")

# Query only some
print(f"First five vis-receive scripts with version 1.3.* follow:")
lazy_query = txn.script.query_values(name="vis-receive", version_prefix="1.3.")
for key, value in islice(lazy_query, 5):
   print(f"{key=}, {value=}")

There are 2000 scripts
First five vis-receive scripts with version 1.3.* follow:
key=Key(kind='realtime', name='vis-receive', version='1.3.0'), value=Script(key=Key(kind='realtime', name='vis-receive', version='1.3.0'), image='vis-receive:1.3.0')
key=Key(kind='realtime', name='vis-receive', version='1.3.1'), value=Script(key=Key(kind='realtime', name='vis-receive', version='1.3.1'), image='vis-receive:1.3.1')
key=Key(kind='realtime', name='vis-receive', version='1.3.2'), value=Script(key=Key(kind='realtime', name='vis-receive', version='1.3.2'), image='vis-receive:1.3.2')
key=Key(kind='realtime', name='vis-receive', version='1.3.3'), value=Script(key=Key(kind='realtime', name='vis-receive', version='1.3.3'), image='vis-receive:1.3.3')
key=Key(kind='realtime', name='vis-receive', version='1.3.4'), value=Script(key=Key(kind='realtime', name='vis-receive', version='1.3.4'), image='vis-receive:1.3.4')

Supported entities

The table below summarises the supported entities and their operations. Note that for those entities modelled via pydantic there is a model class, while for plain dict entities there are key parts.

Entities supported by Transaction

Attribute

Prefix/path

Model class

Key parts

Has ownership?

Has state?

component

/component

–

component_name

True

False

controller

/controller

–

–

False

False

deployment

/deploy

Deployment

–

False

True

execution_block

/eb

–

eb_id

False

False

processing_block

/pb

ProcessingBlock

–

True

True

script

/script

Script

–

False

False

subarray

/subarray

–

subarray_id

False

False

Arbitrary path access

On top of the entity-specific access provided by Transaction, there is also an extra arbitrary attribute that allows operating over any arbitrary path in the database. This is useful for tools performing operations over generic paths, or to quickly experiment with storing data under certain paths before committing to adding a new entity in this package.

To access a particular path users need to index into arbitrary as described in Indexing, although here a single positional argument indicates the path. The result of that call is the set of operations described in Entity operations.

The following example uses arbitrary path access to first write into "/my_path", then obtain the value that has just been written:

with warnings.catch_warnings(record=True) as warns:
    txn.arbitrary("/my_path").create({"my_value": 3})
    print(txn.arbitrary("/my_path").get())
print(f"{len(warns)} warnings generated")

{'my_value': 3}
2 warnings generated

The example above is accessing /my_path, which is not a path that is handled by any of the supported entities. In those cases, a warning is raised so that users are aware of this unofficial access.

Note

This arbitrary path support shouldn’t be abused as a long-term solution if the goal is to introduce a new entity to the system. New entities should be explicitly added to this package and used instead.

Accessing known paths raises no warnings:

txn.subarray("01").create({"subarray_id": "01"})
with warnings.catch_warnings(record=True) as warns:
    print(f'Subarray: {txn.arbitrary("/lmc/subarray/01").get()}')
print(f"{len(warns)} warnings generated")

Subarray: {'subarray_id': '01'}
0 warnings generated