.. _entities_ops:

Entities and Operations
=======================

.. testsetup:: *

   from ska_sdp_config import Config
   config = Config(backend="memory", global_prefix="/docs_tests")
   config.backend.delete("/docs_tests", recursive=True, must_exist=False)
   txn_iter = iter(config.txn())
   txn = next(txn_iter)

.. testcleanup:: *

   config.close()

.. note::

   All code examples below work on a ``txn`` |Txn| object
   created like this:

   .. code-block:: python

      for txn in Config(backend="memory").txn():
          # example code

.. _overview:

Overview
--------

.. py:currentmodule:: ska_sdp_config.config


The SDP Configuration Database manages *entities* of different *types*.
Each type of entity is stored under a well-determined *prefix* or *path*.
Individual entities within a certain type are identified by a *key*,
which is composed by one or more *key parts*.
Each key part accepts values that are constrained by a *pattern*.
Additionally, entities of certain types
have an associated *owner* and *state*.
An entity is said to be *alive*
if its owner entry exists in the database.

The |Txn| class provides a series of attributes (e.g., :attr:`Transaction.deployment`)
through which users are meant to operate
on the different entities that are stored in the database.
This section describes how this interaction takes place.


Entity models
-------------

.. note::

   Support for entity model classes has been added,
   but not all entities have been modelled.
   Check the :ref:`supported_entities` section
   for information of which entity types
   are currently modelled.
   Long-term the aim is to model all of them.

Historically the SDP Configuration Database has dealt
with plain ``dict`` objects as entity values.
On top of that,
two classes modelling :class:`~ska_sdp_config.entity.pb.ProcessingBlock`
and :class:`~ska_sdp_config.entity.deployment.Deployment` had been provided to users
allowing for conversion from/to dictionaries
for operating with the (old, ``dict``-based) :class:`Transaction` API.

Entities now are expected to be modelled
as individual classes
using `pydantic <https://docs.pydantic.dev/latest/>`_.
This has several benefits:

 * Provides users with a well-defined in-memory representation
   for each entity type.
 * Makes it possible to easily distinguish
   between entities of different types.
 * Allows to model restrictions on each field,
   triggering validation errors when invalid content is provided.

When an entity is modelled using a ``pydantic`` model class,
the corresponding operations described below
are defined in terms of instances of that class
rather than plain dictionaries.
The model class also defines
the entity key as a field
that should be modelled itself with ``pydantic``.


Operating on entities
---------------------

To access entities of a particular type
users need to operate on the corresponding attribute of |Txn|.
See :ref:`supported_entities` for a list of all supported entities
and their corresponding attributes.
For example, to operate on processing block entities
one selects the :attr:`~Transaction.processing_block` attribute:

.. doctest:: overview

    >>> txn.processing_block  # doctest: +ELLIPSIS
    <ska_sdp_config.operations.processing_block.ProcessingBlockOperations object at ...>


These attributes provide *operations*
on entities of that type.
There are two types of operations:
those that act over individual entities,
and those that operate over the whole set of entities
for a given entity type.


The ``self`` attribute
^^^^^^^^^^^^^^^^^^^^^^

The |Txn| class has a special :attr:`~Transaction.self` attribute
pointing to the operations for the entity pointed to
by the ``owned_entity`` argument
of the :class:`Config` constructor.
The argument is a two-tuple with:

 * The entity type as a :class:`Transaction` attribute name
 * The full key of the entity in question.

See :ref:`supported_entities` for references for both values.

For example, the following :class:`Config` class
is configured to get convenient access
to the ``test`` :attr:`~Transaction.component`:

.. testcode:: self

   cfg2 = Config(backend="memory", owned_entity=("component", "test"))
   for txn in cfg2.txn():
      print(txn.self.path)

.. testoutput:: self

   /component/test


If no identity is provided
(like in the case of the :class:`Config` object used in these examples),
:attr:`~Transaction.self` will be ``None``.

.. doctest:: no-self

   >>> txn.self is None
   True


``dict`` entities
-----------------

.. _indexing_dict:

Indexing
^^^^^^^^

To operate on an individual entity
modelled by a plain dictionary
users need to "index" into the entity type
to identify the entity of interest.

This indexing is done by invoking the corresponding attribute
with the key information that identifies an individual entity.
For example, to operate over the Controller device component
identified by ``"lmc-controller"`` one calls the
:attr:`~Transaction.component` attribute with the key:

.. doctest:: indexing

   >>> txn.component(key="lmc-controller")
   <EntityOperations path="/component/lmc-controller">
   >>> txn.component(component_name="lmc-controller")
   <EntityOperations path="/component/lmc-controller">
   >>> txn.component("lmc-controller")
   <EntityOperations path="/component/lmc-controller">

Note how here we can provide either:

* The special ``key`` keyword argument.
  This specifies the *full* key as a single value
  (see :ref:`overview` for the distinction between
  *key* and *key parts*).
  For entities with a single key part (like component)
  this is no different
  than giving the key part value,
  but for entities with multiple key parts
  it means that a single value can be used to index.
  If ``key`` is given,
  no other keyword arguments are accepted.
* The ``component_name`` keyword argument.
  This keyword argument is accepted because its name
  matches one of the key parts (and the only one).
* Since components have only a single key part,
  the value for this single key part can also be given
  as a single positional parameter.

Incorrect indexing results in errors raised
even before trying to hit the database:

.. doctest:: indexing

   >>> txn.component(key="lmc-controller", component_name="lmc-subarray-01")
   Traceback (most recent call last):
   ValueError: 'key' cannot be combined with other keyword arguments
   >>> txn.component(wrong_key_part="lmc-controller")
   Traceback (most recent call last):
   ValueError: "wrong_key_part" is not a valid key part under /component
   >>> txn.component("lmc-controller", "lmc-subarray-01")
   Traceback (most recent call last):
   ValueError: Only single positional argument can be given

Note also that providing key or key part values
that don't adhere to the key part patterns
supported by the entity
also results in errors:

.. doctest:: indexing

   >>> txn.component(key="invalid&char")
   Traceback (most recent call last):
   ska_sdp_config.operations.entity_operations.InvalidKey: Invalid key='invalid&char' for pattern=re.compile('^(?P<component_name>[a-zA-Z0-9_-]+)$')
   >>> txn.component(component_name="invalid&char")
   Traceback (most recent call last):
   ValueError: "invalid&char" is not a valid value for key part "component_name" under /component
   >>> txn.component("invalid&char")
   Traceback (most recent call last):
   ValueError: "invalid&char" is not a valid value for key part "component_name" under /component

See the :doc:`API documentation <api/operations>`
to see all patterns in detail.


.. _entity_ops:

Entity operations
^^^^^^^^^^^^^^^^^

.. py:currentmodule:: ska_sdp_config.operations.entity_operations

Once an entity has been indexed into,
all its operations become available:

* The :attr:`~BaseEntityOperations.key`,
  :attr:`~BaseEntityOperations.key_parts` and
  :attr:`~BaseEntityOperations.path` attributes
  inform users of the key, key parts (names and values)
  and database path for the particular entity.
* The :meth:`~BaseEntityOperations.create`,
  :meth:`~BaseEntityOperations.get`,
  :meth:`~BaseEntityOperations.update` and
  :meth:`~BaseEntityOperations.delete` methods
  allow users to perform basic
  `CRUD <https://en.wikipedia.org/wiki/Create,_read,_update_and_delete>`_ operations.
* The :meth:`~BaseEntityOperations.create_or_update` and
  :meth:`~BaseEntityOperations.exists` methods
  are small utilities built on top of the basic methods.

Additionally, if an entity type might give access to two extra attributes:

* :attr:`~OwnedEntityOperationsMixIn.ownership` is available on entity types
  that have an owner
  (that is, a process in the system that is in charge of them).
  It offers the basic
  :meth:`~OwnershipOperations.take` and
  :meth:`~OwnershipOperations.get` methods,
  together with the utility
  :meth:`~OwnershipOperations.is_owned` and
  :meth:`~OwnershipOperations.is_owned_by_this_process` ones.
  Additionally, the :meth:`~OwnedEntityOperationsMixIn.is_alive`
  and :meth:`~OwnedEntityOperationsMixIn.take_ownership_if_not_alive` methods
  are also provided.

* :attr:`~StatefulEntityOperationsMixIn.state` is available on entity types
  that have an associated state.
  It offers the basic
  :meth:`~StateOperations.create`,
  :meth:`~StateOperations.update` and
  :meth:`~StateOperations.get` methods.

.. _pydantic_entities:

``pydantic`` entities
---------------------

.. _indexing_pydantic:

Indexing
^^^^^^^^

For entities that are modelled via ``pydantic``,
manual indexing with strings as shown above
is still possible, but not encouraged.
Instead, high-level functions
are provided that take a key, or a value, or either,
and they internally perform the necessary indexing.
See :ref:`entity_ops_pydantic` (below) for more information.

To enforce this expected API usage when performing manual indexing,
users must use the :meth:`~CollectiveEntityOperations.index_by_key_parts` method
explicitly instead of using the magic ``__call__`` function:

.. doctest:: indexing-with-model

   >>> from ska_sdp_config.entity import Deployment
   >>> txn.deployment.index_by_key_parts(key="my-deployment")
   <EntityOperations path="/deploy/my-deployment">


.. _entity_ops_pydantic:

Entity operations
^^^^^^^^^^^^^^^^^

If an entity is modelled using ``pydantic``
then all of the operations listed above for ``dict`` entities are also available
directly at the corresponding |Txn| attribute.
and require a entity value (or key) as argument.
For writing operations it is convenient to specify the entity itself as argument,
since the full value **needs** to be given anyway.
For the rest, only the key parts from the entity are extracted and used,
with the rest ignored, and in those cases the methods can accept the key as argument instead.

For example, the following shows the different indexing methods with the ``deployment`` entity:

.. doctest:: entity-ops-for-pydantic

   >>> from ska_sdp_config.entity import Deployment
   >>> deployment1 = Deployment(key="my-deployment1", kind="helm", args={})
   >>> deployment2 = Deployment(key="my-deployment2", kind="helm", args={})

   # Indexing happens as part of the operation
   # Here we can index by key or by value to check if the deployment exists
   >>> txn.deployment.exists("my-deployment1")
   False
   >>> txn.deployment.exists(deployment2)
   False

   # Writing operations require a full value; indexing still happens by key only
   >>> txn.deployment.create(deployment1)
   >>> txn.deployment.create(deployment2)

   # Check both were created
   >>> len(txn.deployment.list_keys())
   2

   # Reading via top-level get() can be indexed by value or by key
   >>> all(txn.deployment.get(dpl) for dpl in (deployment1, deployment2))
   True
   >>> all(txn.deployment.get(key) for key in ("my-deployment1", "my-deployment2"))
   True


Collective operations
---------------------

Other operations are performed over the entire entity type.
As such, these operations are directly available
on the corresponding |Txn| attribute.
The only operations currently supported
over a whole entity type is querying.

To query entities of a given type, users invoke the
:meth:`~MultiEntityOperations.query_keys`,
:meth:`~MultiEntityOperations.list_keys`,
:meth:`~MultiEntityOperations.query_values` or
:meth:`~MultiEntityOperations.list_values` methods
on the corresponding attribute.
By default they all return all keys/values
for the given entity type.
Constrains can be given in the form of keyword arguments
that must match the entity's key parts.

For entities that are modelled with ``pydantic``,
the returned keys and values
are instances of the corresponding model.
If entities are modelled with plain dictionaries,
keys are strings and values are dictionaries.

For example, this code queries all subarray keys:

.. doctest:: queries

   >>> txn.component.list_keys()
   []
   >>> txn.component("lmc-controller").create({"value": 1})
   >>> txn.component("lmc-subarray-01").create({"value": 2})
   >>> sorted(txn.component.list_keys())
   ['lmc-controller', 'lmc-subarray-01']

If we wanted keys *and* values for all components
whose IDs start with ``lmc`` we'd write:

.. doctest:: queries

   >>> txn.component.list_values(component_name_prefix="lmc")
   [('lmc-controller', {'value': 1}), ('lmc-subarray-01', {'value': 2})]

The next example creates a whole set of scripts,
which are modelled with ``pydantic``,
then iterates over the first 5 ``vis-receive``
scripts with version ``4.2.*``:

.. testcode:: queries

   from ska_sdp_config.entity import Script
   from itertools import islice, product

   # Create loads of scripts
   names = ("vis-receive", "test-recv-addresses")
   for name, major, minor, patch in product(names, range(10), range(10), range(10)):
      full_version = f"{major}.{minor}.{patch}"
      key = Script.Key(name=name, version=full_version, kind="realtime")
      image = f"{name}:{full_version}"
      sdp_version = "==0.21.0"
      script = Script(key=key, image=image, sdp_version=sdp_version)
      txn.script.create(script)
   print(f"There are {len(txn.script.list_keys())} scripts")

   # Query only some
   print(f"First five vis-receive scripts with version 4.2.* follow:")
   lazy_query = txn.script.query_values(name="vis-receive", version_prefix="4.2.")
   for key, value in islice(lazy_query, 5):
      print(f"{key=}, {value=}")

.. testoutput:: queries

    There are 2000 scripts
    First five vis-receive scripts with version 4.2.* follow:
    key=Key(kind='realtime', name='vis-receive', version='4.2.0'), value=Script(key=Key(kind='realtime', name='vis-receive', version='4.2.0'), image='vis-receive:4.2.0', parameters={}, sdp_version='==0.21.0', resources=[])
    key=Key(kind='realtime', name='vis-receive', version='4.2.1'), value=Script(key=Key(kind='realtime', name='vis-receive', version='4.2.1'), image='vis-receive:4.2.1', parameters={}, sdp_version='==0.21.0', resources=[])
    key=Key(kind='realtime', name='vis-receive', version='4.2.2'), value=Script(key=Key(kind='realtime', name='vis-receive', version='4.2.2'), image='vis-receive:4.2.2', parameters={}, sdp_version='==0.21.0', resources=[])
    key=Key(kind='realtime', name='vis-receive', version='4.2.3'), value=Script(key=Key(kind='realtime', name='vis-receive', version='4.2.3'), image='vis-receive:4.2.3', parameters={}, sdp_version='==0.21.0', resources=[])
    key=Key(kind='realtime', name='vis-receive', version='4.2.4'), value=Script(key=Key(kind='realtime', name='vis-receive', version='4.2.4'), image='vis-receive:4.2.4', parameters={}, sdp_version='==0.21.0', resources=[])



.. _supported_entities:

Supported entities
------------------

.. py:currentmodule:: ska_sdp_config.config

The table below summarises the supported entities
and their operations.
Note that for those entities modelled via ``pydantic``
there is a model class,
while for plain ``dict`` entities there are key parts.

.. list-table:: Entities supported by |Txn|
   :header-rows: 1

   * - Attribute
     - Prefix/path
     - Model class
     - Key parts
     - Has ``ownership``?
     - Has ``state``?

   * - :attr:`~Transaction.component`
     - ``/component``
     - --
     - ``component_name``
     - ``True``
     - ``False``

   * - :attr:`~Transaction.system`
     - ``/system``
     - :class:`~ska_sdp_config.entity.system.System`
     - --
     - ``False``
     - ``False``

   * - :attr:`~Transaction.deployment`
     - ``/deploy``
     - :class:`~ska_sdp_config.entity.deployment.Deployment`
     - --
     - ``False``
     - ``True``

   * - :attr:`~Transaction.execution_block`
     - ``/eb``
     - :class:`~ska_sdp_config.entity.eb.ExecutionBlock`
     - --
     - ``False``
     - ``True``

   * - :attr:`~Transaction.processing_block`
     - ``/pb``
     - :class:`~ska_sdp_config.entity.pb.ProcessingBlock`
     - --
     - ``True``
     - ``True``

   * - :attr:`~Transaction.script`
     - ``/script``
     - :class:`~ska_sdp_config.entity.script.Script`
     - --
     - ``False``
     - ``False``

   * - :attr:`~Transaction.flow`
     - ``/flow``
     - :class:`~ska_sdp_config.entity.flow.Flow`
     - --
     - ``True``
     - ``True``

   * - :attr:`~Transaction.dependency`
     - ``/dependency``
     - :class:`~ska_sdp_config.entity.flow.Dependency`
     - --
     - ``False``
     - ``True``

   * - :attr:`~Transaction.resource`
     - ``/resource``
     - :class:`~ska_sdp_config.entity.resource_management.Resource`
     - --
     - ``False``
     - ``True``

   * - :attr:`~Transaction.request`
     - ``/request``
     - :class:`~ska_sdp_config.entity.resource_management.Request`
     - --
     - ``False``
     - ``True``

   * - :attr:`~Transaction.allocation`
     - ``/allocation``
     - :class:`~ska_sdp_config.entity.resource_management.Allocation`
     - --
     - ``False``
     - ``True``

Arbitrary path access
---------------------

On top of the entity-specific access provided by |Txn|,
there is also an extra :attr:`~Transaction.arbitrary` attribute
that allows operating over any arbitrary path in the database.
This is useful for tools performing operations over generic paths,
or to quickly experiment with storing data under certain paths
before committing to adding a new entity in this package.

To access a particular path
users need to index into :attr:`~Transaction.arbitrary`
as described in :ref:`indexing_dict`,
although here a single positional argument
indicates the path.
The result of that call
is the set of operations
described in :ref:`entity_ops`.

The following example uses arbitrary path access
to first write into ``"/my_path"``,
then obtain the value that has just been written:

.. testsetup:: arbitrary

   import warnings

.. testcode:: arbitrary

   with warnings.catch_warnings(record=True) as warns:
       txn.arbitrary("/my_path").create({"my_value": 3})
       print(txn.arbitrary("/my_path").get())
   print(f"{len(warns)} warnings generated")

.. testoutput:: arbitrary

   {'my_value': 3}
   2 warnings generated

The example above is accessing ``/my_path``,
which is not a path that is handled
by any of the supported entities.
In those cases, a warning is raised
so that users are aware of this unofficial access.

.. note::

   This arbitrary path support shouldn’t be abused as a long-term solution
   if the goal is to introduce a new entity to the system.
   New entities should be explicitly added to this package and used instead.

.. |Txn| replace:: :class:`Transaction`

Accessing known paths raises no warnings:

.. testcode:: arbitrary

   txn.component("lmc-subarray-01").create({"value": 1})
   with warnings.catch_warnings(record=True) as warns:
       print(f'Subarray: {txn.arbitrary("/component/lmc-subarray-01").get()}')
   print(f"{len(warns)} warnings generated")

.. testoutput:: arbitrary

   Subarray: {'value': 1}
   0 warnings generated