.. _cli:

SDP command-line interface
==========================

Command Line Interface: ``ska-sdp``

To run the CLI, you must start a shell in the console pod
(assuming you have SDP deployed in Kubernetes, to do this follow the instructions in
:doc:`ska-sdp-integration:installation/standalone`).

.. code-block:: bash

    kubectl exec -it ska-sdp-console-0 -n <namespace> -- bash

Once in, to access the help window of ``ska-sdp``, run:

.. code-block:: bash

    ska-sdp -h

Note that if you need a text editor (e.g. for the edit command), you may need to install one
manually. (On the console this can be done with ``apt update; apt install vim``)

Relevant environment variables
------------------------------

Backend-related::

  SDP_CONFIG_BACKEND   Database backend (default etcd3)
  SDP_CONFIG_HOST      Database host address (default 127.0.0.1)
  SDP_CONFIG_PORT      Database port (default 2379)
  SDP_CONFIG_PROTOCOL  Database access protocol (default http)
  SDP_CONFIG_CERT      Client certificate
  SDP_CONFIG_USERNAME  User name
  SDP_CONFIG_PASSWORD  User password

When running ``ska-sdp update --edit``::

  EDITOR    Executable of an existing text editor. Recommended: vi, vim, nano (i.e. command line-based editors)

Usage
-----

Some options are global, as can be seen in the following help.

As an example: ``ska-sdp --prefix /my_prefix list --all`` will list all keys
that are under the ``/my_prefix`` prefix, which is useful for testing.

.. note::

    Global options should be before the command.
    So ``ska-sdp --quiet list --all`` works but ``ska-sdp list --quiet --all``
    will not work.

Each command lists the entities they work with, but in general the list of
entities that are allowed are the following:

.. program-output:: python _ext/output_entity_types.py entities
    :language: rst

Global Help:

.. command-output:: ska-sdp --help

List Entities
^^^^^^^^^^^^^

This command can output the items from the DB. The output can either be just
the keys, or the keys and values.

Examples:

- ``ska-sdp list --all --values`` - will output all keys and their values
- ``ska-sdp list --all`` - will output all keys, but without their values
- ``ska-sdp list script vis-receive`` - will output all ``vis-receive`` scripts
- ``ska-sdp list pb 20250617`` - will output all processing blocks that have ``20250617`` in their name

.. command-output:: ska-sdp list --help

Get/Watch Entities
^^^^^^^^^^^^^^^^^^

This command will output the requested key, and optionally watch for changes.

Examples:

- ``ska-sdp get system`` - will output the system configuration.
- ``ska-sdp get --watch flow pb-notebook-20250617-50671`` - will watch for any flow updates for that Processing Block.
- ``ska-sdp get --watch /component/lmc-subarray-01`` -- will watch the first subarray for changes.

.. command-output:: ska-sdp get --help

Delete Entities
^^^^^^^^^^^^^^^

This command can be used to delete a single key, and optionally all sub-keys.

.. note::

    When used with ``--all`` you will need to confirm the action.

Examples:

- ``ska-sdp delete pb <pb-id>`` - will delete just the processing block.
- ``ska-sdp delete --all pb <pb-id>`` - will delete the processing block, and any sub items of this key.
- ``ska-sdp delete pb --all`` - will delete all processing blocks.

.. command-output:: ska-sdp delete --help

Create Entities
^^^^^^^^^^^^^^^

This command allows you to create various entities, and some entities have specific requirements.

Examples:

- ``ska-sdp create <entity type> '<json encoded object>'`` - the basic layout to create an entity.
- ``cat entity.json | ska-sdp create --stdin <entity type>`` - will allow you to create an entity by reading from stdin.
- ``ska-sdp create <entity type>`` - will start a form based entry. This method is not forgiving of errors besides basic errors. So should be used with caution.

.. command-output:: ska-sdp create --help


Example flow definition
***********************

When creating a new Flow entity using the ``create`` command, the parameters that describe the flow must
be given as a dictionary. There are various kinds of Flow sink, which can be checked in the Flow entry
on the API :ref:`entity_api` page. The parameters are checked against the schema and mistakes will show up as
Pydantic errors. Note that in order to create the correct Flow kind, the ``kind`` parameter must be supplied in both the
``key`` and the ``sink``.

A simple example for a ``data-product`` flow is:

.. code-block:: bash

    ska-sdp create flow '{"key": {"pb_id":"pb-test-20250119-00000","kind":"data-product","name":"pointing-offset"}, "sink": {"kind":data-product,"data_dir":"/wow","paths":[]}, "sources":[], "data_model":"PointingTable"}'


Update/Edit Entities
^^^^^^^^^^^^^^^^^^^^

This command allows you to either update the entire structure, or interactivelly
edit the entity using your editor set with the environment variable ``EDITOR``.

The valid list of entities that are allowed are:

.. program-output:: python _ext/output_entity_types.py update
    :language: rst

Example Usage:

- ``ska-sdp update flow-state pb-notebook-20250617-50671:data-queue:metrics-uvcoverage-pb-notebook-20250617-50671 '{"status": "FLOWING", "last_updated":1762432212.123}'`` - will update the state of the provided flow entry to the given JSON string
- ``ska-sdp update --edit pb pb-notebook-20250617-50671`` - will open your editor to update the Processing Block definition of the given Processing Block ID.

.. note::

    For state entries you only need to specify the values you are updating. So the
    following is valid, and will only update the ``status`` part of the state.

    .. code-block:: bash

        ska-sdp update flow-state \
          pb-notebook-20250617-50671:data-queue:metrics-uvcoverage-pb-notebook-20250617-50671 \
          '{"status": "FLOWING"}'


.. command-output:: ska-sdp update --help

Import System and Scripts
^^^^^^^^^^^^^^^^^^^^^^^^^

This command allows for the importing of the system information, and the script
definitions.

Examples:

- ``ska-sdp import system /data/version.yaml`` -- will import the system config from ``/data/version.yaml``
- ``ska-sdp import scripts tmdata::gitlab://gitlab.com/ska-telescope/sdp/ska-sdp-script#tmdata`` -- will import the scripts found at that location.

.. command-output:: ska-sdp import --help

Example system definitions file
*******************************

You can use a Yaml file to import system definitions into the Config DB. An example system definitions file looks like ::

    version: "0.17.0"
    components:
        lmc-controller:
            devicename: "test-sdp/controller/0"
            image: "artefact.skao.int/ska-sdp-lmc"
            version: "0.24.0"
        lmc-subarray-01:
            devicename: "test-sdp/subarray/01"
            image: "artefact.skao.int/ska-sdp-lmc"
            version: "0.24.0"
    dependencies:
        ska-tango-base:
            repository: "https://artefact.skao.int/helm-internal"
            version: "0.4.6"

After the import, you can check via

.. code-block:: bash

    ska-sdp get system

It will output a list of SDP components that have been imported.

Example script definitions file
*******************************

You can also use a script definitions file to import processing scripts into the Config DB.
An example script definitions file looks like ::

    scripts:
    - kind: realtime
      name: test_realtime
      version: 0.7.0
      image: artefact.skao.int/ska-sdp-script-test-realtime:0.7.0
      sdp_version: ">=0.21.0"
    - kind: batch
      name: test_batch
      version: 0.5.0
      image: artefact.skao.int/ska-sdp-script-test-batch:0.5.0
      sdp_version: ">=0.19.0, <0.22.0"

Both YAML and JSON files are accepted. After the import, you can check via

.. code-block:: bash

    ska-sdp list script

It will output a list of processing scripts that are available to use.

End Execution Blocks
^^^^^^^^^^^^^^^^^^^^

This command allows for ending of Execution Blocks. There are two methods for
ending execution blocks, either by setting them to FINISHED, or CANCELLED.

Examples:

- ``ska-sdp end <eb-id>`` - will set the status to FINISHED.
- ``ska-sdp end --cancel <eb-id>`` - will set the status to CANCELLED.

.. command-output:: ska-sdp end --help

Deprecated CLI
--------------

The previous version of the CLI is still available as ``ska-sdp-old``. The
documentation is available here while it still exists.

.. code-block:: none

    > ska-sdp-old --help

    Command line utility for interacting with SKA Science Data Processor (SDP).

    Usage:
        ska-sdp-old COMMAND [options] [SDP_OBJECT] [<args>...]
        ska-sdp-old COMMAND (-h|--help)
        ska-sdp-old (-h|--help)

    SDP Objects:
        pb           Interact with processing blocks
        script       Interact with available processing script definitions
        deployment   Interact with deployments
        eb           Interact with execution blocks
        flow         Interact with flows
        controller   Interact with Tango controller device
        subarray     Interact with Tango subarray device
        system       Interact with system configuration

    Commands:
        list           List information of object from the Configuration DB
        get | watch    Print all the information (i.e. value) of a key in the Config DB
        create         Create a new, raw key-value pair in the Config DB;
                       Run a processing script; Create a deployment
        update         Update a raw key value from CLI
        edit           Edit a raw key value from text editor
        delete         Delete a single key or all keys within a path from the Config DB
        end            Stop/Cancel execution block
        import         Import system or processing script definitions from file or URL

    Options:
        --prefix=<prefix>   Global prefix to paths (e.g. /test for testing)

.. code-block:: none

    > ska-sdp-old list --help

    List keys (and optionally values) within the Configuration Database.

    Usage:
        ska-sdp-old list (-a |--all) [options]
        ska-sdp-old list [options] pb [<date>]
        ska-sdp-old list [options] flow [<pb-id>]
        ska-sdp-old list [options] script [<kind>]
        ska-sdp-old list [options] (deployment|eb|controller|subarray)
        ska-sdp-old list (-h|--help)

    Arguments:
        <date>      Date on which the processing block(s) were created. Expected format: YYYYMMDD
                    If not provided, all pbs are listed.
        <pb-id>     Processing block id to list flows for. If not provided, all flows are listed
        <kind>      Kind of processing script definition. Batch or realtime.
                    If not provided, all scripts are listed.

    Options:
        -h, --help         Show this screen
        -q, --quiet        Cut back on unnecessary output
        -a, --all          List the contents of the Config DB, regardless of object type
        -v, --values       List all the values belonging to a key in the config db
        --prefix=<prefix>  Path prefix (in addition to standard Config paths, e.g. /test for testing)

.. code-block:: none

    > ska-sdp-old (get|watch) --help

    Get/Watch all information of a single key in the Configuration Database.

    Usage:
        ska-sdp-old (get|watch) [options] system
        ska-sdp-old (get|watch) [options] <key>
        ska-sdp-old (get|watch) [options] pb <pb-id>
        ska-sdp-old (get|watch) [options] eb <eb-id>
        ska-sdp-old (get|watch) [options] deployment <dpl-id>
        ska-sdp-old (get|watch) [options] flow <flow-id>
        ska-sdp-old (get|watch) (-h|--help)

    Arguments:
        <key>       Key within the Config DB.
                    To get the list of all keys:
                        ska-sdp-old list -a
        <pb-id>     Processing block id to list all entries and their values for.
                    Else, use key to get the value of a specific pb.
        <eb-id>     Execution block id to list all entries and their values for.
                    Else, use key to get the value of a specific eb.
        <dpl-id>    Deployment id to list all entries and their values for.
                    Else, use key to get the value of a specific deployment.
        <flow-id>   Flow to list all entries and their values for. Expected format:
                    pb-id:kind:name
                    Else, use key to get the value of a specific pb.

    Options:
        -h, --help    Show this screen
        -q, --quiet   Cut back on unnecessary output

.. code-block:: none

    > ska-sdp-old create --help

    Create SDP objects (deployment, script) in the Configuration Database.
    Create a processing block to run a script.

    Usage:
        ska-sdp-old create [options] pb <script> [<parameters>] [--eb=<eb-parameters>]
        ska-sdp-old create [options] deployment <item-id> <kind> <parameters>
        ska-sdp-old create [options] script <item-id> <value>
        ska-sdp-old create [options] flow <parameters>
        ska-sdp-old create (-h|--help)

    Arguments:
        <script>            Script that the processing block will run, in the format:
                                kind:name:version
        <parameters>        Optional parameters for a script, with expected format:
                                '{"key1": "value1", "key2": "value2"}'
                            For deployments, expected format:
                                '{"chart": <chart-name>, "values": <dict-of-values>}'
                            For flow, expected format:
                                '{"key":{}, "sink":{}, "sources":[], "data_model":""}'
        <eb-parameters>     Optional eb parameters for a real-time script
        <item-id>           Id of the new deployment or script
        <kind>              Kind of the new deployment ("helm" or "slurm")
        <value>             Script definition

    Options:
        -h, --help     Show this screen
        -q, --quiet    Cut back on unnecessary output

    Example:
        ska-sdp-old create pb realtime:vis-receive:0.0.0 '{"": ""}'

    Note: You cannot create processing blocks apart from when they are called to run a script.
    Execution blocks are not allowed to be created on their own, but only
    when you create a realtime processing block with --eb option on.

.. code-block:: none

    > ska-sdp-old (update|edit) --help

    Update the value of a single key or pb/eb/deployment state.
    Can either update from CLI, or edit via a text editor.

    Usage:
        ska-sdp-old update [options] (eb-state|dpl-state|pb-state|flow-state) <item-id> <value>
        ska-sdp-old update [options] script <item-id> <value>
        ska-sdp-old update [options] system <value>
        ska-sdp-old update [options] controller <value>
        ska-sdp-old update [options] subarray <item-id> <value>
        ska-sdp-old edit (eb-state|dpl-state|pb-state|flow-state) <item-id>
        ska-sdp-old edit script <item-id>
        ska-sdp-old edit system
        ska-sdp-old edit controller
        ska-sdp-old edit subarray <item-id>
        ska-sdp-old (update|edit) (-h|--help)

    Arguments:
        <item-id>   id of the script, eb, deployment, processing block or subarray
        <value>     Value to use for update, with format '{"key":"value"}'

    Options:
        -h, --help    Show this screen
        -q, --quiet   Cut back on unnecessary output

    Note:
        ska-sdp-old edit needs an environment variable defined:
            EDITOR: Has to match the executable of an existing text editor
                    Recommended: vi, vim, nano (i.e. command line-based editors)
            Example: EDITOR=vi ska-sdp-old edit <key>
        Processing blocks, execution blocks and deployments cannot be changed,
        apart from their state.

    Example:
        ska-sdp-old edit eb-state eb-test-20210524-00000
            --> key that's edited: /eb/eb-test-20210524-00000/state
        ska-sdp-old edit script batch:test:0.0.0
            --> key that's edited: /script/batch:test:0.0.0
        ska-sdp-old edit pb-state some-pb-id-0000
            --> key that's edited: /pb/some-pb-id-0000/state

.. code-block:: none

    > ska-sdp-old delete --help

    Delete a key from the Configuration Database.

    Usage:
        ska-sdp-old delete (-a|--all) [options] (pb|script|eb|deployment|flow|prefix)
        ska-sdp-old delete [options] (pb|eb|deployment|flow) <item-id>
        ska-sdp-old delete [options] script <script>
        ska-sdp-old delete (-h|--help)

    Arguments:
        <item-id>   ID of the processing block, deployment, execution block, or flow
        <script>    Script definition to be deleted. Expected format: kind:name:version
        prefix      Use this "SDP Object" when deleting with a non-object-specific, user-defined prefix

    Options:
        -h, --help             Show this screen
        -q, --quiet            Cut back on unnecessary output
        --prefix=<prefix>      Path prefix (if other than standard Config paths, e.g. for testing)


.. code-block:: none

    > ska-sdp-old end --help

    End execution block in the configuration database.
    By default it sets the status to FINISHED. If the --cancel flag is set, it sets
    the status to CANCELLED.

    Usage:
        ska-sdp-old end eb <eb-id> [options]
        ska-sdp-old end (-h|--help)

    Arguments:
    <eb-id>    ID of execution block to end

    Options:
        -c, --cancel  Cancel the execution block
        -h, --help    Show this screen
        -q, --quiet   Cut back on unnecessary output


.. code-block:: none

    > ska-sdp-old import --help

    Import processing script definitions into the Configuration Database.
    It currently supports input as a yaml file or URL containing a json string.
    Note that the input can contain a system defintion, a single script,
    multiple versions of a script, or multiple scripts.

    Usage:
        ska-sdp-old import (system|script|scripts) [options] <file-or-url>
        ska-sdp-old import (-h|--help)

    Arguments:
        <file-or-url>   File or URL to import script definitions from.
                        URL can also be a pointer to Telescope Model Data in the
                        format of: "tmdata::<tmdata-source-url>::<tmdata-file>"
                        <tmdata-file> is optional and defaults to "ska-sdp/scripts/scripts.yaml"

    Options:
        -h, --help      Show this screen
        --sync          Delete scripts not in the input (only works for script/scripts)