SDP command-line interface
Command Line Interface: ska-sdp
To run the CLI, you must start a shell in the console pod (assuming you have SDP deployed in Kubernetes, to do this follow the instructions in Installing the SDP).
kubectl exec -it ska-sdp-console-0 -n <namespace> -- bash
Once in, to access the help window of ska-sdp, run:
ska-sdp -h
Note that if you need a text editor (e.g. for the edit command), you may need to install one
manually. (On the console this can be done with apt update; apt install vim)
Relevant environment variables
Backend-related:
SDP_CONFIG_BACKEND Database backend (default etcd3)
SDP_CONFIG_HOST Database host address (default 127.0.0.1)
SDP_CONFIG_PORT Database port (default 2379)
SDP_CONFIG_PROTOCOL Database access protocol (default http)
SDP_CONFIG_CERT Client certificate
SDP_CONFIG_USERNAME User name
SDP_CONFIG_PASSWORD User password
When running ska-sdp update --edit:
EDITOR Executable of an existing text editor. Recommended: vi, vim, nano (i.e. command line-based editors)
Usage
Some options are global, as can be seen in the following help.
As an example: ska-sdp --prefix /my_prefix list --all will list all keys
that are under the /my_prefix prefix, which is useful for testing.
Note
Global options should be before the command.
So ska-sdp --quiet list --all works but ska-sdp list --quiet --all
will not work.
Each command lists the entities they work with, but in general the list of entities that are allowed are the following:
- deploy
- eb
- dependency
- flow
- pb
- allocation
- request
- resource
- script
- system
Global Help:
$ ska-sdp --help
usage: ska-sdp [-h] [-v] [-d] [-q] [--version] [--prefix PREFIX]
{list,create,update,import,get,delete,end} ...
Command line utility for interacting with SKA Science Data Processor (SDP).
Use --help on the commands for detailed usage.
positional arguments:
{list,create,update,import,get,delete,end}
list List entries in the Config DB.
create Create entities in the Config DB.
update Update entities and states in the Config DB.
import Import system and script definitions into the Config
DB.
get Print the value of an entry in the Config DB.
delete Delete a single key or all keys within a path from the
Config DB.
end End execution block in the Config DB.
options:
-h, --help show this help message and exit
-v, --verbose Output verbose (INFO level) (default: False)
-d, --debug Output debug level verbosity (DEBUG level) (default:
False)
-q, --quiet Reduce output to minimal (Only ERROR logs and output)
(default: False)
--version Output version of these tools, and exit (default:
False)
--prefix PREFIX Global prefix to paths (e.g. /test for testing)
(default: None)
Version: 1.6.1
List Entities
This command can output the items from the DB. The output can either be just the keys, or the keys and values.
Examples:
ska-sdp list --all --values- will output all keys and their valuesska-sdp list --all- will output all keys, but without their valuesska-sdp list script vis-receive- will output allvis-receivescriptsska-sdp list pb 20250617- will output all processing blocks that have20250617in their name
$ ska-sdp list --help
usage: ska-sdp list [-h] [-a] [-v]
[{deploy,eb,dependency,flow,pb,allocation,request,resource,script,system}]
[filter]
List entries in the Config DB.
Sample Usage:
ska-sdp list (-a |--all) [options]
ska-sdp list [options] pb [<date>]
ska-sdp list [options] flow [<pb-id>]
ska-sdp list [options] script [<kind>]
ska-sdp list [options] (deployment|eb|controller|subarray)
positional arguments:
{deploy,eb,dependency,flow,pb,allocation,request,resource,script,system}
The entity type to create (default: None)
filter Filter entities by value (default: )
options:
-h, --help show this help message and exit
-a, --all List all items (default: False)
-v, --values List all the values belonging to a key in the config
db (default: False)
Get/Watch Entities
This command will output the requested key, and optionally watch for changes.
Examples:
ska-sdp get system- will output the system configuration.ska-sdp get --watch flow pb-notebook-20250617-50671- will watch for any flow updates for that Processing Block.ska-sdp get --watch /component/lmc-subarray-01– will watch the first subarray for changes.
$ ska-sdp get --help
usage: ska-sdp get [-h] [--watch]
[{deploy,eb,dependency,flow,pb,allocation,request,resource,script,system}]
key
Print the value of an entry in the Config DB.
Sample Usage:
ska-sdp get system
ska-sdp get <key>
ska-sdp get pb <pb-id>
ska-sdp get eb <eb-id>
ska-sdp get deployment <dpl-id>
ska-sdp get flow <flow-id>
Use with --watch to monitor for changes.
positional arguments:
{deploy,eb,dependency,flow,pb,allocation,request,resource,script,system}
The entity to perform the action on (default: None)
key The entity key
options:
-h, --help show this help message and exit
--watch, --follow Watch the entry (update output live as changes occur)
(default: False)
Delete Entities
This command can be used to delete a single key, and optionally all sub-keys.
Note
When used with --all you will need to confirm the action.
Examples:
ska-sdp delete pb <pb-id>- will delete just the processing block.ska-sdp delete --all pb <pb-id>- will delete the processing block, and any sub items of this key.ska-sdp delete pb --all- will delete all processing blocks.
$ ska-sdp delete --help
usage: ska-sdp delete [-h] [-a]
{deploy,eb,dependency,flow,pb,allocation,request,resource,script,system}
[id]
Delete a single key or all keys within a path from the Config DB.
Sample Usage:
ska-sdp delete <entity type> <item id>
positional arguments:
{deploy,eb,dependency,flow,pb,allocation,request,resource,script,system}
The entity to perform the action on
id ID of entity, or Script as kind:name:version.
(default: None)
options:
-h, --help show this help message and exit
-a, --all Delete all items under the specified entity (default:
False)
Create Entities
This command allows you to create various entities, and some entities have specific requirements.
Examples:
ska-sdp create <entity type> '<json encoded object>'- the basic layout to create an entity.cat entity.json | ska-sdp create --stdin <entity type>- will allow you to create an entity by reading from stdin.ska-sdp create <entity type>- will start a form based entry. This method is not forgiving of errors besides basic errors. So should be used with caution.
$ ska-sdp create --help
usage: ska-sdp create [-h] [--stdin] [--script SCRIPT]
[--parameters PARAMETERS] [--eb EB]
{deploy,eb,dependency,flow,pb,allocation,request,resource,script,system}
[<json parameters> ...]
Create entities in the Config DB.
This command supports three input styles for most entities:
- JSON via --stdin
- JSON as positional args
- Interactive form based on the entity's Pydantic model
One exception is for processing blocks:
- Requires: --script kind:name:version
- optional: --parameters, --eb
For a realtime script, an execution block is created automatically.
If --eb contains a key, that is used as the eb_id. The eb is linked
to the pb and starts with status=active.
Sample Usage:
ska-sdp create <entity> '<json>'
ska-sdp create pb --script realtime:vis-receive:1.0.0
ska-sdp create pb --script realtime:vis-receive:1.0.0 \
--parameters '<json>' \
--eb '<json>'
To create an entity interactively (with prompts):
ska-sdp create <entity>
positional arguments:
{deploy,eb,dependency,flow,pb,allocation,request,resource,script,system}
The entity type to create
<json parameters> JSON encoded entities to create (default: None)
options:
-h, --help show this help message and exit
--stdin Read a single JSON entity from STDIN (default: False)
--script SCRIPT Script the pb will run (kind:name:version) (default:
None)
--parameters PARAMETERS
json parameters for the script (default: None)
--eb EB json eb parameters (default: None)
Example flow definition
When creating a new Flow entity using the create command, the parameters that describe the flow must
be given as a dictionary. There are various kinds of Flow sink, which can be checked in the Flow entry
on the API Entities page. The parameters are checked against the schema and mistakes will show up as
Pydantic errors. Note that in order to create the correct Flow kind, the kind parameter must be supplied in both the
key and the sink.
A simple example for a data-product flow is:
ska-sdp create flow '{"key": {"pb_id":"pb-test-20250119-00000","kind":"data-product","name":"pointing-offset"}, "sink": {"kind":data-product,"data_dir":"/wow","paths":[]}, "sources":[], "data_model":"PointingTable"}'
Update/Edit Entities
This command allows you to either update the entire structure, or interactivelly
edit the entity using your editor set with the environment variable EDITOR.
The valid list of entities that are allowed are:
- allocation
- dependency
- dpl-state
- eb-state
- flow
- flow-state
- pb-state
- request
- resource
- script
- system
Example Usage:
ska-sdp update flow-state pb-notebook-20250617-50671:data-queue:metrics-uvcoverage-pb-notebook-20250617-50671 '{"status": "FLOWING", "last_updated":1762432212.123}'- will update the state of the provided flow entry to the given JSON stringska-sdp update --edit pb pb-notebook-20250617-50671- will open your editor to update the Processing Block definition of the given Processing Block ID.
Note
For state entries you only need to specify the values you are updating. So the
following is valid, and will only update the status part of the state.
ska-sdp update flow-state \
pb-notebook-20250617-50671:data-queue:metrics-uvcoverage-pb-notebook-20250617-50671 \
'{"status": "FLOWING"}'
$ ska-sdp update --help
usage: ska-sdp update [-h] [--edit] [--stdin]
{allocation,dependency,dpl-state,eb-state,flow,flow-state,pb-state,request,resource,script,system}
[key] [<json parameters> ...]
Update entities and states in the Config DB.
Sample Usage:
ska-sdp update <entity> '<json>'
ska-sdp update pb-state <pb-id> '<json>'
ska-sdp update --edit <entity> <key>
ska-sdp update --edit pb-state <pb-id>
positional arguments:
{allocation,dependency,dpl-state,eb-state,flow,flow-state,pb-state,request,resource,script,system}
The entity or state to update
key Key of the state to update (default: None)
<json parameters> JSON encoded entities to update (default: None)
options:
-h, --help show this help message and exit
--edit Open the target in $EDITOR (default: False)
--stdin Read a single JSON entity from STDIN (default: False)
Import System and Scripts
This command allows for the importing of the system information, and the script definitions.
Examples:
ska-sdp import system /data/version.yaml– will import the system config from/data/version.yamlska-sdp import scripts tmdata::gitlab://gitlab.com/ska-telescope/sdp/ska-sdp-script#tmdata– will import the scripts found at that location.
$ ska-sdp import --help
usage: ska-sdp import [-h] [--sync] {system,scripts} file_or_url
Import system and script definitions into the Config DB.
Input can be either a file, a URL or a Telescope model data string. The input
should be YAML encoded.
This script currently can import a system definition, a single script,
multiple versions of a script, or multiple scripts.
positional arguments:
{system,scripts} Type to import
file_or_url File or URL to import script definitions from. URL can
also be a pointer to Telescope Model Data in the format
of: "tmdata::<tmdata-source-url>::<tmdata-file>" <tmdata-
file> is optional and defaults to "ska-
sdp/scripts/scripts.yaml"
options:
-h, --help show this help message and exit
--sync Delete scripts not in the input (only works for
script/scripts) (default: False)
Example system definitions file
You can use a Yaml file to import system definitions into the Config DB. An example system definitions file looks like
version: "0.17.0"
components:
lmc-controller:
devicename: "test-sdp/controller/0"
image: "artefact.skao.int/ska-sdp-lmc"
version: "0.24.0"
lmc-subarray-01:
devicename: "test-sdp/subarray/01"
image: "artefact.skao.int/ska-sdp-lmc"
version: "0.24.0"
dependencies:
ska-tango-base:
repository: "https://artefact.skao.int/helm-internal"
version: "0.4.6"
After the import, you can check via
ska-sdp get system
It will output a list of SDP components that have been imported.
Example script definitions file
You can also use a script definitions file to import processing scripts into the Config DB. An example script definitions file looks like
scripts:
- kind: realtime
name: test_realtime
version: 0.7.0
image: artefact.skao.int/ska-sdp-script-test-realtime:0.7.0
sdp_version: ">=0.21.0"
- kind: batch
name: test_batch
version: 0.5.0
image: artefact.skao.int/ska-sdp-script-test-batch:0.5.0
sdp_version: ">=0.19.0, <0.22.0"
Both YAML and JSON files are accepted. After the import, you can check via
ska-sdp list script
It will output a list of processing scripts that are available to use.
End Execution Blocks
This command allows for ending of Execution Blocks. There are two methods for ending execution blocks, either by setting them to FINISHED, or CANCELLED.
Examples:
ska-sdp end <eb-id>- will set the status to FINISHED.ska-sdp end --cancel <eb-id>- will set the status to CANCELLED.
$ ska-sdp end --help
usage: ska-sdp end [-h] [-c] {eb} eb-id
End execution block in the Config DB.
By default it sets the status to FINISHED. If the --cancel flag is set, it sets
the status to CANCELLED.
Sample Usage:
ska-sdp end eb <eb-id>
ska-sdp end --cancel eb <eb-id>
positional arguments:
{eb} The entity type
eb-id ID of execution block to end
options:
-h, --help show this help message and exit
-c, --cancel Cancel the execution block (default: False)
Deprecated CLI
The previous version of the CLI is still available as ska-sdp-old. The
documentation is available here while it still exists.
> ska-sdp-old --help
Command line utility for interacting with SKA Science Data Processor (SDP).
Usage:
ska-sdp-old COMMAND [options] [SDP_OBJECT] [<args>...]
ska-sdp-old COMMAND (-h|--help)
ska-sdp-old (-h|--help)
SDP Objects:
pb Interact with processing blocks
script Interact with available processing script definitions
deployment Interact with deployments
eb Interact with execution blocks
flow Interact with flows
controller Interact with Tango controller device
subarray Interact with Tango subarray device
system Interact with system configuration
Commands:
list List information of object from the Configuration DB
get | watch Print all the information (i.e. value) of a key in the Config DB
create Create a new, raw key-value pair in the Config DB;
Run a processing script; Create a deployment
update Update a raw key value from CLI
edit Edit a raw key value from text editor
delete Delete a single key or all keys within a path from the Config DB
end Stop/Cancel execution block
import Import system or processing script definitions from file or URL
Options:
--prefix=<prefix> Global prefix to paths (e.g. /test for testing)
> ska-sdp-old list --help
List keys (and optionally values) within the Configuration Database.
Usage:
ska-sdp-old list (-a |--all) [options]
ska-sdp-old list [options] pb [<date>]
ska-sdp-old list [options] flow [<pb-id>]
ska-sdp-old list [options] script [<kind>]
ska-sdp-old list [options] (deployment|eb|controller|subarray)
ska-sdp-old list (-h|--help)
Arguments:
<date> Date on which the processing block(s) were created. Expected format: YYYYMMDD
If not provided, all pbs are listed.
<pb-id> Processing block id to list flows for. If not provided, all flows are listed
<kind> Kind of processing script definition. Batch or realtime.
If not provided, all scripts are listed.
Options:
-h, --help Show this screen
-q, --quiet Cut back on unnecessary output
-a, --all List the contents of the Config DB, regardless of object type
-v, --values List all the values belonging to a key in the config db
--prefix=<prefix> Path prefix (in addition to standard Config paths, e.g. /test for testing)
> ska-sdp-old (get|watch) --help
Get/Watch all information of a single key in the Configuration Database.
Usage:
ska-sdp-old (get|watch) [options] system
ska-sdp-old (get|watch) [options] <key>
ska-sdp-old (get|watch) [options] pb <pb-id>
ska-sdp-old (get|watch) [options] eb <eb-id>
ska-sdp-old (get|watch) [options] deployment <dpl-id>
ska-sdp-old (get|watch) [options] flow <flow-id>
ska-sdp-old (get|watch) (-h|--help)
Arguments:
<key> Key within the Config DB.
To get the list of all keys:
ska-sdp-old list -a
<pb-id> Processing block id to list all entries and their values for.
Else, use key to get the value of a specific pb.
<eb-id> Execution block id to list all entries and their values for.
Else, use key to get the value of a specific eb.
<dpl-id> Deployment id to list all entries and their values for.
Else, use key to get the value of a specific deployment.
<flow-id> Flow to list all entries and their values for. Expected format:
pb-id:kind:name
Else, use key to get the value of a specific pb.
Options:
-h, --help Show this screen
-q, --quiet Cut back on unnecessary output
> ska-sdp-old create --help
Create SDP objects (deployment, script) in the Configuration Database.
Create a processing block to run a script.
Usage:
ska-sdp-old create [options] pb <script> [<parameters>] [--eb=<eb-parameters>]
ska-sdp-old create [options] deployment <item-id> <kind> <parameters>
ska-sdp-old create [options] script <item-id> <value>
ska-sdp-old create [options] flow <parameters>
ska-sdp-old create (-h|--help)
Arguments:
<script> Script that the processing block will run, in the format:
kind:name:version
<parameters> Optional parameters for a script, with expected format:
'{"key1": "value1", "key2": "value2"}'
For deployments, expected format:
'{"chart": <chart-name>, "values": <dict-of-values>}'
For flow, expected format:
'{"key":{}, "sink":{}, "sources":[], "data_model":""}'
<eb-parameters> Optional eb parameters for a real-time script
<item-id> Id of the new deployment or script
<kind> Kind of the new deployment ("helm" or "slurm")
<value> Script definition
Options:
-h, --help Show this screen
-q, --quiet Cut back on unnecessary output
Example:
ska-sdp-old create pb realtime:vis-receive:0.0.0 '{"": ""}'
Note: You cannot create processing blocks apart from when they are called to run a script.
Execution blocks are not allowed to be created on their own, but only
when you create a realtime processing block with --eb option on.
> ska-sdp-old (update|edit) --help
Update the value of a single key or pb/eb/deployment state.
Can either update from CLI, or edit via a text editor.
Usage:
ska-sdp-old update [options] (eb-state|dpl-state|pb-state|flow-state) <item-id> <value>
ska-sdp-old update [options] script <item-id> <value>
ska-sdp-old update [options] system <value>
ska-sdp-old update [options] controller <value>
ska-sdp-old update [options] subarray <item-id> <value>
ska-sdp-old edit (eb-state|dpl-state|pb-state|flow-state) <item-id>
ska-sdp-old edit script <item-id>
ska-sdp-old edit system
ska-sdp-old edit controller
ska-sdp-old edit subarray <item-id>
ska-sdp-old (update|edit) (-h|--help)
Arguments:
<item-id> id of the script, eb, deployment, processing block or subarray
<value> Value to use for update, with format '{"key":"value"}'
Options:
-h, --help Show this screen
-q, --quiet Cut back on unnecessary output
Note:
ska-sdp-old edit needs an environment variable defined:
EDITOR: Has to match the executable of an existing text editor
Recommended: vi, vim, nano (i.e. command line-based editors)
Example: EDITOR=vi ska-sdp-old edit <key>
Processing blocks, execution blocks and deployments cannot be changed,
apart from their state.
Example:
ska-sdp-old edit eb-state eb-test-20210524-00000
--> key that's edited: /eb/eb-test-20210524-00000/state
ska-sdp-old edit script batch:test:0.0.0
--> key that's edited: /script/batch:test:0.0.0
ska-sdp-old edit pb-state some-pb-id-0000
--> key that's edited: /pb/some-pb-id-0000/state
> ska-sdp-old delete --help
Delete a key from the Configuration Database.
Usage:
ska-sdp-old delete (-a|--all) [options] (pb|script|eb|deployment|flow|prefix)
ska-sdp-old delete [options] (pb|eb|deployment|flow) <item-id>
ska-sdp-old delete [options] script <script>
ska-sdp-old delete (-h|--help)
Arguments:
<item-id> ID of the processing block, deployment, execution block, or flow
<script> Script definition to be deleted. Expected format: kind:name:version
prefix Use this "SDP Object" when deleting with a non-object-specific, user-defined prefix
Options:
-h, --help Show this screen
-q, --quiet Cut back on unnecessary output
--prefix=<prefix> Path prefix (if other than standard Config paths, e.g. for testing)
> ska-sdp-old end --help
End execution block in the configuration database.
By default it sets the status to FINISHED. If the --cancel flag is set, it sets
the status to CANCELLED.
Usage:
ska-sdp-old end eb <eb-id> [options]
ska-sdp-old end (-h|--help)
Arguments:
<eb-id> ID of execution block to end
Options:
-c, --cancel Cancel the execution block
-h, --help Show this screen
-q, --quiet Cut back on unnecessary output
> ska-sdp-old import --help
Import processing script definitions into the Configuration Database.
It currently supports input as a yaml file or URL containing a json string.
Note that the input can contain a system defintion, a single script,
multiple versions of a script, or multiple scripts.
Usage:
ska-sdp-old import (system|script|scripts) [options] <file-or-url>
ska-sdp-old import (-h|--help)
Arguments:
<file-or-url> File or URL to import script definitions from.
URL can also be a pointer to Telescope Model Data in the
format of: "tmdata::<tmdata-source-url>::<tmdata-file>"
<tmdata-file> is optional and defaults to "ska-sdp/scripts/scripts.yaml"
Options:
-h, --help Show this screen
--sync Delete scripts not in the input (only works for script/scripts)