Helm charts

Overview of SDP Helm charts and the values that can be used to customise deployments. These charts are published in the SKAO Central Artefact Repository.

SDP chart

The ska-sdp chart is intended to deploy the SDP as a subsystem of a telescope alongside other subsystems. It does not deploy the ancillary services needed by the SDP such as the Tango database or Kafka (for data queues).

See the SDP testing chart for an umbrella chart to deploy the SDP for stand-alone testing.

The SDP chart’s default values can be found in the values file.

Global

Settings that affect the entire deployment.

Parameter	Description	Default
global.data-product-pvc-name	Name of data product PVC	test-pvc
global.tango_host	Host address and port for the Tango database	databaseds-tango-base:10000
global.operator	Use Tango operator to deploy LMC devices	true
global.kafka_host	Host and port of Kafka broker	ska-sdp-kafka

Common settings

Common settings for SDP components.

Parameter	Description	Default
processingNamespace	Namespace for SDP dynamic deployments	sdp
env	Environment variables to be passed into every SDP component	[]

Feature flags

SDP features under development behind feature flags.

Parameter	Description	Default
feature-flags.resourceManagement	If resource management (including the Buffer Manager) should be enabled	false

Configuration database

Configuration database settings implemented on top of etcd.

Parameter	Description	Default
etcd.image	etcd container image	artefact.skao.int/ska-sdp-etcd
etcd.version	etcd container version	See values file
etcd.imagePullPolicy	etcd container image pull policy	IfNotPresent
etcd.replicas	Number of etcd server replicas	1
etcd.logLevel	Logging level for etcd	info
etcd.resources	Resource requests and limits for the pod	{}
etcd.persistence.enabled	Enable persistence of etcd database in persistent volume claim	false
etcd.persistence.size	Size of persistent volume claim	1Gi
etcd.persistence.storageClassName	Storage class name for persistent volume claim	standard
etcd.persistence.autoCompaction.retention	Auto compaction retention for key-value store in hours	0
etcd.persistence.autoCompaction.mode	Auto compaction mode	periodic
etcd.persistence.defrag.enabled	Enable periodic etcd defragmentation	false
etcd.persistence.defrag.cronjob.schedule	Schedule in Cron format for etcd defragmentation	30 23 * * 3,7
etcd.persistence.defrag.cronjob.historyLimit	Number of successful finished jobs to retain	1
etcd.maxTxnOps	Maximum number of operations per transaction	1024
etcd.env	Custom environment variables to set in the application	See values file

LMC (Tango devices)

Settings related to LMC Tango devices.

Parameter	Description	Default
lmc.enabled	Enable the LMC. If set to false, the SDP will run in headless mode	true
lmc.image	LMC container image	artefact.skao.int/ska-sdp-lmc
lmc.version	LMC container version	See values file
lmc.imagePullPolicy	LMC container image pull policy	IfNotPresent
lmc.logLevel	Logging level for LMC devices	info
lmc.env	Custom environment variables to set in the application	See values file
lmc.nsubarray	Number of subarrays to deploy	1
lmc.prefix	Telescope prefix for Tango device names (e.g. low or mid)	test
lmc.allCommandsHaveArgument	Enable all Tango device commands to receive a transaction ID	false
lmc.strictValidation	Enable strict validation of subarray command schemas	false
lmc.schemaDevVersion	Allow using development versions of subarray command schemas	false
lmc.loopInterval	LMC devices idle loop time (seconds)	See values file
lmc.livenessProbe	LMC devices Kubernetes liveness probe configuration	See values file
lmc.readinessProbe	LMC devices Kubernetes readiness probe configuration	See values file
lmc.controller	Controller device settings	See values file
lmc.subarray	Subarray device settings	See values file
lmc.subarray.timeoutAssignResources	Timeout for AssignResources command (seconds)	60
lmc.subarray.timeoutConfigure	Timeout for Configure command (seconds)	60
lmc.controller	Controller device settings	See values file
lmc.subarray	Subarray device settings	See values file
lmc.queueconnector	QueueConnector device settings	See values file
lmc.queueconnector.attributes	List of QueueConnector static attributes.	[]
lmc.queueconnector.attributes[].name	Attribute name.	undefined
lmc.queueconnector.attributes[].data_format	Data format. "SCALAR", "SPECTRUM" or "IMAGE".	SCALAR
lmc.queueconnector.attributes[].data_type	Tango Data Type. See Data types.	DevDouble
lmc.queueconnector.attributes[].max_dim_x	Maximum size for x dimension.	1
lmc.queueconnector.attributes[].max_dim_y	Maximum size for y dimension.	0
lmc.queueconnector.attributes[].writeable	Attribute access modifier. "READ", "WRITE", or READ_WRITE.	"READ"

Console

The console provides a command-line interface to monitor and control the SDP by interacting with the configuration database.

Parameter	Description	Default
console.enabled	Enable the console	true
console.image	Console container image	artefact.skao.int/ska-sdp-console
console.version	Console container version	See values file
console.imagePullPolicy	Console container image pull policy	IfNotPresent
console.replicas	Number of console replicas	1
console.resources	Resource requests and limits for the pod	{}
console.env	Custom environment variables to set in the application	See values file

Processing controller

Parameter	Description	Default
proccontrol.image	Processing controller container image	artefact.skao.int/ska-sdp-proccontrol
proccontrol.version	Processing controller container version	See values file
proccontrol.imagePullPolicy	Processing controller container image pull policy	IfNotPresent
proccontrol.resources	Resource requests and limits for the pod	{}
proccontrol.logLevel	Logging level for processing controller	info
proccontrol.scriptChartVersion	Version of ska-sdp-helmdeploy-script to use for deploying processing scripts	See values file
proccontrol.loopInterval	Processing controller idle loop time (seconds)	See values file
proccontrol.deletePBDelay	Delay after which the processing controller removes finished EBs and PBs	See values file
proccontrol.livenessProbe	Processing controller Kubernetes liveness probe configuration	See values file
proccontrol.readinessProbe	Processing controller Kubernetes readiness probe configuration	See values file
proccontrol.env	Custom environment variables to set in the application	See values file

Quality Assessment (QA)

Parameter	Description	Default
qa.enabled	Enable the QA components	true
qa.api.image	QA API container image	artefact.skao.int/ska-sdp-qa-data-api
qa.api.version	QA API container version	See values file
qa.api.imagePullPolicy	QA API container image pull policy	IfNotPresent
qa.api.workerCount	Number of API worker processes	2
qa.api.allowProfiling	Enable API profiling	false
qa.api.debugMode	Enable debug mode	false
qa.api.dataExpiryHours	Hours after which data expires	12
qa.api.waterfallPlotsExpireSeconds	Waterfall plot expiration in seconds	3600
qa.api.thumbnailWidth	Width of thumbnails in pixels	500
qa.api.thumbnailHeight	Height of thumbnails in pixels	164
qa.api.grafanaDashboard	Grafana dashboard ID	"dad0e09f-32f5-4181-b273-c7a1017221ff/kubernetes-resource-monitoring-dashboard"
qa.api.grafanaBaseUrl	Grafana base URL	"https://monitoring.skao.int"
qa.api.kibanaBaseUrl	Kibana base URL	"https://k8s.stfc.skao.int"
qa.api.resources.requests.cpu	CPU request for QA API	500m
qa.api.resources.requests.memory	Memory request for QA API	512Mi
qa.api.resources.limits.cpu	CPU limit for QA API	2000m
qa.api.resources.limits.memory	Memory limit for QA API	1024Mi
qa.api.env	Custom environment variables to set in the application	See values file
qa.display.image	QA display container image	artefact.skao.int/ska-sdp-qa-display
qa.display.version	QA display container version	See values file
qa.display.imagePullPolicy	QA display image pull policy	IfNotPresent
qa.display.urls.override	Override API URLs	false
qa.display.urls.apiUrl	API URL for display	"http://localhost:8002"
qa.display.urls.wsAPI	WebSocket API URL for display	"ws://localhost:8002/consumer"
qa.display.workflowStatisticsIntervalSeconds	Interval for workflow statistics in seconds	10
qa.display.skipPreflightCheck	Skip preflight check	true
qa.display.env	Custom environment variables to set in the application	See values file
qa.ingress.enabled	Enable ingress for QA	true
qa.ingress.pathStart	Ingress path prefix	"/qa"
qa.ingress.namespaced	Use namespaced ingress	true

Helm deployer

Parameter	Description	Default
helmdeploy.image	Helm deployer container image	artefact.skao.int/ska-sdp-helmdeploy
helmdeploy.version	Helm deployer container version	See values file
helmdeploy.imagePullPolicy	Helm deployer container image pull policy	IfNotPresent
helmdeploy.resources	Resource requests and limits for the pod	{}
helmdeploy.logLevel	Logging level for Helm deployer	info
helmdeploy.releasePrefix	Prefix for Helm release names	""
helmdeploy.chartPrefix	Prefix for Helm chart names	ska-sdp-helmdeploy
helmdeploy.chartRepo.url	Chart repository URL	https://artefact.skao.int/repository/helm-internal
helmdeploy.chartRepo.refresh	Chart repository refresh interval (in seconds)	0
helmdeploy.listMax	Maximum number of releases to list at once	1024
helmdeploy.loopInterval	Helm deployer idle loop time (seconds)	See values file
helmdeploy.livenessProbe	Helm deployer Kubernetes liveness probe configuration	See values file
helmdeploy.readinessProbe	Helm deployer Kubernetes readiness probe configuration	See values file
helmdeploy.enableNADClusterRole	If true then add credentials for helm deployer to list/get/watch the network-attachment-definitions system CRD.	false
helmdeploy.env	Custom environment variables to set in the application	See values file

Slurm Deployer

Parameter	Description	Default
slurmdeploy.enabled	Enables the Slurm Deployer if True.	false
slurmdeploy.image	Slurm Deployer container image.	artefact.skao.int/ska-sdp-slurmdeploy
slurmdeploy.version	Slurm Deployer container image version/tag	See values file
slurmdeploy.imagePullPolicy	Slurm Deployer container image pull policy	IfNotPresent
slurmdeploy.slurmURL	URL for the Slurm REST API endpoint	https://api.staging.hpc.skao.int/slurm/v0.0.40
slurmdeploy.logLevel	Log level for the Slurm Deployer	info
slurmdeploy.resources	Resource requests/limits for the Slurm Deployer Pod	{}
slurmdeploy.listMax	Maximum number of Slurm deployments to list at once	1024
slurmdeploy.loopInterval	Slurm Deployer idle loop time (seconds)	60
slurmdeploy.livenessProbe	Kubernetes liveness probe for Slurm Deployer	See values file
slurmdeploy.readinessProbe	Kubernetes readiness probe for Slurm Deployer	See values file
slurmdeploy.env	Custom environment variables to set in the application	See values file

The Slurm deployer requires access to several secrets. The secret resource needs to be defined outside of the SDP chart. (Refer to SDP Secrets for more information on secrets.) By default, JWT authentication is used.

Parameter	Description	Default
slurmdeploy.secrets.kubeconfig.name	Name of the Secret for the KubeConfig	ska-sdp-slurm-kubeconfig
slurmdeploy.secrets.kubeconfig.secretName	The name of the KubeConfig key in the secret	kubeconfig
slurmdeploy.secrets.slurmToken.name	Name of the Secret for the Slurm Service Auth	ska-sdp-slurm-token
slurmdeploy.secrets.slurmToken.clientIdKey	The name of the JWKS Client ID key in the secret	See values file
slurmdeploy.secrets.slurmToken.clientSecretKey	The name of the JWKS Client Secret key in the secret	See values file
slurmdeploy.secrets.slurmToken.jwtUserKey	Name of JWT user name key in the secret	See values file
slurmdeploy.secrets.slurmToken.jwtKey	Name of the JWT token key in the secret	See values file
slurmdeploy.azure.url	Azure address of JKWS which issues JWTs	"https://login.microsoftonline.com/78887040-bad7-494b-8760-88dcacfb3805/oauth2/v2.0/token"

Buffer Manager

:widths: auto :header-rows: 1

Parameter	Description	Default
buffermanager.image	Buffer Manager container image	artefact.skao.int/ska-sdp-buffermanager
buffermanager.version	Buffer Manager container version	See values file
buffermanager.imagePullPolicy	Buffer Manager container image pull policy	IfNotPresent
buffermanager.resources	Resource requests and limits for the pod	{}
buffermanager.logLevel	Logging level for Helm deployer	info
buffermanager.loopInterval	Buffer Manager idle loop time (seconds)	60
buffermanager.contingencyReservePercent	Percentage of total capacity that is reserved for contingency and is not allocatable	10
buffermanager.livenessProbe	Buffer manager Kubernetes liveness probe configuration	See values file
buffermanager.readinessProbe	Buffer manager Kubernetes readiness probe configuration	See values file
buffermanager.env	Custom environment variables to set in the application	See values file

Global Sky model

:widths: auto :header-rows: 1

Parameter	Description	Default
gsm.enabled	Enable the GSM	false
gsm.ingres.enabled	Enable the GSM’s ingress	true
gsm.ingress.path	What the path should be after the namespace	/gsm
gsm.image	The container image	artefact.skao.int/ska-sdp-global-sky-model
gsm.version	Image version	See values file
gsm.imagePullPolicy	Pull policy for image	IfNotPresent
gsm.logLevel	The logger level to use	info
gsm.loopInterval	The maximum wait time between checking for requests (seconds)	30
gsm.livenessProbe.initialDelaySeconds	The amount of seconds to wait before checking if it is alive.	30
gsm.livenessProbe.periodSeconds	The time between liveness probe checks	60
gsm.startupProbe.initialDelaySeconds	The amount of seconds to wait before checking if it has started.	10
gsm.startupProbe.periodSeconds	The time between startup probe checks.	10
gsm.resources	The resource restrictions on the GSM.	See values file
gsm.env	Custom environment variables to set in the application	See values file
gsm.secrets.name	The name of the secret resource.	ska-sdp-gsm
gsm.secrets.databaseHost	The name of the postgres host key.	PGHOST
gsm.secrets.databaseName	The name of the postgres name key.	PGDATABASE
gsm.secrets.databaseUser	The name of the postgres user key.	PGUSER
gsm.secrets.databasePassword	The name of the postgres password key.	PGPASSWORD
gsm.secrets.databaseSchema	The name of the postgres schema key.	PGSCHEMA

Data Persistent Volume Claim (PVC)

Configure the SDP to use an existing volume or create a new one, and configure the processing scripts to write the data to that volume. When creating new PVCs, one will be created in the control system namespace and the other in the processing namespace.

If the clone-pvc and clone-pvc-namespace values are set, then the PVCs will be created as clones of an existing PVC in another namespace. This allows the storage volume to be shared between namespaces. It depends on the SKAO-specific shared volume infrastructure being available in the cluster.

Parameter	Description	Default
global.data-product-pvc-name	Name of data product PVC that is set as a global value to allow subcharts to access the same PVC	test-pvc
data-pvc.create.enabled	Enable creation of PVCs	false
data-pvc.create.size	Size of PVC	5Gi
data-pvc.create.storageClassName	Storage class name for PVC	standard
data-pvc.create.clone-pvc	Name of PVC to clone	Not set
data-pvc.create.clone-pvc-namespace	Namespace containing the PVC to clone	Not set
data-pvc.pod.enabled	Create a pod that connects to the data-pvc	false

Processing scripts

Processing script definitions to be used by SDP. These map the script kind, name and version to a container image. By default the definitions from the scripts repository are loaded as Telescope Model data at SDP installation time. A different URL may be specified. A schedule can be established to reload the definitions periodically. Alternatively a list of script definitions can be passed to the chart.

Parameter	Description	Default
scripts.url	URL from which to read the script definitions	tmdata::gitlab://gitlab.com/ska-telescope/sdp/ska-sdp-script?master#tmdata
scripts.definitions	List of script definitions. If present, used instead of the URL. See the example below	Not set
scripts.updateSchedule	The cronjob schedule on which script definitions are re-loaded. Only relevant when not using scripts.definitions	Not set

Example of script definitions in a values file:

scripts:
  definitions:
  - type: realtime
    id: test-realtime
    version: 0.3.0
    image: artefact.skao.int/ska-sdp-script-test-batch:0.3.0
  - type: batch
    id: test-batch
    version: 0.3.0
    image: artefact.skao.int/ska-sdp-script-test-realtime:0.3.0

Configuration jobs

Settings that control some of the aspects of the configuration jobs that run when SDP is deployed.

Parameter	Description	Default
jobs.ttl	Time-to-live of jobs after successful completion in seconds	30

Proxy settings

Proxy settings are applied to the components that retrieve configuration data via HTTPS: the script definitions and the Helm charts.

Parameter	Description	Default
proxy.server	Address of proxy server	Not set
proxy.noproxy	List of addresses or subnets for which the proxy should not be used	Not set

Example of proxy settings in a values file:

proxy:
  server: http://proxy.mydomain
  noproxy:
  - 192.168.0.1
  - 192.168.0.2

SDP testing chart

The ska-sdp-testing chart is an umbrella chart intended to deploy the SDP stand-alone for testing. It depends on the SDP chart and other SKA charts for ancillary services. It is also capable of deploying Kafka.

Parameter	Description	Default
kafka.enabled	Enable Kafka	true
ska-tango-base.enabled	Enable Tango base chart; deploys Tango database	true
ska-tango-alarmhandler.enabled	Enable Tango alarm handler	false
ska-tango-tangogql.enabled	Enable TangoGQL	false
ska-tango-taranta.enabled	Enable Taranta	false

The testing chart is also able to deploy a local Postgres database for the Global Sky Model (GSM) service:

Parameter	Description	Default
local-gsm-db.enabled	Enable the creation of a local Postgres database	true
local-gsm-db.image	OCI image to deploy	See values file
local-gsm-db.version	Version of the OCI image that will be deployed	See values file
local-gsm-db.migrations.image	Image name for the Local DB Migrations	See values file
local-gsm-db.migrations.version	Image version for the Local DB Migrations	See values file
local-gsm-db.migrations.pullPolicy	The pull policy for the Local DB Migrations image	IfNotPresent

The testing chart is responsible for creating the secrets for the Slurm deployer, and the GSM service. The relevant parameters are as follows.

Parameter	Description	Default
secrets.useVault	Enable the creation of secrets from Vault	true
secrets.slurm.slurmTokenMount	The name of the engine in Vault	aws-eu-north-1
secrets.slurm.slurmTokenPath	The path in the vault to the Slurm Token	See values file
secrets.slurm.vaultAuthRef	The auth reference to use in Vault	See values file
secrets.slurm.kubeconfigMount	The name of the engine in Vault	dev
secrets.slurm.kubeconfigPath	The path in the vault to the Kube Config	phoenix/aws-dp-kubeconfig
secrets.slurm.refreshAfter	How often to refresh the secrets	24h
secrets.gsm.engine	The name of the engine in Vault	dev
secrets.gsm.pathToSecret	The path in the vault to the GSM secret	phoenix/sdhp-stfc/ci-dev/ska-sdp-global-sky-model/local-db
secrets.gsm.refreshAfter	How often to refresh the secrets	24h