CSP.LMC Mid project =========================== ## Table of contents - [Documentation](#documentation) - [Repository organization](#repository-organization) - [Containerised Mid CSP.LMC in Kubernetes](#containerised-mid-csplmc-in-kubernetes) - [Minikube installation](#Minikube-installation) - [Clone and build the Mid CSP.LMC docker image](#clone-and-build-the-mid-csplmc-docker-image) - [Mid CSP.LMC Kubernetes Deployment via Helm Charts](#mid-csplmc-kubernetes-deployment-via-helm-charts) - [Deployment of Mid CSP.LMC with real Mid CBF sub-system devices](#deployment-of-mid-csplmc-with-real-mid-cbf-sub-system-devices) - [Deployment of Mid CSP.LMC with CSP sub-system simulators devices](#deploym-mid--cbf-sdp-csp-sub-system-simulators-devices) - [Deployment of Mid CSP.LMC with CBF-SDP emulator](#deployment-of-mid-csplmc-with-cbf-sdp-emulator) - [Run integration tests on a local k8s/minikube cluster](#run-integration-te-a-lo-cbf-sdp-cluster) - [Connect an interactive Tango Client to Mid CSP.LMC](#connect-an-interactive-tango-client-to-mid-csplmc) - [itango](#itango) - [Taranta](#taranta) - [JupyterHub](#jupyterhub) - [Use a local version of ska-csp-lmc-common](#use-a-local-version-of-ska-csp-lmc-common) - [Known bugs](#known-bugs) - [Troubleshooting](#troubleshooting) - [License](#license) ## Documentation [![Documentation Status](https://readthedocs.org/projects/ska-telescope-ska-csp-lmc-mid/badge/?version=latest)](https://developer.skatelescope.org/projects/ska-csp-lmc-mid/en/latest/?badge=latest) The documentation with Architecture description can be found at SKA developer portal: [CSP.LMC mid documentation](https://developer.skatelescope.org/projects/ska-csp-lmc-mid/en/latest/index.html) ## Repository organization The repository has the following organization: * src: the folder with all the project source code * resources: contains the pogo directory with POGO files of the TANGO Device Classes of the project and taranta_dashboards directory * tests: this folder is organized in sub-folders with unit tests and bdd tests to be run with real and simulated sub-system devices. * charts: stores the HELM charts to deploy the Mid CSP.LMC system under kubernets environment. * docs: contains all the files to generate the documentation for the project. ## Containerised Mid CSP.LMC in Kubernetes The TANGO devices of the CSP_Mid.LMC prototype run in a containerised environment. Currently only a limited number of Mid CSP.LMC and Mid CBF devices are run in Docker containers: * the MidCspController and MID CbfController * three instances of the Mid CSP_Mid Mid CBF sub-arrays * four instances of the Very Coarse Channelizer (VCC) devices * four instance of the Frequency Slice Processor (FPS) devices * two instances of the TM TelState Simulator devices * one instance of the TANGO database The Mid CSP.LMC containerised TANGO servers are managed via Kubernetes. The system is setup so that each k8s Pod has only one Docker container that in turn runs only one Tango Device Server application. Mid CSP.LMC TANGO Servers rely on two different Docker images: `ska-csp-lmc-mid` and `ska-mid-cbf-mcs`.
The first one runs the Mid CSP.LMC TANGO devices (real and simulators) and the second those of the Mid CBF.LMC prototype. ## Minikube installation The Mid CSP.LMC project fully relies on the standard SKA CI/CD makefiles. In order to locally deploy and test the project, Minikube has to be installed. [*ska-cicd-deploy-minikube*](https://gitlab.com/ska-telescope/sdi/ska-cicd-deploy-minikube) provides all the instructions to setup a minikube machine running in a virtual environment. The instructions are very detailed and cover many frequent issues. You can check the deployment with a *make vars*. Be aware of heavy HW requirements: 4 cores or more, and more than 8GB ram. Following a short installation procedure: ```bash git clone git@gitlab.com:ska-telescope/sdi/deploy-minikube.git cd deploy-minikube ``` to use Pod driver: ```bash make all ``` to use Docker driver: ```bash make all DRIVER=docker ``` To check that the minikube environment is up and runing, issue the command ```bash minikube status ``` the output should be: ```bash minikube type: Control Plane host: Running kubelet: Running apiserver: Running kubeconfig: Configured ``` To use the built image in Low CSP.LMC system deployment, the environment should be configured to use the local minikube's Docker daemon running with ```bash eval $(minikube docker-env) ``` Note that this command has to be issued in any terminal since the validity of this command is only the terminal in which it is issued. The local Makefile, in the root folder of the project, defines the setting and variables used to customize the docker image building and the deployment of the system. ## Clone and build the Mid CSP.LMC docker image After having a Minikube running, the ska-csp-lmc-mid has to be installed. To do that, issue the command: ```bash $ git clone git@gitlab.com:ska-telescope/ska-csp-lmc-mid.git $ cd ska-csp-lmc-mid $ git submodule update --init --recursive ``` To build the Mid CSP.LMC docker image, issue the command from the project root: ```bash make oci-build ``` ## Mid CSP.LMC Kubernetes Deployment via Helm Charts The deployment of the system is handled by the Helm tool, via the Helm Charts, a set of YAML files describing how the Kubernetes resources are related.
The Mid CSP.LMC Helm Charts are stored in the `charts` directory, organized in several sub-folders: * ska-csp-lmc-mid with the Helm chart to deploy only the Mid CSP.LMC devices:(MidCspController and MidCspSubarray (3 instances) * mid-csp-umbrella with the Helm chart to deploy the whole Mid CSP.LMC system, including the TANGO Database and the Mid CBF.LMC devices. Using custom `values` YAML files stored in this folder, the Mid CSP.LMC can be deployed with a set of simulators devices for all the In particular, the `mid-csp-umbrella` chart depends on the Mid CSP.LMC, Mid CBF.LMC, the Tango DB and Taranta charts and these dependencies are dynamically linked specifying the `dependencies` field in the Chart.yaml.
### Deployment of Mid CSP.LMC with real Mid CBF sub-system devices In the following there are the instructions to deploy the *Mid CSP.LMC system* with the real Mid CBF devices (https://gitlab.com/ska-telescope/ska-mid-cbf-mcs). To deploy the Mid CSP devices in a k8s environment, without a GUI support, issue the command: ```bash make k8s-install-chart ``` This command uses the values file `values-default.yaml` to install only the Mid CSP.LMC and Mid CBF real devices. This configuration does not deploy Taranta pods. The TANGO Devices can accessed using a itango or jupiter shell If a GUI support is desiderable, the following command can be specified: ```bash make VALUES_FILE=charts/mid-csp-umbrella/values-taranta.yaml k8s-install-chart ``` In both cases, the output of the command should be something like the following one: ```bash k8s-dep-update: updating dependencies +++ Updating mid-csp-umbrella chart +++ Getting updates for unmanaged Helm repositories... ...Successfully got an update from the "https://artefact.skao.int/repository/helm-internal" chart repository ...Successfully got an update from the "https://artefact.skao.int/repository/helm-internal" chart repository ...Successfully got an update from the "https://artefact.skao.int/repository/helm-internal" chart repository ...Successfully got an update from the "https://artefact.skao.int/repository/helm-internal" chart repository ...Successfully got an update from the "https://artefact.skao.int/repository/helm-internal" chart repository ...Successfully got an update from the "https://artefact.skao.int/repository/helm-internal" chart repository ...Successfully got an update from the "https://artefact.skao.int/repository/helm-internal" chart repository Saving 11 charts Downloading ska-tango-base from repo https://artefact.skao.int/repository/helm-internal Downloading ska-mid-cbf from repo https://artefact.skao.int/repository/helm-internal Downloading ska-mid-cbf-tmleafnode from repo https://artefact.skao.int/repository/helm-internal Downloading ska-tango-taranta from repo https://artefact.skao.int/repository/helm-internal Downloading ska-taranta-auth from repo https://artefact.skao.int/repository/helm-internal Downloading ska-tango-taranta-dashboard from repo https://artefact.skao.int/repository/helm-internal Downloading ska-tango-taranta-dashboard-pvc from repo https://artefact.skao.int/repository/helm-internal Deleting outdated charts Name: mid-csp Labels: kubernetes.io/metadata.name=mid-csp Annotations: Status: Active No resource quota. No LimitRange resource. install-chart: install charts/mid-csp-umbrella/ release: test in Namespace: mid-csp with params: --set global.minikube= --set global.tango_host=tango-databaseds:10000 --set ska-tango-base.display=192.168.49.1:0 --set ska-tango-base.xauthority=/home/egiani/.Xauthority --set ska-tango-base.jive.enabled= --set sim-cbf.simsubsystem.image.tag=0.11.5-dirty --set sim-pss.simsubsystem.image.tag=0.11.5-dirty --set sim-pst.simsubsystem.image.tag=0.11.5-dirty --set ska-csp-lmc-mid.midcsplmc.image.tag=0.11.5-dirty --values charts/mid-csp-umbrella/values-default.yaml helm upgrade --install test \ --set global.minikube= --set global.tango_host=tango-databaseds:10000 --set ska-tango-base.display=192.168.49.1:0 --set ska-tango-base.xauthority=/home/egiani/.Xauthority --set ska-tango-base.jive.enabled= --set sim-cbf.simsubsystem.image.tag=0.11.5-dirty --set sim-pss.simsubsystem.image.tag=0.11.5-dirty --set sim-pst.simsubsystem.image.tag=0.11.5-dirty --set ska-csp-lmc-mid.midcsplmc.image.tag=0.11.5-dirty --values charts/mid-csp-umbrella/values-default.yaml \ charts/mid-csp-umbrella/ --namespace mid-csp Release "test" does not exist. Installing it now. NAME: test LAST DEPLOYED: Thu May 5 09:48:18 2022 NAMESPACE: mid-csp STATUS: deployed REVISION: 1 TEST SUITE: None ``` The CSP system is deployed in the namespace 'mid-csp': to access any information about pods, logs etc. please specify this namespace. To monitor the deployment progress and wait its completion, issue the command: ```bash make k8s-wait ``` The deployment takes some time because if the docker images are not already present on the disk, they are downloaded from the CAR repository. The command output it's similar to the following one: ```bash k8sWait: waiting for pods to be ready in mid-csp gio 5 mag 2022, 09:48:39, CEST NAME READY STATUS RESTARTS AGE cbfcontroller-controller-0 0/1 Init:0/10 0 17s cbfmcs-mid-configuration-test-ljn4c 0/1 Init:0/1 0 18s cbfsubarray01-cbfsubarray-01-0 0/1 Init:0/2 0 17s cbfsubarray02-cbfsubarray-02-0 0/1 Init:0/2 0 16s cbfsubarray03-cbfsubarray-03-0 0/1 Init:0/2 0 18s csp-lmc-configuration-test-vp744 0/1 Init:0/1 0 18s fsp01-fsp-01-0 0/1 Init:0/5 0 17s fsp02-fsp-02-0 0/1 Init:0/5 0 17s fsp03-fsp-03-0 0/1 Init:0/5 0 16s fsp04-fsp-04-0 0/1 Init:0/5 0 18s midcspcontroller-controller-0 0/1 Init:0/1 0 18s midcspsubarray01-subarray1-0 0/1 Init:0/1 0 18s midcspsubarray02-subarray2-0 0/1 Init:0/1 0 18s midcspsubarray03-subarray3-0 0/1 Init:0/1 0 17s powerswitch001-powerswitch-001-0 0/1 Init:0/1 0 18s ska-tango-base-tangodb-0 1/1 Running 0 18s talonlru001-talonlru-001-0 0/1 Init:0/2 0 16s tango-databaseds-0 1/1 Running 0 18s tangotest-config-4bfqp 1/1 Running 0 18s tangotest-test-0 0/1 Init:0/2 0 15s tmcspsubarrayleafnodetest-tm-0 0/1 Init:0/2 0 16s tmcspsubarrayleafnodetest2-tm2-0 0/1 Init:0/2 0 18s tmsimulator-mid-configuration-test-4g6r2 0/1 Init:0/1 0 18s vcc001-vcc-001-0 0/1 Init:0/2 0 17s vcc002-vcc-002-0 0/1 Init:0/2 0 18s vcc003-vcc-003-0 0/1 Init:0/2 0 18s vcc004-vcc-004-0 0/1 Init:0/2 0 16s gio 5 mag 2022, 09:48:39, CEST k8sWait: Jobs found: cbfmcs-mid-configuration-test csp-lmc-configuration-test tangotest-config tmsimulator-mid-configuration-test job.batch/cbfmcs-mid-configuration-test condition met job.batch/csp-lmc-configuration-test condition met job.batch/tangotest-config condition met job.batch/tmsimulator-mid-configuration-test condition met real 0m10,320s user 0m0,062s sys 0m0,033s k8sWait: Jobs complete - cbfmcs-mid-configuration-test csp-lmc-configuration-test tangotest-config tmsimulator-mid-configuration-test gio 5 mag 2022, 09:48:49, CEST k8sWait: Pods found: cbfcontroller-controller-0 cbfsubarray01-cbfsubarray-01-0 cbfsubarray02-cbfsubarray-02-0 cbfsubarray03-cbfsubarray-03-0 fsp01-fsp-01-0 fsp02-fsp-02-0 fsp03-fsp-03-0 fsp04-fsp-04-0 midcspcontroller-controller-0 midcspsubarray01-subarray1-0 midcspsubarray02-subarray2-0 midcspsubarray03-subarray3-0 powerswitch001-powerswitch-001-0 talonlru001-talonlru-001-0 vcc001-vcc-001-0 vcc002-vcc-002-0 vcc003-vcc-003-0 vcc004-vcc-004-0 k8sWait: going to - kubectl -n mid-csp wait --for=condition=ready --timeout=360s pods cbfcontroller-controller-0 cbfsubarray01-cbfsubarray-01-0 cbfsubarray02-cbfsubarray-02-0 cbfsubarray03-cbfsubarray-03-0 fsp01-fsp-01-0 fsp02-fsp-02-0 fsp03-fsp-03-0 fsp04-fsp-04-0 midcspcontroller-controller-0 midcspsubarray01-subarray1-0 midcspsubarray02-subarray2-0 midcspsubarray03-subarray3-0 powerswitch001-powerswitch-001-0 talonlru001-talonlru-001-0 vcc001-vcc-001-0 vcc002-vcc-002-0 vcc003-vcc-003-0 vcc004-vcc-004-0 pod/cbfcontroller-controller-0 condition met pod/cbfsubarray01-cbfsubarray-01-0 condition met pod/cbfsubarray02-cbfsubarray-02-0 condition met pod/cbfsubarray03-cbfsubarray-03-0 condition met pod/fsp01-fsp-01-0 condition met pod/fsp02-fsp-02-0 condition met pod/fsp03-fsp-03-0 condition met pod/fsp04-fsp-04-0 condition met pod/midcspcontroller-controller-0 condition met pod/midcspsubarray01-subarray1-0 condition met pod/midcspsubarray02-subarray2-0 condition met pod/midcspsubarray03-subarray3-0 condition met pod/powerswitch001-powerswitch-001-0 condition met pod/talonlru001-talonlru-001-0 condition met pod/vcc001-vcc-001-0 condition met pod/vcc002-vcc-002-0 condition met pod/vcc003-vcc-003-0 condition met pod/vcc004-vcc-004-0 condition met ``` The command: ```bash helm list -n mid-csp ``` returns information about the release name (test) and the namespace (mid-csp). ```bash NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION test mid-csp 1 2022-05-05 09:32:18.964172768 +0200 CEST deployed mid-csp-umbrella-0.11.5 0.11.5 ``` To display the information about the system deployed in the `mid-csp` namespace: ```bash make k8s-watch ``` or ```bash kubectl get all -n mid-csp ``` If all the system pods are correctly deployed, the output of the above command should be like this one: ```bash NAME READY STATUS RESTARTS AGE cbfcontroller-controller-0 1/1 Running 0 3m53s cbfsubarray01-cbfsubarray-01-0 1/1 Running 0 3m53s cbfsubarray02-cbfsubarray-02-0 1/1 Running 0 3m54s cbfsubarray03-cbfsubarray-03-0 1/1 Running 0 3m53s fsp01-fsp-01-0 1/1 Running 0 3m53s fsp02-fsp-02-0 1/1 Running 0 3m52s fsp03-fsp-03-0 1/1 Running 0 3m52s fsp04-fsp-04-0 1/1 Running 0 3m53s midcspcontroller-controller-0 1/1 Running 0 3m53s midcspsubarray01-subarray1-0 1/1 Running 0 3m54s midcspsubarray02-subarray2-0 1/1 Running 0 3m51s midcspsubarray03-subarray3-0 1/1 Running 0 3m54s powerswitch001-powerswitch-001-0 1/1 Running 0 3m53s ska-tango-base-tangodb-0 1/1 Running 0 3m52s talonlru001-talonlru-001-0 1/1 Running 0 3m53s tango-databaseds-0 1/1 Running 0 3m53s tangotest-test-0 1/1 Running 0 3m51s tmcspsubarrayleafnodetest-tm-0 1/1 Running 0 3m54s tmcspsubarrayleafnodetest2-tm2-0 1/1 Running 0 3m54s vcc001-vcc-001-0 1/1 Running 0 3m52s vcc002-vcc-002-0 1/1 Running 0 3m52s vcc003-vcc-003-0 1/1 Running 0 3m53s vcc004-vcc-004-0 1/1 Running 0 3m51s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/cbfcontroller-controller ClusterIP None 16m service/cbfsubarray01-cbfsubarray-01 ClusterIP None 16m service/cbfsubarray02-cbfsubarray-02 ClusterIP None 16m service/cbfsubarray03-cbfsubarray-03 ClusterIP None 16m service/fsp01-fsp-01 ClusterIP None 16m service/fsp02-fsp-02 ClusterIP None 16m service/fsp03-fsp-03 ClusterIP None 16m service/fsp04-fsp-04 ClusterIP None 16m service/midcspcontroller-controller ClusterIP None 16m ..... ``` Other Makefile targets, such as `k8s-describe` and `k8s-podlogs`, provide some useful information in case of pods failures. When all the pods (Low CSP, CBF and Taranta) are in running, you can access the system via itango shell or, via the taranta GUI interface, if taranta has been deployed. To uninstall the `mid-csp-umbrella` chart and delete the `test` release in the `mid-csp` namespace, issue the command: ```bash make k8s-uninstall-chart ``` ### Customize the number of Subarrrays and PstBeams Using the command desribed above, the system will deploy 4 CSP.LMC and CBF Subarrays and 1 Pst Beam. To change this values please refer to the yaml configuration in charts/low-csp-umbrella/values-customs.yaml. To install with custom values: ```bash make VALUES_FILE=charts/low-csp-umbrella/values-custom.yaml k8s-install-chart ``` ### Deployment of Mid CSP.LMC with CSP sub-system simulators devices To deploy the Mid CSP.LMC with the simulators, a different `values` file from the default one has to be specified. Set the variable VALUES_FILE to point to `values-sim-devs.yaml` file to deploy the system with the Low CSP simulators devices for the sub-systems. The file `values-sim-with-taranta.yaml` can be used to enable also the deployment of Taranta. To make use of CSP subsystem's simulators the `ska-csp-simulators` chart has to be [included in the umbrella charts](https://gitlab.com/ska-telescope/ska-csp-lmc-mid/-/blob/master/charts/mid-csp-umbrella/Chart.yaml?ref_type=heads#L337) ```bash make VALUES_FILE=charts/mid-csp-umbrella/values-sim-devs.yaml k8s-install-chart ``` In this case the list of deployed pods is the following one: ```bash NAME READY STATUS RESTARTS AGE databaseds-ds-tango-databaseds-0 1/1 Running 0 78s databaseds-tangodb-tango-databaseds-0 1/1 Running 0 84s ds-midcbfctrl-ctrl-0 1/1 Running 0 44s ds-midcbfsubarray-sub1-0 1/1 Running 0 49s ds-midcbfsubarray-sub2-0 1/1 Running 0 42s ds-midcbfsubarray-sub3-0 1/1 Running 0 41s ds-midcspcapabilityfsp01-capabilityfsp-0 1/1 Running 0 40s ds-midcspcapabilityvcc01-capabilityvcc-0 1/1 Running 0 48s ds-midcspcontroller-controller-0 1/1 Running 0 45s ds-midcspsubarray01-subarray1-0 1/1 Running 0 46s ds-midcspsubarray02-subarray2-0 1/1 Running 0 32s ds-midcspsubarray03-subarray3-0 1/1 Running 0 52s ds-midpssctrl-ctrl-0 1/1 Running 0 47s ds-midpsssubarray-sub1-0 1/1 Running 0 51s ds-midpsssubarray-sub2-0 1/1 Running 0 50s ds-midpsssubarray-sub3-0 1/1 Running 0 52s ds-tangotest-test-0 1/1 Running 0 43s ska-tango-base-itango-console 1/1 Running 0 85s ``` This deployment is used to test the Mid CSP.LMC system behavior: * with all the systems, including those not yet developed such as PSS and PST * when fault or anomalous conditions are injected in the simulated devices To use a setup without PSS simulators (i.e. the one expected for AA 0.5) refer to [`values-sim-devs-aa05.yaml`](https://gitlab.com/ska-telescope/ska-csp-lmc-low/-/blob/master/charts/mid-csp-umbrella/values-sim-devs-aa05.yaml?ref_type=heads) file. Further information on how to drive CSP simulators can be found in the [documentation of `ska-csp-simulators`](https://developer.skao.int/projects/ska-csp-simulators/en/latest/?badge=latest). ### Run integration tests on a local k8s/minikube cluster The project includes a set of BDD tests that can be run both with real and simulated TANGO Devices. The tests with real devices are in the `tests/integration` folder, while those with simulators are in `tests/simulated-system`. To run the tests on the local k8s cluster, deploy either the real or symulated system (see above). To run integration tests with *real devices* issue the command, from the root project directory: ```bash make k8s-test ``` For integration test with *simulated* devices the default `TEST_FOLDER` variable has to be changed. To run those tests, issue the command, from the root project directory: ```bash make TEST_FOLDER=simulated-system k8s-test ``` On test completion, uninstall the mid-csp-umbrella chart ## Connect an interactive Tango Client to Mid CSP.LMC To test Mid CSP.LMC functionalities, it is possible to connect an interactive Tango Client using the following tools: *itango*, *Jupyter Notebook* and *Taranta*. The following sections will guide the user step by step. ### itango Just give the command ```bash kubectl exec -it ska-tango-base-itango-console -n mid-csp -- itango3 ``` The command completion is enabled, just give a ``. ### Taranta To monitor and control the Mid CSP.LMC via a GUI interface, the Low project provides a set of Taranta dashboards: they can be found in `resources/taranta_dashbords` folder. To deploy the Mid CSP.LMC with the support of Taranta, set the enviromental variable "TARANTA" to true: ```bash make k8s-install-chart TARANTA=true ``` to work with the real CSP sub-system, or ```bash make k8s-install-chart VALUES_FILE=charts/low-csp-umbrella/values-sim-devs.yaml TARANTA=true ``` to work with the CSP sub-systems simulators. #### Start the Taranta dashboard To start and use it, execute the following steps: Open a browser (preferibly Chrome) and specify the url: ```bash 192.168.49.2/mid-csp/taranta/devices ``` or ```bash minikube/mid-csp/taranta/devices. ``` If successful, the browser displays a page like the one in the image below.
Login is required to issue any command on the system. Press the Login button (top right) and specify your team credentials (https://developer.skao.int/projects/ska-tango-taranta-suite/en/latest/taranta_users.html) using capitol letters. In general these are: ```bash uid: TeamName pwd: TeamName_SKA ``` To load a Taranta Dashboard: - click on Dashbords button (top left) - click on 'Import Dashboard' button - select one of the available dashboards int the *resources/taranta_dashboards* folder Run the dashbord pressing the button 'Start' on the top left of the page and enjoy the tour!
If the minikube is running inside a remote machine, you can still access Taranta by ssh redirection. In another terminal, give: ```bash ssh -L 8081:192.168.49.2:80 @ ``` And point your browser to ```bash http://localhost:8081/mid-csp/taranta/dashboard ``` Taranta Dashboards to control CSP.LMC MId can be found at [`/resources/taranta_dashboards`](https://gitlab.com/ska-telescope/ska-csp-lmc-mid/-/tree/master/resources/taranta_dashboards) ### JupyterHub It is possible to have jupyter-hub as a client in a local (minikube) environment. The first thing to do is to include the correspondant helm chart in the deployment. To do this: ```bash make k8s-install-chart VALUES_FILE=charts/mid-csp-umbrella/values-jupyter.yaml ``` a few pods are added to the deployment. They should look like as below: ```bash NAME READY STATUS RESTARTS AGE continuous-image-puller-mqwcw 1/1 Running 0 38s hub-5d84f6dffd-qtzlh 1/1 Running 0 38s user-scheduler-8f6d6d4c6-cwqg5 1/1 Running 0 38s user-scheduler-8f6d6d4c6-kwd7s 1/1 Running 0 38s ``` This also add the following services: (to access them run `kubectl get svc -n mid-csp`) ```bash NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE hub ClusterIP 10.107.242.98 8081/TCP 2m12s proxy-api ClusterIP 10.104.177.160 8001/TCP 2m12s proxy-public LoadBalancer 10.104.155.147 192.168.49.97 80:30492/TCP 2m12s ``` In particular, the `proxy-public` is a LoadBalancer that exposes an external-ip (`192.168.49.97` in the above example) to access the JupiterHub (note: this could change for each deployment). To open JupyterHub, navigate to `http:///hub` in a web browser. If the deployment is in a remote machine accessed via ssh, a port forwardin is needed as for Taranta: ```bash ssh -L 8081::80 @ ``` And point your browser to ```bash http://localhost:8081/hub ``` Username and password can be anything. Please note that a pod will be created for each username, so in case of multiple login it is suggested to use the same one (while password can be changed everytime) A collection of notebook to be run is in `notebooks`. ## Use a local version of ska-csp-lmc-common During development could be useful to test the local changes on the `ska-csp-lmc-common` before releasing a new version of it. It is possible to use it using some Makefile commands that act ont the `pyproject.toml` and `poetry.lock` files. **Note: ska-csp-lmc-common folder must be in the parent directory of ska-csp-lmc-mid.** To set poetry for the installation of local ska-csp-lmc-common: ```bash make pre-local-install-common ``` while to restore the original files: ```bash make post-local-install-common ``` These commands are integrated into others in order to simplify the procedure. They are presented in the following To build the image with the local ska-csp-lmc-common: ```bash make local-oci-build ``` This will automatically change the poetry files and restore them after the installation. After image is build, **integration tests** can be performed as usual. To perform **unit tests**, a new command will open a shell in a container with local ska-csp-lmc-common already installed: ```bash make dev-container ``` After launching this command, the tests can be performed as usual: ```bash make python-test ``` In the same container also linting can be performed with the local package. ## Known bugs ## Troubleshooting If the command ```bash kubectl logs -f pod/ -n mid-csp ``` aborts with a *failed to watch file : no space left on device*, you can correct by connecting to the k8s node and enlarging the space to be used for log: ```bash $ ssh 192.168.49.2 -l root $ sysctl fs.inotify.max_user_watches=1048576 $ sysctl fs.inotify.max_user_watches ``` If the configurations pods gives a lot of errors, and the TangoDB pod gives the following message: [Warning] Aborted connection 3 to db: 'unconnected' user: 'unauthenticated' host: '172.17.0.1' (This connection closed normally without authentication) Then, before making a deployment you need to give: ```bash unset TANGO_HOST ``` ## License See the [LICENSE](https://gitlab.com/ska-telescope/ska-csp-lmc-mid/-/blob/master/LICENSE?ref_type=heads) file for details.