Secret & Configuration Management
HashiCorp Vault is a secrets management tool specifically designed to control access to sensitive credentials in low-trust environments. Currently, the SKAO uses Vault as a secure configuration store, be it secrets or Helm charts or networking configurations for infrastructure.
This change, in perspective, helps drive a consistent approach having as top priority security, efficient access control, de-duplication of data and enabling cross-component configuration sharing. As it has various types of authentication - Kubernetes authentication, OpenId Connect for Gitlab - it can be used in all development-to-production use cases.
This is also made possible by recent innovations in the Vault ecosystem, mainly the Vault Secrets Operator (also referred to as VSO). It brings Vault integration with Kubernetes to a whole new level, making it a first-class solution for secret synchronisation and management.
Note
Currently the SKAO clusters support legacy integrations - Vault Injector and Vault CSI Driver - but, with VSO’s support introduction, these are deprecated. Their support will be terminated by Sprint #2 of PI 25 and any deployments using ska-tango-util will be migrated automatically when upgrading to version 0.4.13.
In the following sections we will cover how developers should use Vault, in accordance to the SKAO’s Vault Structure, as well as provide migration guides to using VSO instead of the deprecated solutions.
We are also promoting the usage of Vault in Gitlab CICD pipelines as a configuration management solution. If you are interested in that, please follow this tutorial.
Adding Secrets to Vault
SKAO developers can log in to Vault using their Gitlab account. This login method brings group information that is used to control access to team-specific KV engines and paths.
Note
To correctly use Vault, your team should have a corresponding Gitlab Group. If this isn’t the case, please reach out to the System Team via STS.
When accessing Vault, you should be presented with:
After logging in, using the Sign in with GitLab option, you should be given access to the secrets page.
Each user should have access to a self-named path on the kv KV engine where they can create their own secrets. To do so you need to create secrets in the path kv/users/<gitlab username>/<secret path>. In the example below, the Gitlab username is pedroosorio:
If you try to use the root path of your user, i.e. kv/users/<gitlab username>/, you won’t be able to add any secrets since it is mandatory to have directories to group the secrets created. If your team is properly configured, as mentioned above, you should also have access to kv/users/groups/ska-dev/<team slug in gitlab> and dev/<team slug in gitlab>/:
Note
The kv/users/groups/ska-dev/<team slug in gitlab>/ path is deprecated and will be removed by Sprint #2 of PI 25, in favor of dev/<team slug in gitlab>/
Note that you can see paths to other users’ secrets but cannot read them.
Before adding secrets, please read on the SKAO’s Vault Structure so that you are following the standard put forward for configuration and secret management.
Migration Guide
If you want to migrate to use the Vault Secrets Operator, for a non-TANGO deployment, you can follow the use cases below as a comparison of usage between the Injector, CSI driver and VSO. We highly advise you to follow the Vault tutorial which covers the use of VSO in depth.
If you are using secrets to deploy TANGO devices, using the ska-tango-util, you simply need to upgrade to version 0.4.13.
Vault Integration with Kubernetes
To use vault secrets, inside the pods, we used to have to resort to using Vault Sidecar Injector or the Vault CSI driver. Both of these will be examplified in the sections below, so you can choose the one that best suites your needs.
It should be highlighted, once again, these are no longer the de-facto way of using Vault secrets in Kubernetes and their use is deprecated.
When you create a Pod - the basic unit of work in Kubernetes - you can set environment variables for the containers that run in that Pod. You can do it either using the env configuration field or the envFrom field, to refer to other Kubernetes resources.
Lets consider the following example of a simple deployment with 2 environment variables USERNAME and PASSWORD:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
replicas: 3
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
env:
- name: USERNAME
value: "{{ .Values.username }}"
- name: PASSWORD
value: "{{ .Values.password }}"
With this approach we need to pass the environment variables to the Helm values file where these can be overridden.
To do so we need to have those variables stored somewhere - like Gitlab CI variables - and pass them using Helm arguments in the Makefile.
This is insecure - as we cannot efficiently manage access control to the variables - and it is not traceable.
Deprecated: Vault Sidecar Injector
The Vault Agent Injector alters pod specifications to include Vault Agent containers that render Vault secrets to a shared memory volume using Vault Agent Templates. This method is inefficient as it requires multiple annotations to be written in the Pod and increases the workload due to the injection of another container.
To use the Vault Sidecar Injector in the previous example, we can do:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
replicas: 3
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/role: "kube-role"
vault.hashicorp.com/agent-inject-status: "update"
vault.hashicorp.com/agent-inject-secret-config: "<engine>/data/<path/to/secret>"
vault.hashicorp.com/agent-inject-template-config: |
{{`{{- with secret `}}"<engine>/data/<path/to/secret>"{{` -}}`}}
{{`{{- range $k, $v := .Data.data }}`}}
{{`export {{ $k }}={{ $v }}`}}
{{`{{- end }}`}}
{{`{{- end }}`}}
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
This will create an init-container that will inject inside the file /vault/secrets/config all the secrets stored in Vault at <engine>/<path/to/secret>. The file injected should look something like:
export USERNAME=user
export PASSWORD=1111
To use these variables, you either need to source /vault/secrets/config in the OCI image’ entrypoint script or your application needs to read it. A working example on the SKA projects of this method can be found here:
This method is very inefficient as it requires modifying the application in some way to be able to load the retrieved secrets. They could also be added as JSON or YAML and that would be a better pattern, but adding an init-container to perform this task is sub-optimal.
Deprecated: Vault CSI Provider
The CSI Secrets Store driver allows users to inject data in pods as volumes, regardless of the provider, if these follow the CSI - Container Storage Interface.
For secret stores, we can define a SecretProviderClass that defines which secret provider to use and what secrets to retrieve. When, using Vault, pods requesting CSI volumes are created, the CSI Secrets Store driver will send the request to the Vault CSI Provider. The CSI Provider will then use the SecretProviderClass specification and the pod’s service account to retrieve the secrets´ from Vault and mount them into the pod’s CSI volume.
To adapt the previous example to use the CSI Provider, we first need to add a SecretProviderClass resource:
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: my-app-secret-class
spec:
provider: vault
secretObjects:
- secretName: my-app-secret
type: Opaque
data:
- objectName: username
key: username
- objectName: password
key: password
parameters:
vaultAddress: https://vault.skao.int
roleName: kube-role
objects: |
- objectName: username
secretPath: <engine>/data/<path/to/secret>
secretKey: username
- objectName: password
secretPath: <engine>/data/<path/to/secret>
secretKey: password
This is more convenient than adding annotations to pods, as now we can actually construct a secret in Kubernetes that we can compose from multiple secrets.
Under parameters we specify how to get to and authenticate with Vault and which objects - secret keys named by objectName - to pull. Then, under secretObjects, we instruct the CSI driver what Kubernetes Secrets to create and how to structure them.
Now, we can change our deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
replicas: 3
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
env:
- name: USERNAME
valueFrom:
secretKeyRef:
name: my-app-secret
key: username
- name: PASSWORD
valueFrom:
secretKeyRef:
name: my-app-secret
key: password
volumeMounts:
- name: secrets-store-inline
mountPath: "/mnt/secrets-store"
readOnly: true
volumes:
- name: secrets-store-inline
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: my-app-secret-class
A working example on the SKA projects of this method can be found here:
Although it synchronises secrets into Kubernetes secrets, it requires the CSI volume to be mounted to a pod that is scheduled. This is again inefficient and blocks some high-level behaviours that depend on the existence of a secret to begin with.
Vault Secrets Operator
The Vault Secrets Operator breaks away from the inefficient limitations of the previous solutions by implementing an operator and CRDs. The main difference to the previous solutions is that it is no longer needed for a Vault secret “link” to be present on a workload - simply defining the VaultStaticSecret or VaultDynamicSecret CRDs is enough to have the operator synchronise secrets as Kubernetes secrets.
From that point onwards we can leverage secrets the way we would any other secret without having second considerations.
Adapting the previous example, we no longer create a SecretProviderClass resource but a VaultStaticSecret:
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
name: my-app-secret
spec:
type: kv-v2
mount: <engine>
path: <path/to/secret>
refreshAfter: 60s
destination:
name: my-app-secret
create: true
overwrite: true
transformation:
excludeRaw: true
includes:
- username
- password
Now, we can simplify the deployment manifest compared to either of the previous solutions:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
replicas: 3
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
env:
- name: USERNAME
valueFrom:
secretKeyRef:
name: my-app-secret
key: username
- name: PASSWORD
valueFrom:
secretKeyRef:
name: my-app-secret
key: password
Note that, now, we don’t need to define Volumes or VolumeMounts and the secret will be created regardless of a pod referring to it.
VSO not only brings the simplicity of defining secrets but also brings new powerful features. To know more about them, please follow the tutorial where we cover, end-to-end, the configuration of a Vault instance in a Minikube cluster, the deployment of Vault Secrets Operator and we explore some of its novel features like automatic rollout restarts and transformations.
A working example on the SKA projects of this method can be found here: