How to Setup Vault on Kubernetes - Linux Tutorial

Vault is one of the most used secret solution in the Kubernetes, it’s used by big player. Vault is application that store, and tightly control access to tokens, passwords, certificates, encryption keys for protecting secrets, and other sensitive data using a UI, CLI, or HTTP API.
We can use Vault for different use case, but for this tutorial we’ll only cover the secret solution.

prerequisite:
– Kubernetes cluster
– helm
– kubectl
– vault CLI

Create a new namespace for vault, I prefer to use infra for all application/service related to infrastructure

kubectl create namespace infra

Install Vault

Add hashicorp repo to helm

helm repo add hashicorp https://helm.releases.hashicorp.com

Install vault with helm

helm install vault hashicorp/vault -n infra

Verify the pods are running and working

kubectl get pods -l app.kubernetes.io/instance=vault  -n infra
# output
NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 0/1     Running   0          27s
vault-agent-injector-66d89875b5-wk7cw   1/1     Running   0          28s

the vault-0 pod, still not running state because we need to initiate the setup

$ kubectl -n infra  exec --stdin=true --tty=true vault-0 -- vault operator init
Unseal Key 1: me4wGqItig7RLoqifpniw7qt+Ebi4yGUrdj+pz7VyVLI
Unseal Key 2: vIc9k4zKv5476hDjGq/ENmJF9cEA+9rHKC5TEjm60Thr
Unseal Key 3: DXGOw94XL820R/FPY/eQE16D9TqeHaofVdiD4FcSGIg1
Unseal Key 4: P9UWabRAJoQNu11CZcA4ErQmE6mL5dF4cHI/LTDdFH5P
Unseal Key 5: EbUH67gmplE6GxnMe/knzlazDCs9F5qaL28xXPy+tKuM
 
Initial Root Token: hvs.o3s2mxPSBjKZZfy6ngsn9p4x
 
Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.
 
Vault does not store the generated root key. Without at least 3 keys to
reconstruct the root key, Vault will remain permanently sealed!
 
It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.

keep the unseal and root token save. We’ll need this later.

Unseal the vault, copy the first 3 unseal key and run the following command 3 times.

$ kubectl -n infra exec --stdin=true --tty=true vault-0 -- vault operator unseal
# output
Unseal Key (will be hidden):
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    1/3
Unseal Nonce       a6416b90-76b2-8c87-8875-51e6a471b6a8
Version            1.15.2
Build Date         2023-11-06T11:33:28Z
Storage Type       file
HA Enabled         false

this step is to unseal the vault, so we can access, delete and update the vault data. It’s to decrypt the data in vault.

After we run that command 3 times, the logs will inform us that the vault is unsealed and ready to use

$ kubectl logs -l app.kubernetes.io/instance=vault  -n infra
2024-02-12T13:52:24.243Z [INFO]  identity: entities restored
2024-02-12T13:52:24.244Z [INFO]  identity: groups restored
2024-02-12T13:52:24.244Z [INFO]  core: usage gauge collection is disabled
2024-02-12T13:52:24.248Z [INFO]  core: post-unseal setup complete
2024-02-12T13:52:24.248Z [INFO]  core: vault is unsealed

Install Vault CLI

To install the vault CLI, follow this command

wget https://releases.hashicorp.com/vault/1.15.5/vault_1.15.5_linux_amd64.zip
unzip vault_1.15.5_linux_amd64.zip
chmod +x vault
sudo mv vault /usr/local/bin/vault

this command only works in Linux, check the binary for the others

Create Secret

To create a kv-v2 secret, we’ll run the vault command from our computer.
Port forward vault to localhost

$ kubectl port-forward services/vault 8200:8200 -n infra
# output
Forwarding from 127.0.0.1:8200 -> 8200
Forwarding from [::1]:8200 -> 8200
Handling connection for 8200
Handling connection for 8200

export VAULT_ADDR=http://127.0.0.1:8200
vault login

Enable the secret with path secret

vault secrets enable -path=secret kv-v2

Create a secret

vault kv put secret/app/config username="admin" password="superpassword"

We’ll use these secret in our dummy app for testing purpose. To check the secret that we just created, we can use vault CLI

$ vault kv get secret/app/config
# output
===== Secret Path =====
secret/data/app/config
 
======= Metadata =======
Key                Value
---                -----
created_time       2024-02-12T14:17:29.325317406Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1
 
====== Data ======
Key         Value
---         -----
password    superpassword
username    admin

or using curl, for using curl the format for API {MOUNT}/data/{SECRET}

$ curl -H "X-Vault-Token: hvs.o3s2mxPSBjKZZfy6ngsn9p4x" http://127.0.0.1:8200/v1/secret/data/app/config
# output
{"request_id":"78be0c29-a963-72ea-5386-f37ef9054137","lease_id":"","renewable":false,"lease_duration":0,"data":{"data":{"password":"superpassword","username":"admin"},"metadata":{"created_time":"2024-02-12T14:17:29.325317406Z","custom_metadata":null,"deletion_time":"","destroyed":false,"version":1}},"wrap_info":null,"warnings":null,"auth":null}

Kubernetes Authentication

To access the secret from Vault, we’ll utilize the kubernetes authentication build-in in the Vault. Basically it’s a ServiceAccount, that we can attach to any pod that need access to the secret.

Create a file vault-auth-serviceaccount.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-auth
  namespace: infra
---
apiVersion: v1
kind: Secret
metadata:
 name: vault-auth
 namespace: infra
 annotations:
   kubernetes.io/service-account.name: vault-auth
type: kubernetes.io/service-account-token

then deploy the file

kubectl apply -f vault-auth-serviceaccount.yaml -n infra
# output
serviceaccount/vault-auth created
secret/vault-auth created

From Vault CLI, enable the Kubernetes

vault auth enable kubernetes

update the config to disable disable_local_ca_jwt

TOKEN_JWT=$(kubectl get secret vault-auth -o go-template='{{ .data.token }}' | base64 --decode) 
KUBE_CA=$(kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}' | base64 --decode)
vault write auth/kubernetes/config \
    kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
    token_reviewer_jwt="$TOKEN_JWT" \
    kubernetes_ca_cert="$KUBE_CA" \
    disable_local_ca_jwt="true"

Verify the Kubernetes config

$ vault read auth/kubernetes/config
# output
Key                       Value
---                       -----
disable_iss_validation    true
disable_local_ca_jwt      true
issuer                    n/a
kubernetes_ca_cert        -----BEGIN CERTIFICATE-----
MIIDBkDCAe6gAwIBAgIBATANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwptaW5p
a3ViZZNBMB4XDTIyMDUxMTEyMjMxOFoXDTMyMDUwOTEyMjMxOFowFTETMBEGA1UE
AxMKbWluaWt1YmVDQTCCASIwDQYJKoZIhvcNAQEBBQADgxEPADCCAQoCggEBALU2
zoPTCr11vUK8CfpoUl3Tjk6wxHBrqbL2YLsqELBYjDn4cP40T8ssKx2R4K7NJNuR
jjUgLuUsR1kfbF1xEY/n78wIQALvoQ+fTwHIHqtvwqnQ+6W11P6zrt2fqF7r2Qo7
YaFCiEI/h43caMAOIZVjp0UBizJlkSsm6F/tjAqq+KGIutFDA2cGpPQVyXOHOERo
UfXQrYytQHN+C0iP2recQLpz7uHTonwOH/IVztmxAzSJJ2IySrZ1dEj/ElDX74rY
n2BDLi4IIM69TGPcqYTIcdeoqf6Lap22N4+ToA/eGQgGsjTmaW2GUJa1L/Oftv3s
O9Jnbdb+Sb7WU51vRssCAwEAAaNhMF8wDgYDVR0PAQH/BAQDAgKkMB0GA1UdJQQW
MBQGCCsGAQUFBwMCBggrBgEFBQcDATAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQW
BBTg4KdnbL+jaifFUXQtkqkh9qyqBTANBgkqhkiG9w0BAQsFAAOCAQEAU0Ru18De
mmZ2R7ziFDXWiJH2huGEdJx+Y/2rBQLoR3TaWCuxui/vlFNwhcVvE0HNxqBXJpPb
7pnasxIRRpE2LnRJ0j785AapFawqPhDCwtMKCK/1vqebxC8g8BKE2q3MSb+Mgm3F
hi2NwMd7CS2HWPJMxF3SHpnwUzQjCHhngVaVzT5xn8dNWZs0Os6Ny2Fj48tvaG83
5X+p6wU05+PBOEZTsnKQllQK6Mz1vVn8hLW34htwAMa70UJ0+CE8KrszNwj++8ex
EsNPSunbgxboUQQFA2glXfApdejcNx+4ddUzCP1GCmmKDhLJ9arGgLfMHJi5tJRS
EWn5fQoXupw9ug==
-----END CERTIFICATE-----
kubernetes_host           https://192.168.49.2:8443
pem_keys                  []

Create a policy to allow access to certain path in the secret, let’s create a policy that allow read-only access to secret/app/config

vault policy write vault-auth - <<EOF
path "secret/app/config" {
  capabilities = ["read"]
}
EOF

for some use case, I prefer to allow read-only to all secret under secret/app, but for simplicity sake let’s put it that way.

Update the vault to allow access only from namespace infra

vault write auth/kubernetes/role/vault-auth \
        bound_service_account_names=vault-auth \
        bound_service_account_namespaces=infra \
        policies=vault-auth \
        ttl=720h

Deploy Test App

To access the secret from the POD we can utilize the Vault Agent Injector, which will mount the secret to the file, it’ll store the file under directory /vault/secrets To enable this injector we need to use annotation with format

vault.hashicorp.com/agent-inject-secret-<file-name>: vault/secret/path

: is a unique file name where the secret will stored
/vault/secret/path : PATH to the secret in Vault

For example, we create a secret/app/config

vault.hashicorp.com/agent-inject-secret-app-config: secret/app/config

Test App: Init containers

Let’s create a simply job (demo-job.yaml) for demo-ing this capability

apiVersion: apps/v1
apiVersion: batch/v1
kind: Job
metadata:
  name: demo-app
spec:
  backoffLimit: 0
  template:
    metadata:
      annotations:
         vault.hashicorp.com/agent-inject: "true"
         vault.hashicorp.com/role: "vault-auth"
         vault.hashicorp.com/agent-inject-secret-app-config: "secret/app/config"
         vault.hashicorp.com/agent-pre-populate-only: "true"
    spec:
      restartPolicy: Never
      serviceAccountName: vault-auth
      containers:
        - name: alpine
          image: alpine:3.19
          command: ["cat", "/vault/secrets/app-config"]

vault.hashicorp.com/agent-pre-populate-only: "true" it’s good for job, or pod that only need to run, otherwise the vault will add sidecar which is always running.

kubectl apply -f demo-job.yaml -n infra

Get the logs, it should return the result from Vault secret

$ kubectl logs pods/demo-app-f6gvg
Defaulted container "alpine" out of: alpine, vault-agent-init (init)
data: map[password:superpassword username:admin]
metadata: map[created_time:2024-02-12T14:17:29.325317406Z custom_metadata:<nil> deletion_time: destroyed:false version:1]

Test App: Sidecar containers

Create a demo-app-sidecar.yaml with following contents

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-app-sidecar
spec:
  selector:
    matchLabels:
      app: alpine
  replicas: 1
  template:
    metadata:
      annotations:
         vault.hashicorp.com/agent-inject: "true"
         vault.hashicorp.com/role: "vault-auth"
         vault.hashicorp.com/agent-inject-secret-app-config: "secret/app/config"
      labels:
        app: alpine
    spec:
      serviceAccountName: vault-auth
      containers:
      - name: alpine
        image: alpine
        command: ["/bin/sh", "-c"]
        args: ["cat /vault/secrets/app-config && sleep infinity"]

then deploy it to kubernetes

kubectl apply -f demo-app-sidecar.yaml -n infra

Check the logs, it should return the same result, the only differences it the sidecar deployment keep running.