How to Setup Vault on Kubernetes

Vault is one of the most used secret solution in the Kubernetes, it’s used by big player. Vault is application that store, and tightly control access to tokens, passwords, certificates, encryption keys for protecting secrets, and other sensitive data using a UI, CLI, or HTTP API.
We can use Vault for different use case, but for this tutorial we’ll only cover the secret solution.

– Kubernetes cluster
– helm
– kubectl
– vault CLI

Create a new namespace for vault, I prefer to use infra for all application/service related to infrastructure

kubectl create namespace infra

Install Vault

Add hashicorp repo to helm

helm repo add hashicorp

Install vault with helm

helm install vault hashicorp/vault -n infra

Verify the pods are running and working

kubectl get pods -l  -n infra
# output
NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 0/1     Running   0          27s
vault-agent-injector-66d89875b5-wk7cw   1/1     Running   0          28s

the vault-0 pod, still not running state because we need to initiate the setup

$ kubectl -n infra  exec --stdin=true --tty=true vault-0 -- vault operator init
Unseal Key 1: me4wGqItig7RLoqifpniw7qt+Ebi4yGUrdj+pz7VyVLI
Unseal Key 2: vIc9k4zKv5476hDjGq/ENmJF9cEA+9rHKC5TEjm60Thr
Unseal Key 3: DXGOw94XL820R/FPY/eQE16D9TqeHaofVdiD4FcSGIg1
Unseal Key 4: P9UWabRAJoQNu11CZcA4ErQmE6mL5dF4cHI/LTDdFH5P
Unseal Key 5: EbUH67gmplE6GxnMe/knzlazDCs9F5qaL28xXPy+tKuM
Initial Root Token: hvs.o3s2mxPSBjKZZfy6ngsn9p4x
Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.
Vault does not store the generated root key. Without at least 3 keys to
reconstruct the root key, Vault will remain permanently sealed!
It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.

keep the unseal and root token save. We’ll need this later.

Unseal the vault, copy the first 3 unseal key and run the following command 3 times.

$ kubectl -n infra exec --stdin=true --tty=true vault-0 -- vault operator unseal
# output
Unseal Key (will be hidden):
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    1/3
Unseal Nonce       a6416b90-76b2-8c87-8875-51e6a471b6a8
Version            1.15.2
Build Date         2023-11-06T11:33:28Z
Storage Type       file
HA Enabled         false

unseal vault token
this step is to unseal the vault, so we can access, delete and update the vault data. It’s to decrypt the data in vault.

After we run that command 3 times, the logs will inform us that the vault is unsealed and ready to use

$ kubectl logs -l  -n infra
2024-02-12T13:52:24.243Z [INFO]  identity: entities restored
2024-02-12T13:52:24.244Z [INFO]  identity: groups restored
2024-02-12T13:52:24.244Z [INFO]  core: usage gauge collection is disabled
2024-02-12T13:52:24.248Z [INFO]  core: post-unseal setup complete
2024-02-12T13:52:24.248Z [INFO]  core: vault is unsealed

Install Vault CLI

To install the vault CLI, follow this command

chmod +x vault
sudo mv vault /usr/local/bin/vault

this command only works in Linux, check the binary for the others

Create Secret

To create a kv-v2 secret, we’ll run the vault command from our computer.
Port forward vault to localhost

$ kubectl port-forward services/vault 8200:8200 -n infra
# output
Forwarding from -> 8200
Forwarding from [::1]:8200 -> 8200
Handling connection for 8200
Handling connection for 8200

Login using cli, for the token copy the value from the initial setup

export VAULT_ADDR=
vault login

login to vault from cli

Enable the secret with path secret

vault secrets enable -path=secret kv-v2

Create a secret

vault kv put secret/app/config username="admin" password="superpassword"

We’ll use these secret in our dummy app for testing purpose. To check the secret that we just created, we can use vault CLI

$ vault kv get secret/app/config
# output
===== Secret Path =====
======= Metadata =======
Key                Value
---                -----
created_time       2024-02-12T14:17:29.325317406Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1
====== Data ======
Key         Value
---         -----
password    superpassword
username    admin

or using curl, for using curl the format for API {MOUNT}/data/{SECRET}

$ curl -H "X-Vault-Token: hvs.o3s2mxPSBjKZZfy6ngsn9p4x"
# output

Kubernetes Authentication

To access the secret from Vault, we’ll utilize the kubernetes authentication build-in in the Vault. Basically it’s a ServiceAccount, that we can attach to any pod that need access to the secret.

Create a file vault-auth-serviceaccount.yaml

apiVersion: v1
kind: ServiceAccount
  name: vault-auth
  namespace: infra
apiVersion: v1
kind: Secret
 name: vault-auth
 namespace: infra
 annotations: vault-auth

then deploy the file

kubectl apply -f vault-auth-serviceaccount.yaml -n infra
# output
serviceaccount/vault-auth created
secret/vault-auth created

From Vault CLI, enable the Kubernetes

vault auth enable kubernetes

update the config to disable disable_local_ca_jwt

TOKEN_JWT=$(kubectl get secret vault-auth -o go-template='{{ .data.token }}' | base64 --decode) 
KUBE_CA=$(kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}' | base64 --decode)
vault write auth/kubernetes/config \
    kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
    token_reviewer_jwt="$TOKEN_JWT" \
    kubernetes_ca_cert="$KUBE_CA" \

Verify the Kubernetes config

$ vault read auth/kubernetes/config
# output
Key                       Value
---                       -----
disable_iss_validation    true
disable_local_ca_jwt      true
issuer                    n/a
kubernetes_ca_cert        -----BEGIN CERTIFICATE-----
pem_keys                  []

Create a policy to allow access to certain path in the secret, let’s create a policy that allow read-only access to secret/app/config

vault policy write vault-auth - <<EOF
path "secret/app/config" {
  capabilities = ["read"]

for some use case, I prefer to allow read-only to all secret under secret/app, but for simplicity sake let’s put it that way.

Update the vault to allow access only from namespace infra

vault write auth/kubernetes/role/vault-auth \
        bound_service_account_names=vault-auth \
        bound_service_account_namespaces=infra \
        policies=vault-auth \

vault policy kubernetes

Deploy Test App

To access the secret from the POD we can utilize the Vault Agent Injector, which will mount the secret to the file, it’ll store the file under directory /vault/secrets To enable this injector we need to use annotation with format<file-name>: vault/secret/path

: is a unique file name where the secret will stored
/vault/secret/path : PATH to the secret in Vault

For example, we create a secret/app/config secret/app/config

Test App: Init containers

Let’s create a simply job (demo-job.yaml) for demo-ing this capability

apiVersion: apps/v1
apiVersion: batch/v1
kind: Job
  name: demo-app
  backoffLimit: 0
      annotations: "true" "vault-auth" "secret/app/config" "true"
      restartPolicy: Never
      serviceAccountName: vault-auth
        - name: alpine
          image: alpine:3.19
          command: ["cat", "/vault/secrets/app-config"] "true" it’s good for job, or pod that only need to run, otherwise the vault will add sidecar which is always running.

kubectl apply -f demo-job.yaml -n infra

Get the logs, it should return the result from Vault secret

$ kubectl logs pods/demo-app-f6gvg
Defaulted container "alpine" out of: alpine, vault-agent-init (init)
data: map[password:superpassword username:admin]
metadata: map[created_time:2024-02-12T14:17:29.325317406Z custom_metadata:<nil> deletion_time: destroyed:false version:1]

Test App: Sidecar containers

Create a demo-app-sidecar.yaml with following contents

apiVersion: apps/v1
kind: Deployment
  name: demo-app-sidecar
      app: alpine
  replicas: 1
      annotations: "true" "vault-auth" "secret/app/config"
        app: alpine
      serviceAccountName: vault-auth
      - name: alpine
        image: alpine
        command: ["/bin/sh", "-c"]
        args: ["cat /vault/secrets/app-config && sleep infinity"]

then deploy it to kubernetes

kubectl apply -f demo-app-sidecar.yaml -n infra

Check the logs, it should return the same result, the only differences it the sidecar deployment keep running.

Leave a Comment