Deploy HashiCorp Vault for Secrets Management in Kubernetes

In a real Kubernetes production environment, you do not only have containers, but also other tightly-integrated collections of related components. The microservices act as mini-applications that work together as a system. In other words, a microservice is an application that runs on a server or virtual instance and responds to network requests. The main advantage of microservices is that it allows the admins to build specific applications for customers/tasks. The admins can then scale the services as desired.

It is therefore important to understand the fundamentals of Kubernetes microservices and the security features. Normally, Kubernetes allows one to store and manage sensitive information that includes passwords, OAuth tokens, SSH keys e.t.c. as Kubernetes secrets.

The idea of using secrets is to avoid exposing sensitive data in your application code since the Secrets can be deployed independently of the pods using them. There are 3 ways in which a pod can use a secret. There are:

• As files in a volume mounted on one or more of its containers.
• As container environment variable
• In a container image

Once the Kubernetes cluster grows, it becomes hard to manage the secrets. HashiCorp Vault acts as a centrally managed service that handles encryption and storage of secrets in the entire infrastructure. It allows one to store and control access to tokens, passwords, certificates, and encryption keys to protect secrets and other sensitive data using UI, CLI, or HTTP API. Vault is a complex tool with several pieces working together.

The below diagram shows the Vault architecture:

There are several features associated with HashiCorp Vault, these are:

• Open Source – it is available for download and can be used locally or in the cloud.
• Identity-based security – it deeply integrates with trusted identities to automate access to secrets, data, and systems
• Application & machine identity – it allows one to secure applications and systems with machine identity and automate credential issuance, rotation e.t.c

HashiCorp vault is commonly used for:

• Secret management: where you can centrally store and manage the secrets
• Dynamic secrets: that are generated on demand. They are unique to clients instead of using static secrets.
• Kubernetes secrets: where you can securely inject the secrets in your application stack
• Database Credential rotation: you can rotate the database password automatically using the Vault’s database engine.
• Automated PKI infrastructure: quickly create X.509 certificates on demand.
• Identity-based access: ability to authenticate to other clouds, systems, and endpoints using trusted identities.

In this guide, we will learn how to deploy HashiCorp Vault for Secrets Management in Kubernetes

Getting Started

For this guide, you need the following:

• A Kubernetes Cluster – This can be deployed using the aid of the below guides:
  • Install Kubernetes Cluster on Ubuntu with kubeadm
  • Deploy Kubernetes Cluster on AlmaLinux 8 with Kubeadm
  • Run Kubernetes on Debian with Minikube
  • Deploy Kubernetes Cluster on Linux With k0s

We will also use the below tools:

• Kubectl
• Helm v3
• Vault CLI

Kubectl can be installed using the command:

curl -LO "https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl
sudo mv kubectl /usr/local/bin

Allow kubectl to access the cluster:

# For k0s
export KUBECONFIG=/var/lib/k0s/pki/admin.conf

Check the available nodes:

# kubectl get nodes
NAME     STATUS   ROLES           AGE     VERSION
master   Ready    control-plane   2m54s   v1.24.3+k0s
node1    Ready    <none>          112s    v1.24.3+k0s
node2    Ready    <none>          113s    v1.24.3+k0s
node3    Ready    <none>          108s    v1.24.3+k0s

Step 1 – Create a Persistent Storage for Vault

Vault requires one to create a Storage class with a Persistent volume configured. Begin by creating the namespace and set it as default:

kubectl create namespace vault
kubectl config set-context --current --namespace vault

Example on manual storage class creation (Only for reference)

If you have an existing default StorageClass configured in your cluster you can skip this step.

Create a storage class;

vim storageClass.yml

Add the below lines to the file:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: my-local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

Create the storage class and set it as default.

kubectl create -f storageClass.yml
kubectl patch storageclass my-local-storage -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

Then create a PV using the storage class:

vim vault-pv.yml

Add the lines below:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-local-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: my-local-storage
  local:
    path: /mnt/disk/vol1
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - node1

With the node affinity set to node1, create the storage path on node1 as shown.

DIRNAME="vol1"
sudo mkdir -p /mnt/disk/$DIRNAME 
sudo chcon -Rt svirt_sandbox_file_t /mnt/disk/$DIRNAME
sudo chmod 777 /mnt/disk/$DIRNAME

Apply the manifest:

kubectl create -f vault-pv.yml

Step 2 – Run HashiCorp Vault on Kubernetes

This guide offers the easiest way to deploy Vault on Kubernetes using Helm. First, ensure that helm is installed on your system.

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
chmod 700 get_helm.sh
sudo ./get_helm.sh

Once installed, verify as shown:

$ helm version
version.BuildInfo{Version:"v3.9.2", GitCommit:"1addefbfe665c350f4daf868a9adc5600cc064fd", GitTreeState:"clean", GoVersion:"go1.17.12"}

Git clone into the official Vault helm chart maintained by Hashicorp:

git clone https://github.com/hashicorp/vault-helm.git
cd vault-helm

In the directory, you can customize the values.yaml as desired, then install Vault as shown:

helm install vault . --namespace vault --values values.yaml

Verify if the pods are running:

# kubectl get pods -n vault
NAME                                   READY   STATUS    RESTARTS   AGE
vault-0                                0/1     Running   0          75s
vault-agent-injector-57f567889-js274   1/1     Running   0          80s

You will also have a PVC created using the default storage class.

# kubectl get pvc
NAME           STATUS   VOLUME        CAPACITY   ACCESS MODES   STORAGECLASS       AGE
data-vault-0   Bound    my-local-pv   10Gi       RWO            my-local-storage   9s

Initialize the vault cluster using the command:

kubectl exec -ti --namespace vault vault-0 vault operator init

The command should return the initial token and unseal keys as shown;

kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
Unseal Key 1: vjwrDznfPk/7kHWY8L4OQL4PwXSuYFo3z45lt5SHolxj
Unseal Key 2: mnBo0TJGqDI1Qld18gM4kg6b58GYjLzKMAWaSX9uVwEg
Unseal Key 3: 6QWSei7R7re4sFlyz7os1TNpdxoJzpFOCvmhk09xIMWD
Unseal Key 4: iGZm2RiEQK3//RtUosUftb5dFU1R1YlqZmLQJJk7+I1I
Unseal Key 5: cnC9fyyxb4cBgKAKUbjTXT2R+y0CmyP/Ve7AlNvKZbut

Initial Root Token: hvs.QqliIhOLvhZq0nTCbKLAZXor

Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated root key. Without at least 3 keys to
reconstruct the root key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.

Step 3 – Configuring HashiCorp Vault Kubernetes Service

Once HashiCorp Vault has been deployed, a ClusterIP service will be deployed as well running on ports 8200 and 8201.

# kubectl get svc
NAME                       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
vault                      ClusterIP   10.102.36.175   <none>        8200/TCP,8201/TCP   21m
vault-agent-injector-svc   ClusterIP   10.103.189.23   <none>        443/TCP             21m
vault-internal             ClusterIP   None            <none>        8200/TCP,8201/TCP   21m

In order to access this service remotely, we need to configure the service as NodePort, LoadBalancer, or use an Ingress service.

In this guide, we will configure the service as a LoadBalancer. Begin by installing MetalLB using our guide below:

• How To Deploy MetalLB Load Balancer on Kubernetes Cluster

Now edit the HashiCorp Vault service using the command:

kubectl edit svc/vault

In the opened file, change the type to LoadBalancer:

...... 
  sessionAffinity: None
  type: LoadBalancer
.......

Save the file and check the external IP Address associated with the service:

# kubectl get svc
NAME                       TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                         AGE
vault                      LoadBalancer   10.102.36.175   192.168.205.40   8200:30996/TCP,8201:30620/TCP   146m
vault-agent-injector-svc   ClusterIP      10.103.189.23   <none>           443/TCP                         146m
vault-internal             ClusterIP      None            <none>           8200/TCP,8201/TCP               146m

Now you can access the service using Load Balancer on http://external_IP:8200

Unseal vault by providing 3 unseal keys then proceed and provide the initial root token to be authenticated to the dashboard.

On successful authentication, you will see this Vault dashboard

Step 4 – Connect to Vault from Kubernetes

To connect to the vault, you need to start an interactive shell on the pod vault-0 using the command:

kubectl exec -it vault-0 /bin/sh

Login to the vault using the root token

$ vault login 
Token (will be hidden):  <root initial token>

Check the status:

/ $ vault status
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    5
Threshold       3
Version         1.11.2
Build Date      2022-07-29T09:48:47Z
Storage Type    file
Cluster Name    vault-cluster-3d5e5ad2
Cluster ID      be268c68-646d-e4bd-9acf-c20c2ace1a91
HA Enabled      false

A. Creating a secret

For this guide, we will create a secret that will be used by applications later at internal/database/config. First, enable a kv secret engine

/ $ vault secrets enable -path=internal kv-v2
Success! Enabled the kv-v2 secrets engine at: internal/

Create the secret with the variables such as username and password to be used later:

vault kv put internal/database/config username="test_db" password="Passw0rd"

Verify the creation using the command:

vault kv get internal/database/config

Sample Output:

======== Secret Path ========
internal/data/database/config

======= Metadata =======
Key                Value
---                -----
created_time       2022-08-11T09:12:52.86722059Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

====== Data ======
Key         Value
---         -----
password    Passw0rd!
username    test_db

Proceed and create a policy called internal-app that enables create, read, update, delete and list capabilities:

vault policy write internal-app - <<EOH
path "internal/data/database/config" {
  capabilities = ["create", "read", "update", "delete", "list"]
}
EOH

B. Enable Kubernetes Authentication

To enable the clients to authenticate using a Kubernetes Service Account Token, enable the Kubernetes authentication method:

vault auth enable kubernetes

Vault will now accept the service token from any client within your Kubernetes cluster. This is done by querying a configured Kubernetes endpoint. We will configure the authentication method to use the below variables:

vault write auth/kubernetes/config \
        token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
        kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
        kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Create a role(internal-app) for the Kubernetes auth method to be used:

vault write auth/kubernetes/role/internal-app \
        bound_service_account_names=internal-app \
        bound_service_account_namespaces=vault \
        policies=internal-app \
        ttl=24h

The role will connect to a Kubernetes service account internal-app in the vault namespace and the tokens returned are valid for 24 hours.

Now exit the shell on vault-0 pod:

/ $ exit

C. Create a Kubernetes service account

We defined a Kubernetes service account named internal-app but the service account does not exist yet. We need to create the service account

kubectl apply \
  --filename=-<<EOH
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: internal-app
  namespace: vault
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
  namespace: vault
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: internal-app
  namespace: vault
EOH

Apply a cluster role binding for the ExternalSecrets’ service account with the role auth-delegator.

kubectl apply \
  --filename=-<<EOH
---
apiVersion: v1
kind: Secret
metadata:
  name: vault-auth-secret
  annotations:
    kubernetes.io/service-account.name: internal-app
type: kubernetes.io/service-account-token
EOH

Verify if the service account has been created:

# kubectl get sa
NAME                   SECRETS   AGE
default                0         102m
internal-app           0         58m
vault                  0         102m
vault-agent-injector   0         102m

Step 5 – Test Secret Injection to an Application

To demonstrate the secret injection, we will deploy a simple application using the manifest:

vim deployment-01-orgchart.yml

Add the lines below to the file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orgchart
  labels:
    app: vault-agent-injector-demo
spec:
  selector:
    matchLabels:
      app: vault-agent-injector-demo
  replicas: 1
  template:
    metadata:
      annotations:
      labels:
        app: vault-agent-injector-demo
    spec:
      serviceAccountName: internal-app
      containers:
        - name: orgchart
          image: jweissig/app:0.0.1

Apply the manifest using the command:

kubectl apply --filename deployment-01-orgchart.yml

Check if the pod is running:

$ kubectl get pods
NAME                                   READY   STATUS    RESTARTS      AGE
orgchart-56f7795cc5-cn7dp              1/1     Running   0             6s
vault-0                                0/1     Running   2 (57s ago)   19m
vault-agent-injector-dbc799985-sbwk4   1/1     Running   0             19m

The Vault-Agent injector traces the deployments that define specific annotations, none of them currently exist in our deployment. Verify that with the command:

$ kubectl exec orgchart-56f7795cc5-cn7dp --container orgchart -- ls /vault/secrets
ls: /vault/secrets: No such file or directory
command terminated with exit code 1

The Vault-Agent injector will only modify a deployment if it contains a specified set of annotations. We will create the annotations and patch them into the deployment.

vim deployment-02-inject-secrets.yml

Add the below lines to the file:

spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "internal-app"
        vault.hashicorp.com/agent-inject-secret-database-config.txt: "internal/data/database/config"

In the above file:

• agent-inject: set to true enables the Vault Agent injector service
• role: the Vault Kubernetes authentication role created before. It maps to the k8S service account
• agent-inject-secret-FIlEPATH: this is the path of the file, database-config.txt written to /vault/secrets

Now path the deployment:

kubectl patch deployment orgchart --patch "$(cat deployment-02-inject-secrets.yml)"

After this, you will have the deployment with two containers orgchart and the Vault Agent container, with the name vault-agent

# kubectl get pods
NAME                                   READY   STATUS    RESTARTS        AGE
orgchart-77d65d6b59-28xfj          2/2     Running   0               11m
vault-0                                1/1     Running   3 (7m34s ago)   33m
vault-agent-injector-dbc799985-sbwk4   1/1     Running   0               33m

View the logs in the container:

kubectl logs orgchart-77d65d6b59-28xfj  --container vault-agent

Sample Output:

Verify if the secret has been written to the container;

kubectl exec orgchart-77d65d6b59-28xfj --container orgchart -- cat /vault/secrets/database-config.txt

Sample Output:

From the above output, we can verify that HashiCorp Vault is working as desired since we have the secrets injected into the container. For more details, on how to customize a template view the guide on:

• Use Vault-Agent sidecar to inject Secrets in Vault to Kubernetes Pod

Books For Learning Kubernetes Administration:

• Best Kubernetes Study books

Closing Thoughts

At this point, you should be able to deploy HashiCorp Vault for Secrets Management in Kubernetes. This will eliminate the overheads involved when managing application/service secrets in your Kubernetes environment. I hope this was significant

Related:

• Integrate Multiple Kubernetes Clusters to Vault Server
• Manage Google Cloud Service Accounts with Vault and Terraform
• Gitleaks – How to Audit git repository for secrets