Install Vault Cluster in GKE via Helm, Terraform and BitBucket Pipelines

Motivation

The management of secrets in an organisation holds a special place in the running of day to day activities of the business. All the way from access to the building down to securing personal and confidential documents in laptops or computers, secrets continually show up which speaks about the importance secrets wield not only in our personal lives but in the highways of businesses. A secret is anything that you want to tightly control access to, such as API encryption keys, passwords, or certificates.

Taking it away from there, most applications in the current era are going towards the micro-services way and Kubernetes has come strong as the best platform to host the applications designed in this new paradigm. Kubernetes brought about new opportunities and a suite of challenges at the same time. It brought about agility, self healing, ease of scalability, ease of deployment and a good way of running decoupled systems. Now comes the issue of secrets and Kubernetes provides a way of managing them natively. The only problem with it is that it works if the workloads being run are few or the team managing the cluster is relatively small. When the applications being spawned are in the range of hundreds, it becomes difficult to manage secrets in that manner. Moreover, the native Kubernetes secrets engine lacks the capability of encryption which brings an issue of security to the fore.

HashiCorp Vault is a secrets management solution that is strongly designed to provide the management of secrets at scale, with ease and it integrates well with a myriad of other tools Kubernetes included. It is an identity-based secrets and encryption management system. Let us see the features Vault comes with:

Features of Vault

The key features of Vault are: Source Vault Documentation

• Secure Secret Storage: Arbitrary key/value secrets can be stored in Vault. Vault encrypts these secrets prior to writing them to persistent storage, so gaining access to the raw storage isn’t enough to access your secrets. Vault can write to disk, Consul, and more.
• Dynamic Secrets: Vault can generate secrets on-demand for some systems, such as AWS or SQL databases. For example, when an application needs to access an S3 bucket, it asks Vault for credentials, and Vault will generate an AWS key-pair with valid permissions on demand. After creating these dynamic secrets, Vault will also automatically revoke them after the lease is up.
• Data Encryption: Vault can encrypt and decrypt data without storing it. This allows security teams to define encryption parameters and developers to store encrypted data in a location such as SQL without having to design their own encryption methods.
• Leasing and Renewal: All secrets in Vault have a lease associated with them. At the end of the lease, Vault will automatically revoke that secret. Clients are able to renew leases via built-in renew APIs.
• Revocation: Vault has built-in support for secret revocation. Vault can revoke not only single secrets, but a tree of secrets, for example all secrets read by a specific user, or all secrets of a particular type. Revocation assists in key rolling as well as locking down systems in the case of an intrusion.

Project Pre-requisites

• BitBucket account
• BitBucket Pipelines already setup
• Docker or any tool to create images like Podman, Buildah etc
• Existing Google Cloud Credentials (json) in BitBucket Environment variable
• An existing Google Cloud Bucket for Terraform Backend (where it will keep state)

We will create terraform scripts, push it to BitBucket whence BitBucket pipelines will take over and deploy vault in Google Kubernetes Engine (GKE) using the image we will build.

Installation of Vault Cluster in Google Kubernetes Engine

We can now embark on setting up Vault on an existing Google Kubernetes Engine via Helm, BitBucket pipelines and Terraform. The following are the steps that will get you up and running. Some parts are optional in case you do not use BitBucket pipelines in your setup.

Step 1: Prepare Terraform and Google SDK Image

In this step we are going to create a Docker image that has Terraform and Google Cloud SKD and then host it in DockerHub so that BitBucket can pull and use it while deploying the infrastructure. First let us create Dockerfile file and populate it with the following. We will use Google’s cloudsdk image as the base then add Terraform.

$ vim Dockerfile
FROM gcr.io/google.com/cloudsdktool/cloud-sdk:alpine
ENV TERRAFORM_VERSION=1.0.10
# Installing terraform
RUN apk --no-cache add curl unzip && \
    cd /tmp && \
    curl -o /tmp/terraform.zip https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip && \
    unzip /tmp/terraform.zip && \
    cp /tmp/terraform /usr/local/bin && \
    chmod a+x /usr/local/bin/terraform && \
    rm /tmp/terraform /tmp/terraform.zip

After that, let us build and tag the image. Make sure the Dockerfile file is in the same place you are running this command.

docker build -label imagename .

Tag the image

docker tag imagename penchant/cloudsdk-terraform:latest

Then push it to public DockerHub or any registry you prefer

docker push penchant/cloudsdk-terraform:latest

And we are done with the first part

Step 2: Prepare Terraform and Helm scripts

In order to avoid re-inventing the wheel, this project heavily borrows from a project already in GitHub by mohsinrz. We are grateful and celebrate them for the fine work they have done. We will clone the project and then customise it to befit our environment.

cd ~
git clone https://github.com/mohsinrz/vault-gke-raft.git

Since we already have a GKE Cluster, we will not use the module geared towards creating one. We will further disable the use of certificates because BitBucket uses an ephemeral container and will not be able to store certificates in it and we will have trouble joining vault workers to the leader later.

We will add GCP bucket to store Terraform state so that we can track changes in the what we will be deploying. Add the following in the “main.yaml” file. Ensure that the bucket name already exists in GCP.

$ cd ~/vault-gke-raft
$ vim main.tf

## Disable the gke cluster module if you have on already
#module "gke-cluster" {
#  source                     = "./modules/google-gke-cluster/"
#  credentials_file           = var.credentials_file
#  region                     = var.region
#  project_id                 = "project-id"
#  cluster_name               = "dev-cluster-1"
#  cluster_location           = "us-central1-a"
#  network                    = "projects/${var.project_id}/global/networks/default"
#  subnetwork                 = "projects/${var.project_id}/regions/${var.region}/subnetworks/default"
#  initial_node_count         = var.cluster_node_count
#}

module "tls" {
  source                     = "./modules/gke-tls"
  hostname                   = "*.vault-internal"
}

module "vault" {
  source                     = "./modules/gke-vault"
  num_vault_pods             = var.num_vault_pods
  #cluster_endpoint           = module.gke-cluster.endpoint
  #cluster_cert               = module.gke-cluster.ca_certificate
  vault_tls_ca               = module.tls.ca_cert
  vault_tls_cert             = module.tls.cert 
  vault_tls_key              = module.tls.key
}

terraform {
  backend "gcs"{
    bucket      = "terraform-state-bucket"
    credentials = "gcloud-api-key.json"
  }
}

Another modification we shall make is disable TLS because in our setup, an ephemeral container in BitBucket will provision our infrastructure and some of the certificates are meant to be stored where terraform is running. So we get to lose the certificates after the deployment is done. To disable, navigate to the modules folder and into the vault directory module. Then edit the “vault.tf” file and make it like below. Changes made are:

• we changed tlsDisable field to true from false
• we changed VAULT_ADDR environment variable from https to http
• we commented/removed VAULT_CACERT environment variable
• we changed tls_disable field from 0 to 1
• we changed VAULT_ADDR from 127.0.0.1 to 0.0.0.0
• And removed the certificate paths under listener block

The same has been updated in the file below.

$ cd ~/vault-gke-raft/modules/vault
$ vim vault.tf

resource "helm_release" "vault" {
  name          = "vault"
  chart         = "${path.root}/vault-helm"
  namespace     = kubernetes_namespace.vault.metadata.0.name

  values = [<<EOF
global:
## changed tlsDisable field to true from false
  #tlsDisable: false
  tlsDisable: true
server:
  extraEnvironmentVars:
#changed VAULT_ADDR environment variable from https to http
    #VAULT_ADDR: https://127.0.0.1:8200
    VAULT_ADDR: http://0.0.0.0:8200
    VAULT_SKIP_VERIFY: true
#removed VAULT_CACERT environment variable
    #VAULT_CACERT: /vault/userconfig/vault-tls/vault.ca
  extraVolumes:
    - type: secret
      name: vault-tls
  ha:
    enabled: true
    replicas: ${var.num_vault_pods}    

    raft:      
      # Enables Raft integrated storage
      enabled: true
      config: |
        ui = true

        listener "tcp" {
#changed tls_disable field from 0 to 1
          #tls_disable = 0
          tls_disable = 1
          address = "[::]:8200"
          cluster_address = "[::]:8201"
#removed the certificate paths here
          #tls_cert_file = "/vault/userconfig/vault-tls/vault.crt"
          #tls_key_file  = "/vault/userconfig/vault-tls/vault.key"
          #tls_client_ca_file = "/vault/userconfig/vault-tls/vault.ca"           
        }

        storage "raft" {
          path = "/vault/data"
        }
ui:
  enabled: true
  serviceType: "LoadBalancer"
  serviceNodePort: null
  externalPort: 8200
EOF
]
}

We will make one more modification that will enable Kubernetes provider to communicate with GKE API. Navigate to the modules folder and into the vault module directory. Then edit the “provider.tf” file. We have added details of the GKE cluster that already exists and used the values in the Kubernetes provider. We commented the one that we fetched from the repo and added the new one as shown below. The helm provider has been edited as well by updating the host, token and cluster ca certificate with what already exists.

$ cd ~/vault-gke-raft/modules/vault
$ vim provider.tf
data "google_client_config" "provider" {}

data "google_container_cluster" "cluster-name" {
  name     = "cluster-name"
  location = "us-central1-a"
  project  = "project-name"
}

# This file contains all the interactions with Kubernetes
provider "kubernetes" {
  #host = google_container_cluster.vault.endpoint
  host = "https://${data.google_container_cluster.dev_cluster_1.endpoint}"
  token = data.google_client_config.provider.access_token
  cluster_ca_certificate = base64decode(
    data.google_container_cluster.dev_cluster_1.master_auth[0].cluster_ca_certificate,
  )
}

#provider "kubernetes" {
#  host  = var.cluster_endpoint
#  token = data.google_client_config.current.access_token
#
#  cluster_ca_certificate = base64decode(
#    var.cluster_cert,
#  )
#}

provider "helm" {
  kubernetes {
    #host  = var.cluster_endpoint
    host  = "https://${data.google_container_cluster.dev_cluster_1.endpoint}"
    #token = data.google_client_config.current.access_token
    token = data.google_client_config.provider.access_token
    cluster_ca_certificate = base64decode(data.google_container_cluster.dev_cluster_1.master_auth[0].cluster_ca_certificate,
    )
  }
}

After we are done editing the files, let us clone vault-helm in the root directory that will deploy the entire infrastructure for us at a go via terraform helm provider

cd ~/vault-gke-raft
git clone https://github.com/hashicorp/vault-helm

Step 3: Create BitBucket pipelines file

In this step, we are going to create and populate the BitBucket pipelines file that will steer our deployment. As you can see, we are using the image we pushed to DockerHub in Step 1.

$ cd ~
$ vim bitbucket-pipelines.yaml

image: penchant/cloudsdk-terraform:latest ## The image
pipelines:
  branches:
    vault:
      - step:
          name: Deploy to Vault Namespace
          deployment: production
          script:
            - cd install-vault # I placed my files in this directory in the root of the files
            - export TAG=$(git log -1 --pretty=%h)
            - echo $GCLOUD_API_KEYFILE | base64 -d  > ./gcloud-api-key.json
            - gcloud auth activate-service-account --key-file gcloud-api-key.json
            - export GOOGLE_APPLICATION_CREDENTIALS=gcloud-api-key.json
            - gcloud config set project <your_project_id>
            - gcloud container clusters get-credentials <your_cluster> --zone=<your_zone> --project <your_project_id>
            - terraform init
            - terraform plan -out create_vault 
            - terraform apply -auto-approve create_vault
          services:
            - docker

Step 4: Initialise cluster by creating the leader node/pod

After installing vault cluster via terraform and helm, it is time to bootstrap the cluster and unseal it. Initialise the cluster by making vault-0 node as the leader then we can unseal it and then later join the rest of the nodes to the cluster and unseal them as well.

Initialize the cluster by making node vault-0 as the leader as shown follows:

$ kubectl exec -ti vault-0 -n vault -- vault operator init

Unseal Key 1: 9LphBlg31dBKuVCoOYRW+zXrS5zpuGeaFDdCWV3x6C9Y
Unseal Key 2: zfWTzDo9nDUIDLuqRAc4cVih1XzuZW8iEolc914lrMyS
Unseal Key 3: 2O3QUiio8x5W+IJq+4ze45Q3INL1Ek/2cHDiNHb3vXIz
Unseal Key 4: DoPBFgPte+Xh6L/EljPc79ZT2mYwQL6IAeDTLiBefwPV
Unseal Key 5: OW1VTaXIMDt0Q57STeI4mTh1uBFPJ2JvmS2vgYAFuCPJ

Initial Root Token: s.rLs6ycvvg97pQNnvnvzNZgAJ

Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated master key. Without at least 3 keys to
reconstruct the master key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.

Now we have the keys and the root token. We will use the keys to unseal the nodes and join every node to the cluster.

Step 5: Unsealing the leader node/pod

We will use the unseal keys from the output of above command in Step 4 to unseal Vault as shown below. Run the command three times and supply the keys in the order they have been generated above. After running the command, you will be presented with a prompt where you are required to enter one of the seals generated above. Simply copy and paste one of them and hit enter.

$ kubectl exec -ti vault-0 -n vault -- vault operator unseal

Unseal Key (will be hidden): <Enter Unseal Key 1>
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    1/3
Unseal Nonce       f4c34433-6ef1-59ca-c1c9-1a6cc0dfabff
Version            1.8.4
Storage Type       raft
HA Enabled         true

Run it the second time

$ kubectl exec -ti vault-0 -n vault -- vault operator unseal

Unseal Key (will be hidden): <Enter Unseal Key 2>
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    2/3
Unseal Nonce       f4c34433-6ef1-59ca-c1c9-1a6cc0dfabff
Version            1.8.4
Storage Type       raft
HA Enabled         true

Run it again the third time

$ kubectl exec -ti vault-0 -n vault -- vault operator unseal

Unseal Key (will be hidden): <Enter Unseal Key 3>
Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            5
Threshold               3
Version                 1.8.4
Storage Type            raft
Cluster Name            vault-cluster-3d108027
Cluster ID              a75de185-7b51-6045-20ca-5a25ca9d9e70
HA Enabled              true
HA Cluster              n/a
HA Mode                 standby
Active Node Address     <none>
Raft Committed Index    24
Raft Applied Index      24

For now, we have not added the remaining four nodes/pods of vault statefulsets into the cluster. If you check the status of the pods, you will see the they are not ready. Let us confirm that.

$ kubectl get pods -n vault

NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 1/1     Running   0          14m
vault-1                                 0/1     Running   0          14m
vault-2                                 0/1     Running   0          14m
vault-3                                 0/1     Running   0          14m
vault-4                                 0/1     Running   0          14m
vault-agent-injector-5c8f78854d-twllz   1/1     Running   0          14m

As you can see, vault-1 through to vault-4 are not ready (0/1)

Step 6: Add the rest of the nodes to the cluster and Unsealing them

This is the step where we are going to add each one of them to the cluster step by step. The procedure is as follows:

• Add a node to the cluster
• Then unseal it using the number of threshold shown in the status command above (kubectl exec -ti vault-0 -n vault — vault status). It is 3 in this example.
• This means we will run the unseal command three times for each node. When you are unsealing, use the same keys that we used for the node 1 above. Let us get rolling.

Join the other nodes to the cluster starting with vault-1 node.

$ kubectl exec -ti vault-1 -n vault --  vault operator raft join --address "http://vault-1.vault-internal:8200" "http://vault-0.vault-internal:8200"     

Key       Value
---       -----
Joined    true

After it has successfully joined, unseal the node three times.

## First Time
$ kubectl exec -ti vault-1 -n vault -- vault operator unseal
Unseal Key (will be hidden): <Enter Unseal Key 1>
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    1/3
Unseal Nonce       188f79d8-a87f-efdf-4186-73327ade371a
Version            1.8.4
Storage Type       raft
HA Enabled         true

## Second Time
$ kubectl exec -ti vault-1 -n vault -- vault operator unseal
Unseal Key (will be hidden): <Enter Unseal Key 2>
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    2/3
Unseal Nonce       188f79d8-a87f-efdf-4186-73327ade371a
Version            1.8.4
Storage Type       raft
HA Enabled         true

## Third Time
$ kubectl exec -ti vault-1 -n vault -- vault operator unseal
Unseal Key (will be hidden): <Enter Unseal Key 3>
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    0/3
Unseal Nonce       n/a
Version            1.8.4
Storage Type       raft
HA Enabled         true

Then join node vault-2 to the cluster and then unseal it just like it was done in node vault-1

$ kubectl exec -ti vault-2 -n vault --  vault operator raft join --address "http://vault-2.vault-internal:8200" "http://vault-0.vault-internal:8200"

Key       Value
---       -----
Joined    true

Unseal node vault-2 three times entering one of the 3 keys on each run

##First Time
$ kubectl exec -ti vault-2 -n vault -- vault operator unseal

Unseal Key (will be hidden): <Enter Unseal Key 1>
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    1/3
Unseal Nonce       60ab7a6a-e7dc-07c8-e73c-11c55bafc199
Version            1.8.4
Storage Type       raft
HA Enabled         true

##Second time
$ kubectl exec -ti vault-2 -n vault -- vault operator unseal
Unseal Key (will be hidden): <Enter Unseal Key 2>
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    2/3
Unseal Nonce       60ab7a6a-e7dc-07c8-e73c-11c55bafc199
Version            1.8.4
Storage Type       raft
HA Enabled         true

##Third Time
$ kubectl exec -ti vault-2 -n vault -- vault operator unseal
Unseal Key (will be hidden): <Enter Unseal Key 3>
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    0/3
Unseal Nonce       n/a
Version            1.8.4
Storage Type       raft
HA Enabled         true

Do the same for the remaining pods/nodes in your cluster that are still not ready. Simply check your pods as follows

$ kubectl get pods -n vault

NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 1/1     Running   0          14m
vault-1                                 1/1     Running   0          14m
vault-2                                 1/1     Running   0          14m
vault-3                                 0/1     Running   0          14m
vault-4                                 0/1     Running   0          14m
vault-agent-injector-5c8f78854d-twllz   1/1     Running   0          14m

The ones with 0/1 are not yet ready so join them to the cluster and unseal them.

Add Node vault-3

$ kubectl exec -ti vault-3 -n vault --  vault operator raft join --address "http://vault-3.vault-internal:8200" "http://vault-0.vault-internal:8200"

Key       Value
---       -----
Joined    true

Unseal Node vault-3 three times again.

##First Time
$ kubectl exec -ti vault-3 -n vault -- vault operator unseal

Unseal Key (will be hidden): <Enter Unseal Key 1>
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    1/3
Unseal Nonce       733264c0-bfc6-6869-a3dc-167e642ad624
Version            1.8.4
Storage Type       raft
HA Enabled         true

##Second Time
$ kubectl exec -ti vault-3 -n vault -- vault operator unseal

Unseal Key (will be hidden): <Enter Unseal Key 2>
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    2/3
Unseal Nonce       733264c0-bfc6-6869-a3dc-167e642ad624
Version            1.8.4
Storage Type       raft
HA Enabled         true

##Third Time
$ kubectl exec -ti vault-3 -n vault -- vault operator unseal

Unseal Key (will be hidden): <Enter Unseal Key 3>
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    0/3
Unseal Nonce       n/a
Version            1.8.4
Storage Type       raft
HA Enabled         true

Add Node vault-4

$ kubectl exec -ti vault-4 -n vault --  vault operator raft join --address "http://vault-4.vault-internal:8200" "http://vault-0.vault-internal:8200"

Key       Value
---       -----
Joined    true

Unseal Node vault-4 three times once more.

##First Time
$ kubectl exec -ti vault-4 -n vault -- vault operator unseal

Unseal Key (will be hidden): <Enter Unseal Key 1>
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    1/3
Unseal Nonce       543e3a67-28f9-9730-86ae-4560d48c2f2e
Version            1.8.4
Storage Type       raft
HA Enabled         true

##Second Time
$ kubectl exec -ti vault-4 -n vault -- vault operator unseal

Unseal Key (will be hidden): <Enter Unseal Key 2>
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    2/3
Unseal Nonce       543e3a67-28f9-9730-86ae-4560d48c2f2e
Version            1.8.4
Storage Type       raft
HA Enabled         true

##Third Time
$ kubectl exec -ti vault-4 -n vault -- vault operator unseal

Unseal Key (will be hidden): <Enter Unseal Key 3>
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    0/3
Unseal Nonce       n/a
Version            1.8.4
Storage Type       raft
HA Enabled         true

Now lets check our pods

$ kubectl get pods -n vault
NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 1/1     Running   0          32m
vault-1                                 1/1     Running   0          32m
vault-2                                 1/1     Running   0          32m
vault-3                                 1/1     Running   0          32m
vault-4                                 1/1     Running   0          32m
vault-agent-injector-5c8f78854d-twllz   1/1     Running   0          32m

Beautiful. You can see that all of them are successfully ready. And we are finally done with our vault ha cluster setup in GKE platform. Next, we shall cover how to authenticate to Kubernetes/GKE, create secrets then launch a sample app and inject secrets to the pod via sidecar model.

• Use Vault-Agent sidecar to inject Secrets in Vault to Kubernetes Pod

We hope the document provides insight and has been helpful for your use case. In case you have an idea of how to auto-unseal, kindly point us in the right direction as well.

Lastly, the tremendous support and messages we continue to receive is a blessing and we pray that you continue to prosper in your various endeavours as you change the world. Have an amazing end year season and keep at it, keep the safety and may your hard work yield the fruits you desire 🙂.

Other guides you will enjoy include:

• How to install gcloud GCP cli on Apple M1 macOS
• Deploy Nginx Ingress Controller on Kubernetes using Helm Chart
• Separating Jenkinsfile and Kubernetes Deployment YAML from Git Sources
• Deploy Kubernetes Cluster on Linux With k0s
• Run Kubernetes on Debian with Minikube