Configure Kubernetes Dynamic Volume Provisioning With Heketi & GlusterFS

The Dynamic volume provisioning in Kubernetes allows storage volumes to be created on-demand, without manual Administrator intervention. When developers are doing deployments without dynamic provisioning, cluster administrators have to manually make calls to their cloud or storage provider to create new storage volumes, from where the PersistentVolumes are created.

This guide will discuss how you can achieve Dynamic Volume Provisioning on Kubernetes by using GlusterFS distributed storage solution and Heketi RESTful management interface. It is expected you have deployed Heketi and GlusterFS scale-out network-attached storage file system.

For Ceph, check:

• Ceph Persistent Storage for Kubernetes with Cephfs
• Persistent Storage for Kubernetes with Ceph RBD

How Dynamic Provisioning is configured in Kubernetes

In Kubernetes, dynamic volume provisioning is based on the API object StorageClass from the API group storage.k8s.io. As a cluster administrator, you’ll define as many StorageClass objects as needed, each specifying a volume plugin ( provisioner) that provisions a volume and the set of parameters to pass to that provisioner when provisioning.

So below are the steps you’ll use to configure Dynamic Volume Provisioning on Kubernetes using Gluster and Heketi API.

1) Setup GlusterFS and Heketi

It is expected you have a running Gluster and Heketi before you continue with configurations on the Kubernetes end. Refer to our guide below on setting them up.

• Setup GlusterFS Storage With Heketi on CentOS 8 / CentOS 7

At the moment we only have guide for CentOS, but we’re working on a deployment guide for Ubuntu/Debian systems.

For containerized setup, check: Setup Kubernetes / OpenShift Dynamic Persistent Volume Provisioning with GlusterFS and Heketi

Once the installation is done, proceed to step 2:

2) Create StorageClass Object on Kubernetes

We need to create a StorageClass object to enable dynamic provisioning for container platform users. The StorageClass objects define which provisioner should be used and what parameters should be passed to that provisioner when dynamic provisioning is invoked.

• Check your Heketi Cluster ID

$ heketi-cli cluster list
Clusters:
Id:b182cb76b881a0be2d44bd7f8fb07ea4 [file][block]

• Create Kubernetes Secret

Get a base64 format of your Heketi admin user password.

echo -n "PASSWORD" | base64

Then create a secret with the password for accessing Heketi.

$ vim gluster-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: heketi-secret
  namespace: default
type: "kubernetes.io/glusterfs"
data:
  # echo -n "PASSWORD" | base64
  key: cGFzc3dvcmQ=

Where:

• cGFzc3dvcmQ= is the output of echo command.

Create the secret by running the command:

kubectl create -f gluster-secret.yaml

Confirm secret creation.

$ kubectl get secret
NAME                                               TYPE                                  DATA   AGE
heketi-secret                                      kubernetes.io/glusterfs               1      1d

• Create StorageClass

Below is a sample StorageClass for GlusterFS using Heketi.

$ cat glusterfs-sc.yaml
 kind: StorageClass
 apiVersion: storage.k8s.io/v1beta1
 metadata:
   name: gluster-heketi
 provisioner: kubernetes.io/glusterfs
 reclaimPolicy: Delete
 volumeBindingMode: Immediate
 allowVolumeExpansion: true
 parameters:
   resturl: "http://heketiserverip:8080" 
   restuser: "admin" 
   secretName: "heketi-secret"
   secretNamespace: "default"
   volumetype: "replicate:2"
   volumenameprefix: "k8s-dev"
   clusterid: "b182cb76b881a0be2d44bd7f8fb07ea4"

Where:

• gluster-heketi is the name of the StorageClass to be created.
• The valid options for reclaim policy are Retain, Delete or Recycle. The Delete policy means that a dynamically provisioned volume is automatically deleted when a user deletes the corresponding PersistentVolumeClaim.
• The volumeBindingMode field controls when volume binding and dynamic provisioning should occur. Valid options are Immediate & WaitForFirstConsumer. The Immediate mode indicates that volume binding and dynamic provisioning occurs once the PersistentVolumeClaim is created. The WaitForFirstConsumer mode delays the binding and provisioning of a PersistentVolume until a Pod using the PersistentVolumeClaim is created.
• The resturl is the URL of your heketi endpoint
• heketi-secret is the secret created for Heketi credentials.
• default is the name of namespace where secret was created
• replicate:2 indicated the default replication factor for Gluster Volumes created. For more HA, use 3.
• volumenameprefix: By default dynamically provisioned volumes have the naming schema of vol_UUID format. We have provided a desired volume name from storageclass. So the naming scheme will be: volumenameprefix_Namespace_PVCname_randomUUID
• b182cb76b881a0be2d44bd7f8fb07ea4 is the ID of the cluster obtained from the command heketi-cli cluster list

Another parameter that can be set is:

volumeoptions: "user.heketi.zone-checking strict"

The default setting/behavior is:

volumeoptions: "user.heketi.zone-checking none"

This forces Heketi to strictly place replica bricks in different zones. The required minimum number of nodes required to be present in different zones is 3 if the replica value is set to 3.

Once the file is created, run the following command to create the StorageClass object.

$ kubectl create -f gluster-sc.yaml

Confirm StorageClass creation.

$ kubectl get sc
NAME                            PROVISIONER                    AGE
glusterfs-heketi                kubernetes.io/glusterfs        1d
local-storage                   kubernetes.io/no-provisioner   30d

3) Create PersistentVolumeClaim Object

When a user is requesting dynamically provisioned storage, a storage class should be included in the PersistentVolumeClaim.

Let’s create a 1GB request for storage:

$ vim glusterfs-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: gluster-pvc
 annotations:
   volume.beta.kubernetes.io/storage-class: gluster-heketi
spec:
 accessModes:
  - ReadWriteMany
 resources:
   requests:
     storage: 1Gi

Create object:

$ kubectl create --save-config -f glusterfs-pvc.yaml

Confirm:

$ kubectl get  pvc
NAME             STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
glusterfs-pvc    Bound     pvc-34b9b5e9-fbde-11e9-943f-00505692ee7e   1Gi        RWX            glusterfs-heketi  1d

After creation, you can use it in your deployments. To use the volume we reference the PVC in the YAML file of any Pod/Deployment like this for example:

apiVersion: v1
kind: Pod
metadata:
  name: gluster-pod
  labels:
    name: gluster-pod
spec:
  containers:
  - name: gluster-pod
    image: busybox
    command: ["sleep", "60000"]
    volumeMounts:
    - name: gluster-vol
      mountPath: /usr/share/busybox 
      readOnly: false
  volumes:
  - name: gluster-vol
    persistentVolumeClaim:
      claimName: glusterfs-pvc

That’s it for today. You should have a working Dynamic Volume Provisioning With Heketi & GlusterFS for your Kubernetes platform. Check other Kubernetes Storage related guides available in our blog.

• Configure NFS as Kubernetes Persistent Volume Storage
• How To Deploy Rook Ceph Storage on Kubernetes Cluster
• Deploy and Use OpenEBS Container Storage on Kubernetes

Books For Learning Kubernetes Administration:

• Best Kubernetes Study books

Tags:

• Kubernetes GlusterFS Configurration
• Kubernetes GlusterFS storage configuration
• Heketi and GlusterFS setup
• Kubernetes dynamic storage with Heketi and GlusterFS