How To Run Ansible AWX on Kubernetes / OpenShift Cluster

Welcome to today’s article on how to install, configure and use Ansible AWX on Kubernetes / OpenShift Cluster. Ansible AWX is a popular web-based platform created to provide a user-friendly interface for Ansible automation operations. AWX is built on top of the Ansible project and it gives you a centralized management for all Ansible inventories and playbooks.

From the dashboard you can manage inventory, run and schedule Ansible playbooks, monitor job activity and execution results. The built-in RBAC (role-based access control) is used for managing access to resources within AWX platform. An integration with other tools such as Git for version control is also possible.

Key features of Ansible AWX:

• Centralized management: It offers centralized management for Ansible playbooks and inventories for ease of administration.
• Role-based access control: With the built-in RBAC you can ensure there is a secure and efficient management of automation tasks.
• Web-based user interface: AWX comes with an intuitive user-friendly web interface for managing Ansible playbooks, inventory, and automation tasks.
• Third-party integrations: Integration with other tools such as Git for version control can be done making it easy to manage and track changes to Ansible playbooks over time.
• Reporting and analytics: AWX has reporting and analytics capabilities for tracking performance and gains from its use.

Run Ansible AWX on Kubernetes / OpenShift Cluster

In this section we’ll cover the full process of installing Ansible AWX on Kubernetes / OpenShift Cluster. A major pre-requisite for this operation is existing Kubernetes / OpenShift cluster.

Step 1: Deploy Kubernetes Cluster

You can use existing articles on our website to setup a cluster.

• Deploy Kubernetes on KVM using Flatcar Container Linux
• Deploy Kubernetes Cluster with Ansible & Kubespray
• Install Kubernetes Cluster on Ubuntu with kubeadm
• Install Kubernetes Cluster on Rocky Linux 8 with Kubeadm & CRI-O
• Deploy Lightweight Kubernetes Cluster in 5 minutes with K3s
• How To Deploy Highly Available Kubernetes Cluster Using k3sup

Once the cluster is deployed and running confirm using kubectl command:

$ kubectl cluster-info
Kubernetes control plane is running at https://api.k8s.cloudlabske.io:6443

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Step 2: Deploy MetalLB / Ingress solution

For Kubernetes clusters deployed on your on-prem infrastructure you’ll need a solution that provides Load Balancing feature and Ingress solution. For managed Kubernetes clusters such as ones provided by cloud providers like EKS, AKS and GKE, a manual setup is not required for LB.

We have existing articles on this setup.

• How To Deploy MetalLB Load Balancer on Kubernetes Cluster
• Deploy Nginx Ingress Controller on Kubernetes using Helm Chart

Step 3: Configure Persistent Storage Solution

We’ll need persistent storage for AWX data. Any solution integrated to Kubernetes can be used for this. Refer to some of the guides available in our website.

• Configure NFS as Kubernetes Persistent Volume Storage
• Deploy and Manage MinIO Storage on Kubernetes
• How To Deploy Rook Ceph Storage on Kubernetes Cluster
• Deploy and Use OpenEBS Container Storage on Kubernetes

In this setup we’re using NFS storage class to automatically provision persistent volumes on Kubernetes.

$ kubectl get sc
NAME               PROVISIONER          RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
nfs-01 (default)   nfs-provisioner-01   Delete          Immediate           true                   53d

This will be used when deploying AWX on Kubernetes / OpenShift cluster.

Step 4: Deploy AWX Operator on Kubernetes

We shall use the AWX Kubernetes Operator meant to be deployed in your Kubernetes cluster(s) and can manage one or more AWX instances in any namespace.

Install git and make tools:

### Ubuntu / Debian ###
sudo apt update
sudo apt install git build-essential curl jq  -y

### RHEL based systems ###
sudo yum -y install epel-release
sudo yum group install "Development Tools"
sudo yum install curl jq

Use git to clone Operator project source to your workstation with kubectl configured.

$ git clone https://github.com/ansible/awx-operator.git
Cloning into 'awx-operator'...
remote: Enumerating objects: 7925, done.
remote: Counting objects: 100% (403/403), done.
remote: Compressing objects: 100% (256/256), done.
remote: Total 7925 (delta 187), reused 316 (delta 121), pack-reused 7522
Receiving objects: 100% (7925/7925), 2.19 MiB | 739.00 KiB/s, done.
Resolving deltas: 100% (4498/4498), done.

Next we create a namespace where operator will be deployed. We’ll name this awx:

export NAMESPACE=awx
kubectl create ns ${NAMESPACE}

Set current context to value set in NAMESPACE variable:

# kubectl config set-context --current --namespace=$NAMESPACE 
Context "default" modified.

Switch to awx-operator directory:

cd awx-operator/

Save the latest version from AWX Operator releases as RELEASE_TAG variable then checkout to the branch.

RELEASE_TAG=`curl -s https://api.github.com/repos/ansible/awx-operator/releases/latest | grep tag_name | cut -d '"' -f 4`
echo $RELEASE_TAG
git checkout $RELEASE_TAG

Install AWX Operator into your Kubernetes cluster:

export NAMESPACE=awx
make deploy

Successful execution output:

namespace/awx configured
customresourcedefinition.apiextensions.k8s.io/awxbackups.awx.ansible.com created
customresourcedefinition.apiextensions.k8s.io/awxrestores.awx.ansible.com created
customresourcedefinition.apiextensions.k8s.io/awxs.awx.ansible.com created
serviceaccount/awx-operator-controller-manager created
role.rbac.authorization.k8s.io/awx-operator-awx-manager-role created
role.rbac.authorization.k8s.io/awx-operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/awx-operator-metrics-reader created
clusterrole.rbac.authorization.k8s.io/awx-operator-proxy-role created
rolebinding.rbac.authorization.k8s.io/awx-operator-awx-manager-rolebinding created
rolebinding.rbac.authorization.k8s.io/awx-operator-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/awx-operator-proxy-rolebinding created
configmap/awx-operator-awx-manager-config created
service/awx-operator-controller-manager-metrics-service created
deployment.apps/awx-operator-controller-manager created

Wait for the operator pod creation to be successful. The status should be running

$ kubectl get pods -w
NAME                                               READY   STATUS              RESTARTS   AGE
awx-operator-controller-manager-65b875f4bf-757vt   0/2     ContainerCreating   0          68s

Step 5: Install Ansible AWX on Kubernetes

With the AWX operator running we can use it to install AWX instance on the cluster. But first we have to create static data PVC – Ref AWX data persistence:

cat <<EOF | kubectl create -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: awx-static-data-pvc
  namespace: awx
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: nfs-01
  resources:
    requests:
      storage: 10Gi
EOF

Confirm if the PVC is created successfully.

$ kubectl get pvc
NAME                  STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
awx-static-data-pvc   Bound    pvc-184cec87-8447-4055-a672-55fe264acb27   10Gi       RWO            nfs-01         28s

Now create AWX deployment file. See example below.

$ vim awx-deployment.yml
---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
spec:
  service_type: nodeport
  projects_persistence: true
  projects_storage_access_mode: ReadWriteOnce
  web_extra_volume_mounts: |
    - name: static-data
      mountPath: /var/lib/projects
  extra_volumes: |
    - name: static-data
      persistentVolumeClaim:
        claimName:  awx-static-data-pvc

If the service_type is not specified, the ClusterIP service will be used for your AWX Tower service. The service_type supported options are: ClusterIP, LoadBalancer and NodePort.

Apply configuration file.

$ kubectl apply -f awx-deployment.yml
awx.awx.ansible.com/awx created

Give it a few seconds and AWX instance will be deployed and running:

$ kubectl get pods -l "app.kubernetes.io/managed-by=awx-operator" -w
NAME                  READY   STATUS    RESTARTS   AGE
awx-74bd7c8bc-4mqbb   4/4     Running   0          15m
awx-postgres-13-0     1/1     Running   0          31m

 You can track the installation process at the operator pod logs:

kubectl logs -f deployments/awx-operator-controller-manager -c awx-manager

Listing PVCs in the namespace:

$ kubectl get pvc
NAME                            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
awx-projects-claim              Bound    pvc-188c3463-65c0-4265-a00d-2a8573894ca7   8Gi        RWO            nfs-01         33m
awx-static-data-pvc             Bound    pvc-184cec87-8447-4055-a672-55fe264acb27   10Gi       RWO            nfs-01         49m
postgres-13-awx-postgres-13-0   Bound    pvc-84b68384-3c9c-4f81-bfab-f083c1cf10cd   8Gi        RWO            nfs-01         36m

Checking AWX Container’s logs

The awx-xxx-yyy pod will have four containers, namely:

• redis
• awx-web
• awx-task
• awx-ee

As can be seen from below command output:

# kubectl -n awx  logs deploy/awx
error: a container name must be specified for pod awx-75698588d6-r7bxl, choose one of: [redis awx-web awx-task awx-ee]

You’ll need to provide container name after the pod:

kubectl -n awx  logs deploy/awx -c redis
kubectl -n awx  logs deploy/awx -c awx-web
kubectl -n awx  logs deploy/awx -c awx-task
kubectl -n awx  logs deploy/awx -c awx-ee

Access AWX Container’s Shell

Here is how to access each container’s shell:

kubectl exec -ti deploy/awx  -c  awx-task -- /bin/bash
kubectl exec -ti deploy/awx  -c  awx-web -- /bin/bash
kubectl exec -ti deploy/awx  -c  awx-ee -- /bin/bash
kubectl exec -ti deploy/awx  -c  redis -- /bin/bash

Upgrading AWX Operator and instance

We have created a dedicated guide for upgrading the Operator and AWX instance:

• How to upgrade Ansible AWX running in Kubernetes

Step 6: Access Ansible AWX Dashboard

List all available services and check awx-service Nodeport

$ kubectl get svc
NAME                                              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
awx-operator-controller-manager-metrics-service   ClusterIP   10.233.25.242   <none>        8443/TCP       64m
awx-postgres-13                                   ClusterIP   None            <none>        5432/TCP       41m
awx-service                                       NodePort    10.233.28.124   <none>        80:30080/TCP   38m

From the output the service type is set to NodePort. Provide one of the cluster’s node IP address and port to access AWX web console.

http://hostip_or_hostname:30080

On accessing AWX web portal you are presented with the welcome dashboard similar to one below.

The login username is admin. To get the password run the commands below:

kubectl get secret awx-admin-password -o go-template='{{range $k,$v := .data}}{{printf "%s: " $k}}{{if not $v}}{{$v}}{{else}}{{$v | base64decode}}{{end}}{{"\n"}}{{end}}'

Login with admin username and decoded password:

The new AWX interface is beautifully designed with Love. It is a serious redesign with good polishing.

Step 7: Using LoadBalancer Service (Optional)

We can create service configuration for load balancer service. Ensure AWX service is running and you have MetalLB installed and configured properly in your cluster.

$ vim awx-lb-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: awx-lb-service
  namespace: awx
  labels:
    app.kubernetes.io/component: awx
    app.kubernetes.io/name: awx
    app.kubernetes.io/part-of: awx
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8052
  selector:
    app.kubernetes.io/component: awx
    app.kubernetes.io/name: awx
  sessionAffinity: None
  type: LoadBalancer

Create service using defined manifest file.

$ kubectl apply -f awx-lb-service.yaml
service/awx-lb-service created

We can confirm it works by listing service called awx-lb-service in the namespace.

$  kubectl get svc awx-lb-service
NAME             TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)        AGE
awx-lb-service   LoadBalancer   10.233.2.203   192.168.1.32   80:30791/TCP   5m9s

Test connectivity to AWX web interface using Load Balancer IP address.

Step 8: Configure Ingress route

You can use an Ingress controller such as Nginx to expose AWX service, see guide in the link given below:

• How To Expose Ansible AWX Service using Nginx Ingress

Books For Learning Kubernetes Administration;

• Best Kubernetes Study books

Conclusion

To this end we’ve completed the installation, configuration and basic usage of Ansible AWX on Kubernetes. In summary, AWX provides a graphical interface for managing Ansible playbooks, while Ansible Operator is a way to automate the deployment and management of those playbooks using Kubernetes. In the article to follow we shall discuss in detail the use of Ingress to access AWX instance deployed.