Install Ansible AWX on CentOS 8 / Rocky Linux 8

As a system administrator, chances are that there are some tasks that you do repetitively every single day that takes up a lot of your time that could have been invested elsewhere. What if you could solve problems once and then automate your solutions going forward? That is what Ansible is good at and once you get to see its benefits, you will never look back again. Ansible is a simple, yet powerful IT automation engine that thousands of companies are using to drive complexity out of their environments and accelerate DevOps initiatives. In this article we will perform an installation of Ansible AWX on CentOS 8 / Rocky Linux 8 Server.

Be it the deployment of applications, routine maintenance of your servers, Configuration Management, Continuous Delivery, Orchestration or any repetitive work that you can describe, Ansible can handle it for you. To add beauty to this beast, AWX (Ansible Web eXecutable) provides a web-based user interface, REST API, and task engine built on top of Ansible. The AWX Project is an open source community project, sponsored by Red Hat, that enables users to better control their use of Ansible project in IT environments. AWX is the upstream project from which the Red Hat Ansible Tower offering is ultimately derived.

“Don’t wait. The time will never be just right.”
–Napoleon Hill

Setup minimum requirements

Before you can run a deployment, you’ll need the following installed in your local environment

• Kubernetes Cluster / Docker
• User with sudo access
• CentOS 8 / Rocky Linux 8 server
• At least 8GB of RAM – More is better if available
• 4vcpus – Minimum CPU but add more if available
• 25GB minimum disk space

We should be now ready to roll up our sleeves, put on our boots and get to work.

Step 1: Update your system

The first step is performing a system update.

sudo dnf -y update

Disable Firewalld. This is recommended by K3s.

sudo systemctl disable firewalld --now

Once the update is successful perform a system reboot

sudo reboot

Step 2: Install K3s Kubernetes Distribution

AWX is supported and can only be run as a containerized application using Docker images deployed to either an OpenShift cluster, a Kubernetes cluster, or docker-compose. We shall use K3s Kubernetes setup to run AWX on CentOS 8 / Rocky Linux 8.

Put SELinux in permissive mode:

sudo setenforce 0
sudo sed -i 's/^SELINUX=.*/SELINUX=permissive/g' /etc/selinux/config
cat /etc/selinux/config | grep SELINUX=

Install k3s by running the commands below:

curl -sfL https://get.k3s.io | sudo bash -
sudo chmod 644 /etc/rancher/k3s/k3s.yaml

Check k3s service to confirm it is running and working:

$ systemctl status k3s.service
● k3s.service - Lightweight Kubernetes
   Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2021-07-04 22:28:01 UTC; 10s ago
     Docs: https://k3s.io
  Process: 6273 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
  Process: 6265 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
  Process: 6262 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
 Main PID: 6277 (k3s-server)
    Tasks: 30
   Memory: 668.9M
   CGroup: /system.slice/k3s.service
           ├─6277 /usr/local/bin/k3s server
           └─6299 containerd
....

As root user do a validation on use of kubectl Kubernetes management tool:

$ sudo su -
# kubectl get nodes
NAME     STATUS   ROLES                  AGE   VERSION
centos.hirebestengineers.com   Ready    control-plane,master   111s   v1.23.6+k3s1

You can also confirm Kubernetes version deployed using the following command:

# kubectl version --short
Client Version: v1.23.6+k3s1
Server Version: v1.23.6+k3s1

The K3s service will be configured to automatically restart after node reboots or if the process crashes or is killed.

Step 3: Deploy AWX Operator on Kubernetes

This Kubernetes Operator has to be deployed for the management of one or more AWX instances in any namespace.

Install git and make tools:

sudo yum -y install git make

Clone operator deployment code:

# git clone https://github.com/ansible/awx-operator.git
Cloning into 'awx-operator'...
remote: Enumerating objects: 5626, done.
remote: Counting objects: 100% (2840/2840), done.
remote: Compressing objects: 100% (1025/1025), done.
remote: Total 5626 (delta 1910), reused 2434 (delta 1716), pack-reused 2786
Receiving objects: 100% (5626/5626), 1.38 MiB | 12.01 MiB/s, done.
Resolving deltas: 100% (3191/3191), done.

Create namespace where operator will be deployed. I’ll name mine awx:

export NAMESPACE=awx
kubectl create ns ${NAMESPACE}

Set current context to value set in NAMESPACE variable:

# kubectl config set-context --current --namespace=$NAMESPACE 
Context "default" modified.

Switch to awx-operator directory:

cd awx-operator/

Save the latest version from AWX Operator releases as RELEASE_TAG variable then checkout to the branch using git.

sudo yum -y install jq
RELEASE_TAG=`curl -s https://api.github.com/repos/ansible/awx-operator/releases/latest | grep tag_name | cut -d '"' -f 4`
echo $RELEASE_TAG

Deploy AWX Operator into your cluster:

git checkout $RELEASE_TAG
export NAMESPACE=awx
make deploy

Command execution terminal output:

namespace/awx configured
customresourcedefinition.apiextensions.k8s.io/awxbackups.awx.ansible.com created
customresourcedefinition.apiextensions.k8s.io/awxrestores.awx.ansible.com created
customresourcedefinition.apiextensions.k8s.io/awxs.awx.ansible.com created
serviceaccount/awx-operator-controller-manager created
role.rbac.authorization.k8s.io/awx-operator-awx-manager-role created
role.rbac.authorization.k8s.io/awx-operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/awx-operator-metrics-reader created
clusterrole.rbac.authorization.k8s.io/awx-operator-proxy-role created
rolebinding.rbac.authorization.k8s.io/awx-operator-awx-manager-rolebinding created
rolebinding.rbac.authorization.k8s.io/awx-operator-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/awx-operator-proxy-rolebinding created
configmap/awx-operator-awx-manager-config created
service/awx-operator-controller-manager-metrics-service created
deployment.apps/awx-operator-controller-manager created

Wait a few minutes and awx-operator should be running:

# kubectl get pods -n awx
NAME                                               READY   STATUS    RESTARTS   AGE
awx-operator-controller-manager-68d787cfbd-z75n4   2/2     Running   0          48s

Uninstalling AWX Operator – Don’t run, only for reference🙂

You can always remove the operator and all associated CRDs by running the command below:

# export NAMESPACE=awx
# make undeploy
/root/awx-operator/bin/kustomize build config/default | kubectl delete -f -
namespace "awx" deleted
customresourcedefinition.apiextensions.k8s.io "awxbackups.awx.ansible.com" deleted
customresourcedefinition.apiextensions.k8s.io "awxrestores.awx.ansible.com" deleted
customresourcedefinition.apiextensions.k8s.io "awxs.awx.ansible.com" deleted
serviceaccount "awx-operator-controller-manager" deleted
role.rbac.authorization.k8s.io "awx-operator-leader-election-role" deleted
role.rbac.authorization.k8s.io "awx-operator-manager-role" deleted
clusterrole.rbac.authorization.k8s.io "awx-operator-metrics-reader" deleted
clusterrole.rbac.authorization.k8s.io "awx-operator-proxy-role" deleted
rolebinding.rbac.authorization.k8s.io "awx-operator-leader-election-rolebinding" deleted
rolebinding.rbac.authorization.k8s.io "awx-operator-manager-rolebinding" deleted
clusterrolebinding.rbac.authorization.k8s.io "awx-operator-proxy-rolebinding" deleted
configmap "awx-operator-manager-config" deleted
service "awx-operator-controller-manager-metrics-service" deleted
deployment.apps "awx-operator-controller-manager" deleted

Step 4: Install Ansible AWX on CentOS 8 / Rocky Linux 8

Now that we have the operator pod running we are ready to initiate installation of Ansible AWX on CentOS 8 / Rocky Linux 8. But first we’ll need to create a PVC for public and static web data.

Create a file named public-static-pvc.yaml:

vi public-static-pvc.yaml

Input below contents in the file:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: public-static-data-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: local-path
  resources:
    requests:
      storage: 5Gi

Apply configuration manifest:

# kubectl apply -f public-static-pvc.yaml -n awx
persistentvolumeclaim/public-static-data-pvc created

PVC won’t be bound until the pod that uses it is created.

# kubectl get pvc -n awx
NAME                     STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
public-static-data-pvc   Pending                                      local-path     43s

Create AWX deployment file:

vi awx-instance-deployment.yml

Paste below contents to the file created.

---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
spec:
  service_type: nodeport
  projects_persistence: true
  projects_storage_access_mode: ReadWriteOnce
  web_extra_volume_mounts: |
    - name: static-data
      mountPath: /var/lib/projects
  extra_volumes: |
    - name: static-data
      persistentVolumeClaim:
        claimName: public-static-data-pvc

Install AWX on CentOS 8 / Rocky Linux 8:

# kubectl apply -f awx-instance-deployment.yml -n awx
awx.awx.ansible.com/awx created

After few minutes check pods creation status:

# watch kubectl get pods -l "app.kubernetes.io/managed-by=awx-operator" -n awx
NAME                   READY   STATUS    RESTARTS   AGE
awx-postgres-0         1/1     Running   0          2m58s
awx-75698588d6-qz2gf   4/4     Running   0          2m42s

 You can track the installation process at the operator pod logs:

kubectl logs -f deployments/awx-operator-controller-manager -c awx-manager

Extra PVCs are created automatically:

# kubectl  get pvc
NAME                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
postgres-awx-postgres-0   Bound    pvc-34a25045-486c-42a8-9763-d14a7bb3e9e8   8Gi        RWO            local-path     72s
public-static-data-pvc    Bound    pvc-3484e513-8d00-482c-bdce-6e77820f237e   1Gi        RWO            local-path     5m13s
awx-projects-claim        Bound    pvc-e56ab471-97f1-455b-9d51-ba05b7d9982b   8Gi        RWO            local-path     60s

Fixing the error “mkdir: cannot create directory ‘/var/lib/postgresql/data’: Permission denied”

If you see the error message from postgres pod logs:

# kubectl logs awx-postgres-0
mkdir: cannot create directory ‘/var/lib/postgresql/data’: Permission denied

It means the Postgres pod cannot write to the persistent volume directory inside/var/lib/rancher/k3s/storage/:

# ls -lh /var/lib/rancher/k3s/storage/ | grep awx-postgres-0
total 0
drwx------. 3 root root 18 Aug  3 14:04 pvc-8110b494-d9ed-450a-94c0-b9dfd2bd73f7_default_postgres-awx-postgres-0

Try setting the directory mode to 777

# chmod -R 777  /var/lib/rancher/k3s/storage/*
# kubectl delete pods -l "app.kubernetes.io/managed-by=awx-operator" -n awx
pod "awx-75698588d6-x79g2" deleted
pod "awx-postgres-0" deleted

The Postgres container pod should come up in few seconds:

# kubectl get pods -n awx
NAME                            READY   STATUS    RESTARTS   AGE
awx-operator-545497f7d5-bqlcs   1/1     Running   0          65m
awx-postgres-0                  1/1     Running   4          8m22s
awx-75698588d6-7kg9j            4/4     Running   0          8m10s

Checking AWX Container’s logs

The awx-xxx-yyy pod will have four containers, namely:

• redis
• awx-web
• awx-task
• awx-ee

As can be seen from below command output:

# kubectl get deploy
NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
awx-operator-controller-manager   1/1     1            1           9m58s
awx                               1/1     1            1           7m47s

# kubectl -n awx  logs deploy/awx
error: a container name must be specified for pod awx-66596c8fcb-s28tw, choose one of: [redis awx-web awx-task awx-ee] or one of the init containers: [database-check init]

You’ll need to provide container name after the pod:

kubectl -n awx  logs deploy/awx -c redis
kubectl -n awx  logs deploy/awx -c awx-web
kubectl -n awx  logs deploy/awx -c awx-task
kubectl -n awx  logs deploy/awx -c awx-ee

Access AWX Container’s Shell

Here is how to access each container’s shell:

kubectl exec -ti deploy/awx  -c  awx-task -- /bin/bash
kubectl exec -ti deploy/awx  -c  awx-web -- /bin/bash
kubectl exec -ti deploy/awx  -c  awx-ee -- /bin/bash
kubectl exec -ti deploy/awx  -c  redis -- /bin/bash

Upgrading AWX Instance

For upgrade process refer to our guide in the link below:

• How to upgrade Ansible AWX running in Kubernetes

Step 5: Access AWX Web Interface

Get the AWX Web service port:

# kubectl get service -n awx
NAME           TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
awx-postgres   ClusterIP   None           <none>        5432/TCP       5m17s
awx-service    NodePort    10.43.54.203   <none>        80:30080/TCP   5m7s

From the output we can confirm service node port is 30080.

To have access to AWX web console, point your browser to your Ansible’s AWX server IP:

http://your-server-ip-address:30080

You should be welcomed to a Login page well illustrated below.

The login username is admin

Obtain admin user password by decoding the secret with the password value:

kubectl -n awx get secret awx-admin-password -o jsonpath="{.data.password}" | base64 --decode

Better output format:

kubectl -n awx get secret awx-admin-password -o go-template='{{range $k,$v := .data}}{{printf "%s: " $k}}{{if not $v}}{{$v}}{{else}}{{$v | base64decode}}{{end}}{{"\n"}}{{end}}'

Login with the admin username and decoded password from above commands:

Configure Ingress for AWX

If you would like to access your AWX using domain names and SSL, check out our ingress articles:

• How To Expose Ansible AWX Service using Traefik Ingress route
• How To Expose Ansible AWX Service using Nginx Ingress

Once the authentication is successful, you’ll get to AWX administration dashboard. Therein, there is a lot of stuff to do and we shall cover just a few. Along the left side of the Dashboard is the navigation menu, where you can quickly navigate to your Projects, Inventories, Job Templates, and Jobs

Step 6: Create User and Team

AWX provides a default user called admin that you can use to do your tests. But if you are need a different user, you can create another one.

To add a new user apart from the admin one you find by default, click on Users tab found on the left menu. A new page will be displayed where you can add a new user. Click on the green + icon then fill in the details of the new user to be added.

After you are done, click “Save“.

The same applies for creating a new Team. Click on “Teams” tab found on the left menu. A new page will be displayed where you can add a new team. Click on the green + icon then fill in the details of the new team to be added.

After you are done, click “Save“.

Step 7: Create an Organization

An organization is a logical collection of users, teams, projects, and inventories. It is the highest level object in the AWX object hierarchy. From the left navigation bar, click the Organizations icon. You will find that a default organization has been automatically created and is available to all users of Ansible AWX.

It can be used as is or edited as needed. Let us edit it by changing its name and adding users, permissions and more.

Click on the “Users” tab and add a new user. I had not created a new user before, so admin default user appears. But you can create other users as shared in Step 5.

If you had other users, they would be populated therein

Clicking on Users displays all the Users associated with this Organization. A User is someone with access to Ansible AWX with associated roles and Credentials. Adding a user to an organization adds them as a member only, specifying a role for the user can be done in the the Permissions tab, as shown in the example below:

Step 8: Create credentials

In order for Ansible to log in and execute tasks, it will need credentials to access the remote hosts. AWX provides this feature for us. Click on “Credentials” on the left menu and then click the green + to add a new server credential.

Fill in the name, description, organization, username, password and the type of credential which is a long list. We shall choose machine since we intend to configure a single host in this example. You can add your ssh keys in case you prefer key-based authentication mechanism. Also add privilege escalation below to sudo.

Click “Save” once done.

Step 9: Create a new Inventory and add it to the Organization

An inventory is a collection of hosts managed by Ansible. Inventories are assigned to organizations, while permissions to launch playbooks against inventories are controlled at the user and/or team level.

To create and review existing inventories click the Inventories icon from the left navigation bar. To create a new inventory, click the add (+) button and select Inventory from the drop-down menu list.

Enter the name and Organization that this Inventory will belong to. Click “Save” so that the other tabs can be activated.

For this example, we are going to add one host but know that you can create a group that contains one or more hosts you would wish to execute something on them simultaneously. Click on “Hosts” tab and click on + to add a new host.

On the new page loaded, add an IP or resolvable hostname and a description. We are going to install nginx on the given host in this example thus the intuitive name.

Click “Save” once done and the new Inventory should be created.

The concept of Groups and Hosts

Note that inventories are divided into groups and hosts. A group might represent a particular environment (e.g. “Datacenter 1” or “Testing Bed”), a server type (e.g. “Web Servers” or “DB Servers”), or any other representation of your environment.

Step 10: Setting up a Project

A Project is a logical collection of Ansible playbooks, represented in Ansible AWX. You can manage playbooks and playbook directories by either placing them manually under the Project Base Path on your Ansible AWX server, or by placing your playbooks into a source code management (SCM) system supported by Ansible AWX, including Git, Subversion, and Mercurial.

To create a new project, follow the same procedure as the rest we have seen thus far. Click on the “Projects” tab on the left menu and then click on add(+) to create a new project.

Once there, fill the details to suit your needs. On the Source Control Manager (SCM) type, you can create a git repo that has all of your playbooks or create a local folder on your server as advised when you choose manual under SCM type. I preferred adding a git repo in this example. Note that we are still referring to the Organization of your choice in case you have several.

Enter all the details then click “Save“. The file created on the git repo (nginx.yml) has the following:

---
- hosts: all
  gather_facts: true
  become: true
  become_user: root
  tasks:
    - apt:
        name: nginx
      when: ansible_os_family == "Debian"

Attached screenshot below:

Step 11: Setting up a template and launching it

Thus far, we have done a lot and what remains is putting it all together into a template. A job template combines an Ansible playbook from a project and the settings required to launch it. Create a new job template by clicking on “Templates” tab on the left navigation menu. Click on he green add (+) to add a new template just like the others. When the window opens, it is just a matter of picking the Inventory, Project, Credential and Playbook we had already configured in the previous steps. Give your template a name and an optional description as well.

When done scroll down and hit “Save“. Once you save, you can “Launch” it as a job immediately.

Once you hit Launch you will be redirected to the “Jobs” page where you will see it executing live with output being displayed as it happens.

If everything goes well, the playbook should be run successfully by Ansible behind the scenes. All errors encountered along the way will be displayed on the right side.

Login to the server being tested on and check if nginx was installed successfully

Amazing stuff, Ansible AWX installation is officially setup and running properly.

More about Ansible Tower | AWX can be found on RedHat’s Ansible Tower Quick Start Guide.

To Conclude

Ansible can change the way you handle your day to day administration tasks and it can offload a lot of manual work from your hands and gift you with ample time for other tasks. Leverage this technology as your innovation engine, and deliver your applications faster and win big.

Other electrifying guides you might enjoy include:

• Manage Users and Groups on Linux using Ansible
• How To Generate Linux User Encrypted Password for Ansible
• Install Apache Tomcat 9 on Debian 10 With Ansible
• How To Install Ansible AWX on Debian 10 (Buster)
• Set up Ansible AWX on Ubuntu Linux
• Deploy Kubernetes Cluster with Ansible & Kubespray