Prometheus is an open-source monitoring system with a dimensional data model, flexible query language, efficient time series database and modern alerting approach. Prometheus can be installed as standalone service in a Linux machine or deployed in a Kubernetes cluster. In this guide we will walk you through the installation of Prometheus on an EKS Cluster deployed in AWS Cloud.
The main features of Prometheus are:
- a multi-dimensional data model with time series data identified by metric name and key/value pairs
- PromQL, a flexible query language to leverage this dimensionality
- no reliance on distributed storage; single server nodes are autonomous
- time series collection happens via a pull model over HTTP
- pushing time series is supported via an intermediary gateway
- targets are discovered via service discovery or static configuration
- multiple modes of graphing and dashboarding support
Deploying Prometheus on EKS Kubernetes Cluster
You should have a running EKS Cluster before you can use this guide with kubectl installed and able to interact with the API server.
List available clusters.
$ eksctl get cluster
NAME REGION
prod-eks-cluster eu-west-1Step 1: Configure kubectl
To obtain cluster credentials at any point in time from an EKS cluster deployed with eksctl, run:
eksctl utils write-kubeconfig --cluster=<name> [--kubeconfig=<path>][--set-kubeconfig-context=<bool>]See below example:
# Create default kubeconfig directory - Can be custom
mkdir $HOME/.kube
# Obtain cluster credentials
eksctl utils write-kubeconfig --cluster=prod-eks-cluster --kubeconfig=$HOME/.kube/config
Expected command execution output:
[ℹ] eksctl version 0.5.9
[ℹ] using region eu-west-1
[✔] saved kubeconfig as "/Users/jkmutai/.kube/config"Confirm if kubectl command is working as expected.
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-192-168-138-244.eu-west-1.compute.internal Ready <none> 7d21h v1.26.9-eks-4c6976
ip-192-168-176-247.eu-west-1.compute.internal Ready <none> 7d21h v1.26.9-eks-4c6976Step 2: Install Kubernetes Metrics Server
The Kubernetes API server exposes a number of metrics that are useful for monitoring and analysis. These metrics are exposed internally through a metrics endpoint that refers to the /metrics HTTP API.
Install Metrics Server with the following guide:
You can view cluster raw metrics by running the following command:
kubectl get --raw /metricsStep 3: Install Helm
The Helm package manager for Kubernetes helps you install and manage applications on your Kubernetes cluster. We’ll be using help to deploy Prometheus on EKS.
These are the commands you’ll use to install helm.
### Linux ###
curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 > get_helm.sh
chmod 700 get_helm.sh
./get_helm.sh
### macOS ###
brew install helm
### Windows ###
choco install kubernetes-helm
You can query installed version using the command:
$ helm version
version.BuildInfo{Version:"v3.10.2", GitCommit:"50f003e5ee8704ec937a756c646870227d7c8b58", GitTreeState:"clean", GoVersion:"go1.19.3"}Step 4: Deploying Prometheus on EKS Kubernetes Cluster
Prometheus can be installed on Kubernetes cluster using Operator or with helm. We’ll use Helm V3 we just installed to deploy prometheus in an EKS cluster.
First create a prometheus namespace.
$ kubectl create namespace prometheus
namespace/prometheus createdPrometheus needs a way to persist metrics data for historical reference. We’ll use EBS which is provisioned with gp2 storage class.
$ kubectl get sc
kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
efs-sc efs.csi.aws.com Delete Immediate false 2d23h
gp2 (default) kubernetes.io/aws-ebs Delete WaitForFirstConsumer false 7d21h
Add chart repository:
$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
"prometheus-community" has been added to your repositoriesUpdate Helm repos
$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "metrics-server" chart repository
...Successfully got an update from the "traefik" chart repository
...Successfully got an update from the "harbor" chart repository
...Successfully got an update from the "nfs-subdir-external-provisioner" chart repository
...Successfully got an update from the "kubevious" chart repository
...Successfully got an update from the "prometheus-community" chart repository
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈Happy Helming!⎈
Deploy Prometheus using Helm. Syntax is:
helm install [RELEASE_NAME] prometheus-community/kube-prometheus-stack -n prometheusSee example below.
$ helm install prometheus prometheus-community/kube-prometheus-stack -n prometheus
NAME: prometheus
LAST DEPLOYED: Thu Aug 24 17:02:08 2023
NAMESPACE: prometheus
STATUS: deployed
REVISION: 1
NOTES:
kube-prometheus-stack has been installed. Check its status by running:
kubectl --namespace prometheus get pods -l "release=prometheus"
Visit https://github.com/prometheus-operator/kube-prometheus for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator.Step 4: Access Prometheus on EKS Kubernetes Cluster
After installation query all resources in the monitoring namespace:
$ kubectl get all -n prometheus
kubectl get all -n prometheus
NAME READY STATUS RESTARTS AGE
pod/alertmanager-prometheus-kube-prometheus-alertmanager-0 2/2 Running 0 3s
pod/prometheus-grafana-5c7b4d5d87-hh24q 3/3 Running 0 38m
pod/prometheus-kube-prometheus-operator-847c75d56b-97tlx 1/1 Running 0 38m
pod/prometheus-kube-state-metrics-6df4697c45-zxfkh 1/1 Running 0 38m
pod/prometheus-prometheus-kube-prometheus-prometheus-0 1/2 Running 0 3s
pod/prometheus-prometheus-node-exporter-4xvtl 1/1 Running 0 38m
pod/prometheus-prometheus-node-exporter-5787b 1/1 Running 0 38m
pod/prometheus-prometheus-node-exporter-9lq8s 1/1 Running 0 38m
pod/prometheus-prometheus-node-exporter-c8wfz 1/1 Running 0 38m
pod/prometheus-prometheus-node-exporter-dvc82 1/1 Running 0 38m
pod/prometheus-prometheus-node-exporter-ffp69 1/1 Running 0 38m
pod/prometheus-prometheus-node-exporter-k5mjn 1/1 Running 0 38m
pod/prometheus-prometheus-node-exporter-nbgrx 1/1 Running 0 38m
pod/prometheus-prometheus-node-exporter-pbmc5 1/1 Running 0 38m
pod/prometheus-prometheus-node-exporter-t2mt6 1/1 Running 0 38m
pod/prometheus-prometheus-node-exporter-zxslb 1/1 Running 0 38m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 38m
service/prometheus-grafana ClusterIP 10.233.62.161 <none> 80/TCP 38m
service/prometheus-kube-prometheus-alertmanager ClusterIP 10.233.31.222 <none> 9093/TCP,8080/TCP 38m
service/prometheus-kube-prometheus-operator ClusterIP 10.233.40.217 <none> 443/TCP 38m
service/prometheus-kube-prometheus-prometheus ClusterIP 10.233.15.135 <none> 9090/TCP,8080/TCP 38m
service/prometheus-kube-state-metrics ClusterIP 10.233.54.189 <none> 8080/TCP 38m
service/prometheus-operated ClusterIP None <none> 9090/TCP 38m
service/prometheus-prometheus-node-exporter ClusterIP 10.233.13.184 <none> 9100/TCP 38m
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/prometheus-prometheus-node-exporter 11 11 0 11 0 kubernetes.io/os=linux 38m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/prometheus-grafana 1/1 1 1 38m
deployment.apps/prometheus-kube-prometheus-operator 1/1 1 1 38m
deployment.apps/prometheus-kube-state-metrics 1/1 1 1 38m
NAME DESIRED CURRENT READY AGE
replicaset.apps/prometheus-grafana-5c7b4d5d87 1 1 1 38m
replicaset.apps/prometheus-kube-prometheus-operator-847c75d56b 1 1 1 38m
replicaset.apps/prometheus-kube-state-metrics-6df4697c45 1 1 1 38m
NAME READY AGE
statefulset.apps/alertmanager-prometheus-kube-prometheus-alertmanager 1/1 38m
statefulset.apps/prometheus-prometheus-kube-prometheus-prometheus 1/1 38mGet the Prometheus server URL by running these commands in the same shell:
$ kubectl get service -n prometheus
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 26m
prometheus-grafana ClusterIP 10.233.62.161 <none> 80/TCP 26m
prometheus-kube-prometheus-alertmanager ClusterIP 10.233.31.222 <none> 9093/TCP,8080/TCP 26m
prometheus-kube-prometheus-operator ClusterIP 10.233.40.217 <none> 443/TCP 26m
prometheus-kube-prometheus-prometheus ClusterIP 10.233.15.135 <none> 9090/TCP,8080/TCP 26m
prometheus-kube-state-metrics ClusterIP 10.233.54.189 <none> 8080/TCP 26m
prometheus-operated ClusterIP None <none> 9090/TCP 26m
prometheus-prometheus-node-exporter ClusterIP 10.233.13.184 <none> 9100/TCP 26mUsing Kubernetes port forwarding feature to access Grafana service.
kubectl port-forward svc/prometheus-grafana 3000:80 -n prometheusUsing Kubernetes port forwarding feature to access Prometheus Alertmanager.
kubectl port-forward svc/prometheus-kube-prometheus-alertmanager 9093:9093 -n prometheusAccess grafana on 127.0.0.1:3000 from your local machine’s browser.
Default Logins:
- Username: admin
- Password: prom-operator
You can confirm by listing secrets in the prometheus namespace.
$ kubectl get secrets -n prometheus
NAME TYPE DATA AGE
alertmanager-prometheus-kube-prometheus-alertmanager Opaque 1 33m
alertmanager-prometheus-kube-prometheus-alertmanager-generated Opaque 1 33m
alertmanager-prometheus-kube-prometheus-alertmanager-tls-assets-0 Opaque 0 33m
alertmanager-prometheus-kube-prometheus-alertmanager-web-config Opaque 1 33m
prometheus-grafana Opaque 3 33m
prometheus-kube-prometheus-admission Opaque 3 33m
prometheus-prometheus-kube-prometheus-prometheus Opaque 1 33m
prometheus-prometheus-kube-prometheus-prometheus-tls-assets-0 Opaque 1 33m
prometheus-prometheus-kube-prometheus-prometheus-web-config Opaque 1 33m
sh.helm.release.v1.prometheus.v1 helm.sh/release.v1 1 33mDecode the secret to get username and password.
export secret=prometheus-grafana
kubectl -n prometheus get secret $secret -o go-template='{{range $k,$v := .data}}{{printf "%s: " $k}}{{if not $v}}{{$v}}{{else}}{{$v | base64decode}}{{end}}{{"\n"}}{{end}}'Output:
admin-password: prom-operator
admin-user: adminPrometheus Alert Manager UI.
To unistall the chart use:
helm uninstall [RELEASE_NAME]Next place to be is Prometheus documentation, specifically QUERYING PROMETHEUS
More guides:
