How To Install Apache Skywalking APM Tool on Linux

Monitoring is one of the vital tasks for any system admin. With increasing technology, many organizations rely on applications to run their businesses. It is therefore essential to check the server’s and applications’ health frequently. By doing so, you are able to ensure that the applications and systems are working as desired. There are several tools that one can use to monitor environments. The most popular solutions are Prometheus, Dynatrace, Elastic Stack, Zenoss, New Relic, N-able RMM, Zabbix, Datadog Infrastructure Monitoring, Nagios, PRTG Network Monitor e.t.c

SkyWalking is an open-source APM(application performance monitor) system that provides monitoring, tracing, and diagnosing capabilities for microservices, Cloud Native, and container-based environments.

The Apache SkyWalking offers more other features and capabilities that include:

• Root cause analysis: with the profile code on the runtime, powered by an in-process agent and the ebpf profiler. It also has a network profiler for TCP, TCP/TLS, and HTTP(s) traffic.
• Service, service instance, endpoint(URI) metrics analysis with:
  • Metrics collecting, analysis, and customization.
  • Distributed tracing and context propagation
  • Collaboration across metrics, traces, and logs
• Service topology map analysis.
• Performance optimization.
• Service instance and endpoint(URI) dependency analysis.
• Slow services and endpoints detection.
• Message Queue performance and consuming latency monitoring.
• Database performance monitoring. Detect slow SQL statements.
• Browser performance monitoring, and start tracing from the browser.
• Infrastructure monitoring. Kubernetes and Linux(VM, network, disk e.t.c.).
• It also offers alerting

SkyWalking can be used to collect metrics, traces, logs e.t.c across multiple sources and formats. The supported ones include:

• Go, C++, and Rust SDKs.
• Java, NodeJS, .NET Core, PHP, and Python auto-instrument agents.
• Agent profiling for Java and Python.
• Browser agent.
• LUA agent especially for Nginx, OpenResty, and Apache APISIX.
• ebpf network profiling and ON/OFF CPU profiling.
• The metrics system includes Prometheus, OpenTelemetry, Micrometer(Spring Sleuth), and Zabbix.
• Service Mesh Observability. The control plane and data plane.
• Logs, including OpenTelemetry.

Below is an illustration of the Apache Skywalking architecture.

In this guide, we will systematically walk through how to install and use Apache Skywalking Application Performance Monitoring on Linux

Step 1 – Setup Pre-requisites

This guide best works with the following:

• Linux server with JDK 1.8.0 or later installed
• Elasticsearch cluster created, though, it is possible to use any other storage such as H2, OpenSearch, MySQL, TiDB, InfluxDB, PostgreSQL

To setup Elasticsearch, you can use any of the guides below:

• Install Elastic Stack 8 (ELK 8) on RHEL 8|CentOS 8
• How To Install ELK Stack on CentOS / Fedora
• Run Elastic stack (ELK) on Docker Containers with Docker Compose

Once Elasticsearch elastic search is up, verify if it is working:

$ curl -X GET localhost:9200
{
  "name" : "localhost.localdomain",
  "cluster_name" : "my-application",
  "cluster_uuid" : "cIYbRiEwTMyVOpYuyn4dnw",
  "version" : {
    "number" : "8.4.3",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "42f05b9372a9a4a470db3b52817899b99a76ee73",
    "build_date" : "2022-10-04T07:17:24.662462378Z",
    "build_snapshot" : false,
    "lucene_version" : "9.3.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

Also ensure JDK has been installed:

##On Debian/Ubuntu
sudo apt install default-jdk -y

##On RHEL/Rocky_Linux/Alma/CentOS
sudo dnf install java-11-openjdk-devel -y

verify the installation:

$ java --version
openjdk 11.0.16 2022-07-19
OpenJDK Runtime Environment (build 11.0.16+8-post-Ubuntu-0ubuntu120.04)
OpenJDK 64-Bit Server VM (build 11.0.16+8-post-Ubuntu-0ubuntu120.04, mixed mode, sharing)

Step 2 – Install and Configure Apache Skywalking APM

You can download the Apache Skywalking from the official Apache Skywalking downloads page.

You can also download the latest version as shown. First, export the latest version:

VER=$(curl -s https://api.github.com/repos/apache/skywalking/releases/latest|grep tag_name|cut -d '"' -f 4|sed 's/v//')

Then download the version with the command:

wget https://archive.apache.org/dist/skywalking/$VER/apache-skywalking-apm-$VER.tar.gz

Once downloaded, extract the file:

tar -xvf apache-skywalking-apm-*.tar.gz

Move the file to the /opt directory;

sudo mv apache-skywalking-apm-bin /opt/apache-skywalking-apm-bin

Set the correct permissions:

sudo chmod -R 775 /opt/apache-skywalking-apm-bin

Navigate into the directory;

cd /opt/apache-skywalking-apm-bin

Once in the directory, you are required to make a few configurations.

sudo vim config/application.yml

In the file, locate the storage and set it from H2 to elasticsearch as shown:

storage:
  selector: ${SW_STORAGE:elasticsearch}
  elasticsearch:
    namespace: ${SW_NAMESPACE:"my-application"}
    clusterNodes: ${SW_STORAGE_ES_CLUSTER_NODES:localhost:9200}
    protocol: ${SW_STORAGE_ES_HTTP_PROTOCOL:"http"}
    connectTimeout: ${SW_STORAGE_ES_CONNECT_TIMEOUT:3000}
    socketTimeout: ${SW_STORAGE_ES_SOCKET_TIMEOUT:30000}
    responseTimeout: ${SW_STORAGE_ES_RESPONSE_TIMEOUT:15000}
    numHttpClientThread: ${SW_STORAGE_ES_NUM_HTTP_CLIENT_THREAD:0}
    user: ${SW_ES_USER:"elastic"}
    password: ${SW_ES_PASSWORD:"changeme"}
 #   trustStorePath: ${SW_STORAGE_ES_SSL_JKS_PATH:""}
 #   trustStorePass: ${SW_STORAGE_ES_SSL_JKS_PASS:""}
#    secretsManagementFile: ${SW_ES_SECRETS_MANAGEMENT_FILE:""} # Secrets management file in the properties format includes the username, password, which are managed by 3rd party tool.
....

In the above file, we have changed the storage form H2 to elasticsearch since H2 does not support persistent data storage. The parameters above are described below:

• selector: The storage selector. For this example, set the value to elasticsearch
• nameSpace: represents the prefix for the names of all indexes in the Elasticsearch cluster.
• clusterNodes: The endpoint of the Elasticsearch cluster
• user: The username that is used to access the Elasticsearch cluster. The default username is elastic.
• password: The password that is used to access the Elasticsearch cluster. It is usually specified when you create the Elasticsearch cluster. The default setting is changeme

Stil in the config, you can optionally change the default port that SkyWalking uses to communicate with Elasticsearch(12800 for RESTful API operations and over port 11800 for gRPC API operations)

core:
  selector: ${SW_CORE:default}
  default:
    # Mixed: Receive agent data, Level 1 aggregate, Level 2 aggregate
    # Receiver: Receive agent data, Level 1 aggregate
    # Aggregator: Level 2 aggregate
    role: ${SW_CORE_ROLE:Mixed} # Mixed/Receiver/Aggregator
    restHost: ${SW_CORE_REST_HOST:0.0.0.0}
    restPort: ${SW_CORE_REST_PORT:12800}
    restContextPath: ${SW_CORE_REST_CONTEXT_PATH:/}
    gRPCHost: ${SW_CORE_GRPC_HOST:0.0.0.0}
    gRPCPort: ${SW_CORE_GRPC_PORT:11800}

There is also a configuration you can make in the webapp folder.

sudo vim webapp/webapp.yml 

For this guide, we will go with the default, but still you can modify these values based on your business requirements.

server:
  port: 8080

spring:
  cloud:
    gateway:
      routes:
        - id: oap-route
          uri: lb://oap-service
          predicates:
            - Path=/graphql/**
    discovery:
      client:
        simple:
          instances:
            oap-service:
              - uri: http://127.0.0.1:12800
            # - uri: http://<oap-host-1>:<oap-port1>
            # - uri: http://<oap-host-2>:<oap-port2>

  mvc:
    throw-exception-if-no-handler-found: true

  web:
    resources:
      add-mappings: true

management:
  server:
    base-path: /manage

Step 3 – Create systemd unit for Apache Skywalking

We need to create a systemd service that will be used to stop and start the application. Create the file as shown:

sudo vim /etc/systemd/system/skywalking.service

In the file, add the below lines:

[Unit]
Description=Apache Skywalking
After=network.target

[Service]
Type=oneshot
User=root
WorkingDirectory=/opt/apache-skywalking-apm-bin/
ExecStart=/bin/bash /opt/apache-skywalking-apm-bin/bin/startup.sh
RemainAfterExit=yes
RestartSec=5

[Install]
WantedBy=multi-user.target

Save the file and reload the system daemon.

sudo systemctl daemon-reload

For Rhel-based systems, configure SELinux as shown

sudo /sbin/restorecon -v /opt/apache-skywalking-apm-bin/bin/startup.sh

Start and enable the service:

sudo systemctl enable --now skywalking

Verify if the service is running:

$ systemctl status skywalking
● skywalking.service - Apache Skywalking
   Loaded: loaded (/etc/systemd/system/skywalking.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2022-10-15 05:54:11 EDT; 870ms ago
 Main PID: 351646 (bash)
    Tasks: 3 (limit: 23198)
   Memory: 1.1M
   CGroup: /system.slice/skywalking.service
           ├─351646 /bin/bash /opt/apache-skywalking-apm-bin/bin/startup.sh
           ├─351648 sh /opt/apache-skywalking-apm-bin/bin/oapService.sh
           └─351652 sleep 1

Allow the service through the firewall;

##For UFW
sudo ufw allow 8080

##For Firewalld
sudo firewall-cmd --add-port=8080/tcp --permanent
sudo firewall-cmd --reload

Step 4 – Access the Apache Skywalking WebUI

At this point, you can access the Apache Skywalking WebUI using the URL http://IP_Address:8080

At this point, we are not able to view any metrics since we haven’t performed any configurations. In this guide, I will demonstrate how to collect and export metrics from a Linux host/agent.

Step 5 – Install Prometheus Node Exporter

Now we need to set up an agent to be able to see the metrics. Here we will use the Prometheus node exporter to obtain logs from an agent.

First, download the latest archive:

##For AMD64
VERSION=$(curl -s https://api.github.com/repos/prometheus/node_exporter/releases/latest|grep tag_name|cut -d '"' -f 4|sed 's/v//')
wget https://github.com/prometheus/node_exporter/releases/download/v$VERSION/node_exporter-$VERSION.linux-amd64.tar.gz

##For ARM64
VERSION=$(curl -s https://api.github.com/repos/prometheus/node_exporter/releases/latest|grep tag_name|cut -d '"' -f 4|sed 's/v//')
wget https://github.com/prometheus/node_exporter/releases/download/v$VERSION/node_exporter-$VERSION.linux-arm64.tar.gz

##For i386
VERSION=$(curl -s https://api.github.com/repos/prometheus/node_exporter/releases/latest|grep tag_name|cut -d '"' -f 4|sed 's/v//')
wget https://github.com/prometheus/node_exporter/releases/download/v$VERSION/node_exporter-$VERSION.linux-386.tar.gz

Now extract the archive:

tar -xf node_exporter-$VERSION.linux-*.tar.gz

Move the file to the /opt directory:

sudo mv node_exporter-*/node_exporter /usr/local/bin

Create a dedicated user for the service:

sudo adduser -M -r -s /sbin/nologin prometheus

Create a systemd service file:

$ sudo vim /etc/systemd/system/node_exporter.service
[Unit]
Description=Node Exporter
After=network.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/node_exporter
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

Save the file and reload the daemon.

sudo systemctl daemon-reload

On Rhel-based systems, configure SELinux:

sudo /sbin/restorecon -v /usr/local/bin/node_exporter

Start and enable the service:

sudo systemctl enable --now node_exporter

Verify if the service is running:

$ systemctl status node_exporter
 node_exporter.service - Node Exporter
   Loaded: loaded (/etc/systemd/system/node_exporter.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2022-10-15 06:52:23 EDT; 8s ago
 Main PID: 353801 (node_exporter)
    Tasks: 3 (limit: 23198)
   Memory: 4.8M
   CGroup: /system.slice/node_exporter.service
           └─353801 /usr/local/bin/node_exporter

Check the node-exporter port:

$ sudo ss -aplnt | grep node
LISTEN 0      128                *:9100             *:*    users:(("node_exporter",pid=353801,fd=3))               

Allow this port through the firewall;

##For UFW
sudo ufw allow 9100

##For Firewalld
sudo firewall-cmd --add-port=9100/tcp --permanent
sudo firewall-cmd --reload

Step 6 – Install and Configure the OpenTelemetry Collector

The OpenTelemetry Collector was formed after a fusion of the OpenCensus and OpenTracing in May 2019. Its main goal is to provide a set of standardized vendor-agnostic SDKs, APIs, and tools for ingesting, transforming, and sending data to an Observability back-end. This is not an observability back-end like Prometheus or Jager, it supports data export to a variety of open-source and commercial back-ends.

This tool can be installed on your Linux system as shown.

##On Debian/Ubuntu
sudo apt update && sudo apt install wget curl -y
VERSION=$(curl -s https://api.github.com/repos/open-telemetry/opentelemetry-collector-releases/releases/latest|grep tag_name|cut -d '"' -f 4|sed 's/v//')
wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v${VERSION}/otelcol_${VERSION}_linux_amd64.deb
sudo dpkg -i otelcol_${VERSION}_linux_amd64.deb

##On Rhel/Rocky/Alma/CentOS
sudo yum -y install wget curl
VERSION=$(curl -s https://api.github.com/repos/open-telemetry/opentelemetry-collector-releases/releases/latest|grep tag_name|cut -d '"' -f 4|sed 's/v//')
wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v${VERSION}/otelcol_${VERSION}_linux_amd64.rpm
sudo rpm -Uvh otelcol_${VERSION}_linux_amd64.rpm

Once installed, you need to configure it to export metrics from the Prometheus Node exporter to the Apache Skywalking backend.

Move the available config to a backup file

sudo mv /etc/otelcol/config.yaml /etc/otelcol/config.yaml.bak

Now create a new config:

sudo vim /etc/otelcol/config.yaml 

Add the below lines and replace them where required.

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: "vm-monitoring" # make sure to use this in the vm.yaml to filter only VM metrics
          scrape_interval: 10s
          static_configs:
            - targets: ["192.168.205.11:9100"]

processors:
  batch:

exporters:
  ####################################################################################
  # If you want to use otlp exporter please ensure that your OAP version is >= 9.2.0 #
  # or you can use oc exporter, The config format should be:                         #
  # opencensus:                                                                      #
  #   endpoint: "127.0.0.1:11800"                                                          #
  #   tls:                                                                           #
  #      insecure: true                                                              #
  ####################################################################################
  otlp:
    endpoint: "127.0.0.1:11800" # The OAP Server address
    ###############################################################################
    # OTEL previous version before 0.34.0 the config format, eg. 0.29.0 should be:#
    # insecure: true                                                              #
    # The new version should be:                                                  #
    # tls:                                                                        #
    #   insecure: true                                                            #
    ###############################################################################
    tls:
      insecure: true
    #insecure: true
  # Exports data to the console
  logging:
    logLevel: debug

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [batch]
      #########################################
      # oc exporter config format:            #
      # exporters: [opencensus,logging]       #
      #########################################
      exporters: [otlp, logging]

Remember to replace 192.168.205.11 with the IP address on which the node exporter is running and

Save the file and restart the service:

sudo systemctl restart otelcol

Verify if the service is running:

$ systemctl status otelcol
● otelcol.service - OpenTelemetry Collector
   Loaded: loaded (/usr/lib/systemd/system/otelcol.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2022-10-15 08:09:13 EDT; 7s ago
 Main PID: 355530 (otelcol)
    Tasks: 6 (limit: 23198)
   Memory: 12.7M
   CGroup: /system.slice/otelcol.service
           └─355530 /usr/bin/otelcol --config=/etc/otelcol/config.yaml

Configure the OpenTelemetry receiver

To configure the OpenTelemetry receiver, edit the Apache Skywalking config file:

sudo vim /opt/apache-skywalking-apm-bin/config/application.yml

In the file, ensure that VM is captured under the enabledOtelRules to be able to collect metrics from the Linux host.

eceiver-otel:
  selector: ${SW_OTEL_RECEIVER:default}
  default:
    enabledHandlers: ${SW_OTEL_RECEIVER_ENABLED_HANDLERS:"oc,otlp"}
    enabledOtelRules: ${SW_OTEL_RECEIVER_ENABLED_OTEL_RULES:"istio-controlplane,k8s-node,oap,vm,mysql,postgresql"}

Normally, the various enabledOtelRules are defined in the config/otel-rules/ path

$ ls config/otel-rules/
istio-controlplane.yaml  k8s-cluster.yaml  k8s-instance.yaml  k8s-node.yaml  k8s-service.yaml  mysql.yaml  oap.yaml  postgresql.yaml  vm.yaml

For this case, we will use the vm.yaml that defines several rules to be used for metrics collection from a Linux host.

If any changes have been made, restart the service

sudo systemctl restart skywalking

Step 7 – View Metrics using Apache Skywalking

Now you can view metrics collected using the Prometheus Node Exporter by navigating to Infrastructure -> Linux

You can load a Graph under Dashboards -> Dashboard List

From the list, identify the dashboard for OS_LINUX and load it as shown.

Voila!

Verdict

That marks the end of this detailed guide on how to install and use Apache Skywalking Application Performance Monitoring on Linux. Using the knowledge obtained here, you can easily configure monitoring for microservices, Cloud Native, and container-based environments. I hope this was informative.

See more:

• Install Centreon Monitoring Tool on CentOS 8 | Rocky Linux 8
• Install Grafana with Prometheus on Rocky Linux 9|AlmaLinux 9
• Deploy Pinpoint APM (Application Performance Management) in Docker Containers