For the past one week, I’ve been working on a centralized Logs management system for VMware vSphere and vCenter environment. Having searched on the web for a simple, better open-source solution for this, the ideal solution seemed to be Rsyslog collector.
So I ended up installing Rsyslog server on CentOS 7 system and configuring it as a central logs server for vCenter Server Appliance(vCSA) logs and all logs generated by vSphere servers. In this guide, we will cover both installation/configuration of Rsyslog server and configuration of vCSA/vSphere to send logs to remote Rsysog server we configured.
Setting up vSphere and vCenter Central Logs Management with Rsyslog
Before we get started, let me state the assumptions here:
- You have installed CentOS 7 server; Debian based system should be fine
- Rsyslog is installed
- Your user account has sudo privileges
If any of the above is not satisfied, make sure you have all set. When done, use steps below to install and configure Rsyslog server to receive logs from your VMware vSphere and vCenter infrastructure.
Step 1: Update your system
Let’s update our system packages to the latest releases:
sudo yum update
For Ubuntu, do the same with apt:
sudo apt-get update && sudo apt-get upgrade
Reboot the system if possible for updates which require an update to take effect, e.g kernel updates:
sudo reboot
Step 2: Configure the Rsyslog server
Both CentOS and Ubuntu/Debian systems come with rsyslog installed and running. We will need to create an additional configuration file for our VMware setup.
For basic configuration of Rsyslog on Ubuntu/Debian, refer to How to Configure Rsyslog Centralized Log Server on Ubuntu 18.04 LTS
The default configuration file is./etc/rsyslog.conf. Any additional configuration can be placed under the directory/etc/rsyslog.d/.
Create a directory for VMware vSphere and vCSA appliance logs:
$ sudo mkdir -p /data/logs/{vcenter,esxi}
Then add the following configuration file to /etc/rsyslog.d/vmware.conf.This is a config for vSphere Infrastructure logging. It has templates which dictate where and how to store logs, match patterns, and log timestamp definition.
# Provides TCP syslog reception
$ModLoad imtcp
#### Create Templates for Log parsing
#### Log store is created folder previously
template(name="TIMESTAMP" type="string" string="%timegenerated:8:25%||%HOSTNAME%||%syslogtag%||%msg%\n")
template(name="ESXI" type="string" string="/data/logs/esxi/%fromhost-ip%/%$YEAR%-%$MONTH%/%$DAY%.%$MONTH%-messages.log")
template(name="VCENTER" type="string" string="/data/logs/vcenter/%fromhost-ip%/%$YEAR%-%$MONTH%/%$DAY%.%$MONTH%-messages.log")
#### Define remote logging RuleSet
$RuleSet remote
### The following rule is created to store logs coming from vCenter with different parameters
### If log is coming from vcenter with IP 192.168.10.50 it will be stored using VCENTER template.
### Replace 192.168.10.50 with your vCSA IP address
if $fromhost == '192.168.10.50' then {
*.* ?VCENTER
}
else {
# Store all other logs using TIMESTAMP and ESXI templates.
*.* ?ESXI;TIMESTAMP
}
### Tell rsyslog to listen TCP port 514.
### All events coming to this port will be parsed using remote RuleSet.
$InputTCPServerBindRuleset remote
$InputTCPServerRun 514
Here we defined three templates:
- TIMESTAMP: Used to format all logs coming from certain IP Address. This is to make logs more readable by having proper time formats.
- ESXI: This tells rsyslog to store logs from each host in a separate folder and logs from each day be stored in a folder corresponding to that day. Logs from vSphere hosts will be stored in /
data/logs/esxi - VCENTER: This tells rsyslog to store logs from a vCenter appliance in
/data/logs/vcenterusing timestamp and structure similar to the one used on ESXI template.
Restart rsyslog service after making the changes:
sudo systemctl restart rsyslog
Check status, it should be in running state:
$ systemctl status rsyslog ● rsyslog.service - System Logging Service Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2018-08-02 11:54:03 EAT; 1min ago Docs: man:rsyslogd(8) Main PID: 12489 (rsyslogd) Tasks: 8 Memory: 7.6M CGroup: /system.slice/rsyslog.service └─12489 /usr/sbin/rsyslogd -n Aug 02 11:54:03 rsyslog.example.com systemd[1]: Starting System Logging Service... Aug 02 11:54:03 rsyslog.example.com rsyslogd[12489]: [origin software="rsyslogd" swVersion="8.24.0" x-pid="12489" x-info="http://www.r...] start Aug 02 11:54:03 rsyslog.example.com systemd[1]: Started System Logging Service. Hint: Some lines were ellipsized, use -l to show in full.
Open Rsyslog firewall ports
Rsyslog listens on port 514, we configured it to user TCP, open it using your firewalld administration tool.
On Ubuntu / Debian with ufw:
sudo ufw allow 514/tcp
On CentOS 7:
sudo firewall-cmd --add-port=514/tcp --permanent sudo firewall-cmd --reload
Step 3: Configure vSphere hosts and vCSA Appliance
The only pending piece is to configure vSphere and vCSA appliance to push logs to our rsyslog server.
How to Configure vSphere remote Syslog
Open your vCenter and navigate to:
Hosts and Clusters > Select Host > Configure > Advanced System Settings > Edit
on Filter box, search “syslog”. You should see “Syslog.global.logHost“. Fill rsyslog server address inside this box:
tcp://192.168.10.35:514
If you want to configure both tcp and udp, separate them using a comma:
tcp://192.168.10.35:514,udp://192.168.10.35:514
It should look like below:
How to Configure vCenter (vCSA) remote Syslog
For vCSA, configure remote rsyslog by opening vCSA administration UI:
http://vCSA-IP:5480
This will give login dashboard:
Enter username and password provided during installation. The initial default username is root, and the default password is vmware.
Procedure:
- In the vCenter Server Appliance Management Interface, select Syslog.
- In the Forwarding Configuration section, click Configure if you have not configured any remote syslog hosts. Click Edit if you already have configured hosts.
- In the Create Forwarding Configuration page, enter the server address of the destination host. The maximum number of supported destination hosts is three.
- From the Protocol drop-down menu, select the protocol to use.
- In the Port text box, enter the port number to use for communication with the destination host.
- In the Create Forwarding Configuration pane, click Add to enter another remote syslog server.
- Click Save.
- Verify that the remote syslog server is receiving messages.
- In the Forwarding Configuration section, click Send Test Message.
- Verify on the remote syslog server that the test message was received.
The new configuration settings are shown in the Forwarding Configuration section.
If you go back to the rsyslog server, there should be two folders created under logs directory configured earlier:
$ ls /data/logs/ esx vcenter $ ls /data/logs/vcenter/192.168.10.50/2018-08/02.08-messages.log /data/logs/vcenter/192.168.10.50/2018-08/02.08-messages.log
And for ESXI hosts:
$ file /data/logs/esx/192.168.10.31/2018-08/02.08-messages.log /data/logs/esx/192.168.10.31/2018-08/02.08-messages.log: ASCII text, with very long lines
You can consider configuring log rotation for the same.
$ cat /etc/logrotate.d/vmware
/data/logs/esxi/*/*/*.log {
rotate 90
daily
notifempty
compress
}
/data/logs/vcenter/*/*/*.log {
rotate 90
daily
notifempty
compress
}
You now have a working rsyslog server for your vSphere Infrastructure. Have a happy logging.
VMware Learning Materials:
Also, read:
