Enterprise Integration: SentryOne, Splunk, and SNMP

A common inquiry that arises when discussing the SentryOne platform is how it integrates with other enterprise monitoring solutions. One of the more popular solutions that we hear about is Splunk. When I was working as a SQL Server DBA at a previous employer, I was asked why we should continue to invest in a monitoring solution such as SentryOne when we also had Splunk in-house as well. The question behind that question is better framed as, “What value does SentryOne provide that Splunk does not?”

The concise version of my answer was that Splunk is an important tool that aggregates logs and events efficiently and allows you to aggregate disparate data sources into a comprehensive overview of your environment. You are also able to trigger response actions to events that Splunk indexes. However, these events (and subsequent actions) would need to be vetted through a matrix of SMEs to determine whether the event is actionable.

With its Conditions and Actions system, SentryOne provides tremendous value in this context. Conditions represent the expert knowledge required to determine if something is actionable in the first place, and Actions provide the flexibility to act on that event.

Since SentryOne and Splunk can be most appropriately viewed as complementary solutions, it’s important that we know to set them up to complement one another’s functionality within the enterprise. SentryOne currently has two primary means of sending actionable information to Splunk: SNMP Traps and PowerShell. Let’s dive into how to configure SentryOne to send SNMP traps to Splunk.

SNMP is a protocol that allows for asynchronous communication between networked devices. Its primary use case has been to monitor and manage network hardware, but SNMP can be used for a variety of purposes. To use SNMP, we need to be able to send SNMP traps, and we need to have a management device that can receive these traps. To implement this style of solution, we need to configure SentryOne to send SNMP traps to management devices. Because of the nature of SNMP, this doesn’t necessarily need to be Splunk; it can be any management/SNMP receiving device as far as SentryOne is concerned.

Configuring SentryOne to Send SNMP Traps

SentryOne makes sending SNMP traps a straightforward process.

1. Configure the SentryOne SNMP settings in Global Settings.
2. Choose the condition for which you’d like to send an SNMP trap and Send SNMP Trap Condition Action.

Configure the SentryOne SNMP Settings in Global Settings

Before we can effectively send traps, we need to configure SentryOne to send traps to a particular “community” on a network. Start by opening the Global Settings node in the SentryOne Navigator Pane.

Access the SNMP menu by clicking SNMP in the accordion menu in the main panel.

Once in the SNMP configuration section, configure the management device hostname (see number 1 in the figure below). BROADCAST (the default) will broadcast the SNMP trap throughout the network and all receivers will receive the trap, while entering a specific hostname will send the trap to that specific host. We will also need to configure the port to which we’re sending the traps (see number 2 in the figure below). 162 is the default port for listeners, but because it is a privileged port, your listener will need to be running as root.

I am going to configure my management device to run its SNMP trap listener as root, so I will leave this as 162. We will also need to configure the version of SNMP that we would like to use (see number 3 in the figure above). I’ve typically used version 2C, but for the sake of this article, we’ll use Version 1. Setting the community is a key configuration because it’s a primary way of routing and sorting SNMP traps on the management device – “public” is a common choice, so that’s what we will choose (see number 4 in the figure above).

Once you’ve completed your SentryOne configuration as I described above, it would look like the following:

Configuring a condition’s action to send an SNMP trap is a trivial matter with SentryOne. Simply access the Conditions menu (View->Conditions or use the Conditions flyout menu on the right side of the client). For this example, we’re going to add a Send SNMP Trap action to the SQL Server: Top SQL: Duration Threshold Max condition. Because this is a General Condition, we need to ensure that we’ve selected “General Conditions” from the Conditions drop-down menu. Click the Add button to add a new action, find the SQL Server: Top SQL: Duration Threshold Max condition, and expand the node to display the Action options. Select Send SNMP Trap and click OK to assign this new action.

That’s it!

Configuring Splunk to Receive Traps

A more complex, but still easily accomplished task, is configuring Splunk to index the traps that SentryOne sends. For the sake of this exercise, I am making the assumptions that you already have Splunk installed, that you have some means of Splunk access and server administration, and that Splunk is installed locally on a Linux distribution (I’m using CentOS 7).

The foundation for my setup was this Splunk support document, which provides the overview for how to achieve SNMP trap indexing. It distills the process down to a few steps:

1. Configure devices to send SNMP traps. (Done! See above.)
2. Write the traps to a file for Splunk to index.
3. Monitor the file with Splunk.

Along the way, it presents some helpful details on how to go about doing these steps, with Step 2 requiring most of the installation and configuration for SNMP trap listening.

Write the Traps to a File for Splunk to Index

Sounds simple enough, right? First, we need to install a daemon that will receive the actual trap from the network and handle it appropriately. Splunk suggested the Net-SNMP project so that is what I chose as well. How you install this will depend on which Linux distribution you’re using.

Since I am using CentOS 7, I installed using sudo yum -y install net-snmp net-snmp-utils.

Now that we’ve got SNMP installed, there are a few things that we need to configure. The Splunk documentation mentions a simple configuration of snmptrapd -Lf /var/log/snmp-traps (as root). This essentially configures snmptrapd to write the trap to a file (think: –LogType file) and the name of the file to which snmptrapd will write the file (/var/log/snmp-traps). This is going to be the file that we monitor with Splunk, so make sure that you keep the filename handy.

Theoretically, we’re able to start indexing traps as soon as we tell Splunk to monitor this file, but there are a few other steps we need to take to allow these traps to make sense. To do this, we’re going to need the SentryOne MIBs files, which are available at your SentryOne installation location: <install path>\SentryOne\18.0\SNMP MIBs\. If you did not install the MIBs during your initial installation, you will need to run the installer again and select that feature.

Once you have access to the SentryOne MIBs, place them on the Splunk server. For this example, I placed MIBs in /usr/share/snmp/mibs/. To load the MIBs at service start-up, we need to configure the MIBs to load and the directory in which to find them. Edit /etc/snmp/snmp.conf (using nano, pico, vi, etc.) and add the following lines:

While we’re in /etc/snmp, we need to edit snmpd.conf and snmptrapd.conf.
At a minimum, snmpd.conf needs the community name to which we configured SentryOne to send traps (public, in our case).

snmptrapd.conf should have a configuration like this:

Finally, we need to enable snmpd and snmptrapd to start as services:

This should enable and start the services and load the configurations we just created. To test if our MIBs have been loaded properly, we can test with snmptranslate -IR -On intercerveSQLSentryVersion61, which should return .1.3.6.1.4.1.20707.1.1.1 as a result.

With this configuration out of the way, we can now configure Splunk to monitor the trap file to which have configured our SNMP service to write (/var/log/snmp-traps).

1. Log in to Splunk as an administrator, click Settings, and choose Data Inputs under the Data heading.

2. Since we’re going to monitor a file, choose Files & Directories.

3. Click New Local File & Directory to add a new file.

4. Type in the filename that we want to monitor, ensuring to continuously monitor the file.

5. Save this new Data Input and Splunk is now configured to monitor the file that contains our SNMP traps. After a while (and pending the occurrence of the Condition-Action pairs in your environment), we should have collected some SNMP traps. Navigate to the Search & Reporting application in Splunk and choose Data Summary.

6. Click the Sources header and choose the SNMP source (file) that we configured earlier, and you should see your SentryOne SNMP traps appearing in your Splunk search.

SentryOne can serve as a valuable input into your company’s Splunk implementation. As more companies move toward Splunk (and similar solutions) to index and report on their broader infrastructure, it’s helpful to remember that specialized solutions such as SentryOne can help inform the employees who rely on solutions like Splunk to make decisions surrounding their incident reporting and management by providing accurate, actionable information. SNMP traps are an efficient way to deliver this information to the enterprise.

Seth Washeck

Seth is a former DBA (and customer) who is still recovering from weekend and overnight on-call rotations and wakes up in the morning stressed that he missed a page overnight. He is interested in how SQL Server interacts with its subsystems and can be found geeking out over storage and statistics. He enjoys spending free time with his family and friends and loves a good Vino Nobile di Montepulciano, or a fantastic Junmai Daiginjo. He loves socc … erm, footbal and is an unashamed Barça supporter.