The Cisco ThousandEyes Alerting App for Splunk app includes dashboards and logic for correlating ThousandEyes alerts and SNMP traps and identifying the root cause. The correlated alerts and root cause information can be used to automatically populate ServiceNow incidents.
The Cisco ThousandEyes Alerting App for Splunk app contains four lookup files that are customer specific. These lookup files allow you to group alerts together and apply different weighting to each ThousandEyes test rule in order to identify the root cause of the issue that has triggered the alerts. SNMP traps can also be grouped together and weighting applied to identify the root cause of the issue based on the SNMP trap. Using the same group name for ThousandEyes alerts and SNMP traps allows them to be correlated, and the root cause determined by both the ThousandEyes alerts and SNMP traps (based on weighting).
The app also supports creating ServiceNow incidents and populating them with correlated alerts and SNMP traps, as well as the root cause (that is determined by the entries made to the lookup files). If ServiceNow integration is required, then the Splunk Add-on for ServiceNow must also be installed and configured.
Post installation documentation and guidelines on configuring the lookup files to define the groupings for ThousandEyes alerts and SNMP traps are available from the author.
Contact the author: email@example.com
Topology and Setting up Splunk Environment
1) Install the main app (Cisco ThousandEyes Alerting App for Splunk).
2) Create the following indexes:
3) Configure a HTTP Event Collector (HEC) token on your Forwarder:
4) Configure the ThousandEyes console to send the ThousandEyes alerts to your Forwarder using the HEC token you created in Step 3. Verify that alerts are being received using the search: index=thousandeyes sourcetype=thousandeyes:alerts
5) Optional: Configure a SNMP trap receiver with a Splunk Universal Forwarder and forward the SNMP traps. Verify that the SNMP traps are being received using the search: index=snmptrap sourcetype=snmptrap
6) Optional: Install the Splunk Add-on for ServiceNow
Configure a ServiceNow user in Splunk called thousandeyes
Enable the incidents input (frequency: 30 seconds)
7) Configure the lookup files to map the ThousandEyes testIds to user defined test groups and to apply weighting and root cause information to ThousandEyes ruleIds.
8) Optional: Configure the SNMP trap lookup files to map source/destination IP addresses to test groups and to apply weighting and root cause information.
More details are available in the post-installation configuration documentation and test configuration guidelines available from the author.
Installation of Add-on
This Add-on app can be installed through UI using "Manage Apps" or extract zip file directly into /opt/splunk/etc/apps/ folder.
As a Splunkbase app developer, you will have access to all Splunk development resources and receive a 10GB license to build an app that will help solve use cases for customers all over the world. Splunkbase has 1000+ apps and add-ons from Splunk, our partners and our community. Find an app or add-on for most any data source and user need, or simply create your own with help from our developer portal.