OverOps for Splunk Application

See the OverOps Splunk integration site for more information.

With real-time streaming of events & metrics from OverOps directly into Splunk, SREs, DevOps and IT Ops can gain clarity of overall application health as visualized in Splunk Metric Dashboards through key performance indicators, such as the number of unique uncaught and swallowed exceptions, and errors that are new or have resurfaced. You can’t find those in log files, but now you can with OverOps data inside of Splunk!

Installation procedure for the Splunk app:

Estimated effort: 5 minutes

The OverOps for Splunk application is compatible with OverOps v4.18 and higher. Use of any previous versions may omit some OverOps tokens and require modication to the Splunk application. Nonetheless, use of this application will provide a great head start into gaining value from the data from OverOps into Splunk.

The OverOps Administrator will need to perform these tasks from within the OverOps user interface:

1. Turn statsd on from publish metrics. (Settings -> Publish Metrics)

2. Define the statsD server address (this reflects a Splunk UDP data input on port 8125)

splunk-server-name-here:8125 

The OverOps metrics should be defined as the following:

overops_views,${viewname},${serviceid},${application},${server},${deployment}

overops_events,${serviceid},${application},${server},${deployment},${class},${method},${eventlink},${eventid},${labels},${eventtype},${eventname},${introducedby},${entrypointclass},${entrypointmethod},${firstseen},${infra},${jiraissuekey}

overops_entrypoints,${serviceid},${application},${server},${deployment},${entrypointclass},${entrypointmetrics}

overops_custom,${serviceid},${application},${server},${deployment},${metric}

overops_jvms,${serviceid},${application},${server},${applicationpid}

The Splunk Administrator will need to perform these tasks from within Splunk:

Note this app utilizes a new events index within Splunk called overops. A new UDP data input is also required in Splunk, recommended on port 8125. Note: An administrator will need to complete these when upgrading to a new version of the app.

1. Download and Install the OverOps for Splunk app from splunkbase.

2. Create New Index ( Settings -> Indexes -> New Index )
   a. index name = overops
   b. index data type = Events

3. Create New Data Input ( Settings -> Data inputs -> UDP -> New Local UPD )
   a. Choose UDP
   b. Port: 8125 (Recommended port. This port should be provided to the OverOps Administrator to publish metrics.)
   c. Source Type: Select -> Metrics -> statsd
   d. App Context: OverOps for Splunk(overops)
   e. Index: overops
   f. Validate the new port can be accessed from the server command line where the OverOps collector is installed.

nc -v -u splunk-server 8125

1. View the OverOps for Splunk app. ( Apps -> OverOps for Splunk )

2. Edit Drill-down to OverOps Root Cause Analysis: Drill-through links to the OverOps root cause analysis will need to be updated with administration access. The dashboards are Event Details, Application Drill-down, Application Drill-down-drillthrough and Continuous Reliability. In each of these dashboards, there is a table visualization.
   Edit dashboards -> More details -> Edit Drilldown -> Link to custom URL

For SaaS or Hybrid model:

https://app.overops.com/tinykey/$row.RootCause|n$

or
For On-Premise model:

https://on-prem-overops-server-here/tinykey/$row.RootCause|n$

Please be aware that some visuals in the Real-Time Application State dashboard will take a couple days to populate as there are some day over day comparisons, hence the index will need to be filled with the appropriate data.

Reports and Dashboards within OverOps for Splunk

Real-Time Application State

For DevOps, IT Ops, and SREs can evalutate the overall application health as visualized through key performance indicators, such as the number of unique uncaught and swallowed exceptions, and errors that are new or have resurfaced.

Continuous Reliability

For SREs and QA Engineers, identify what builds and deployments have introduced new errors into your code, so you can make informed decisions on when to promote code from environment to environment.

Application Drill-down, Exception Analysis, Log Error & Warning Analysis

For the VP of Engineering and Development Leads in your organization, access OverOps data inside Splunk to aggregate specific code locations where errors and exceptions are most prevalent in your code base, and correlate error volumes to application throughput for error rate analysis.

Infrastructure Drill-down ( drill to this report from the "Events by Tier vs Throughput" time chart within Real-Time Application State )

For the VP of Engineering and Development Leads to analyze errors and exceptions by infrastructure tiers, such as db, apache, google, etc, and then immediately go to root cause analysis.

Event Details ( drill to this report from any of the volume metrics from within Real-Time Application State )

Help Development Leads cut through the noise as OverOps can deduplicate billions of logged events into accurate analytics inside Splunk, without the need to parse and search through text.