See the OverOps Splunk integration site for more information.
With real-time streaming of events & metrics from OverOps directly into Splunk, SREs, DevOps and IT Ops can gain clarity of overall application health as visualized in Splunk Metric Dashboards through key performance indicators, such as the number of unique uncaught and swallowed exceptions, and errors that are new or have resurfaced. You can’t find those in log files, but now you can with OverOps data inside of Splunk!
Estimated effort: 5 minutes
The OverOps for Splunk application is compatible with OverOps v4.18 and higher. Use of any previous versions may omit some OverOps tokens and require modication to the Splunk application. Nonetheless, use of this application will provide a great head start into gaining value from the data from OverOps into Splunk.
Turn statsd on from publish metrics. (Settings -> Publish Metrics)
Define the statsD server address (this reflects a Splunk UDP data input on port 8125)
splunk-server-name-here:8125
The OverOps metrics should be defined as the following:
overops_views,${viewname},${serviceid},${application},${server},${deployment}
overops_events,${serviceid},${application},${server},${deployment},${class},${method},${eventlink},${eventid},${labels},${eventtype},${eventname},${introducedby},${entrypointclass},${entrypointmethod},${firstseen},${infra},${jiraissuekey}
overops_entrypoints,${serviceid},${application},${server},${deployment},${entrypointclass},${entrypointmetrics}
overops_custom,${serviceid},${application},${server},${deployment},${metric}
overops_jvms,${serviceid},${application},${server},${applicationpid}
Note this app utilizes a new events index within Splunk called overops. A new UDP data input is also required in Splunk, recommended on port 8125. Note: An administrator will need to complete these when upgrading to a new version of the app.
Download and Install the OverOps for Splunk app from splunkbase.
Create New Index ( Settings -> Indexes -> New Index )
a. index name = overops
b. index data type = Events
Create New Data Input ( Settings -> Data inputs -> UDP -> New Local UPD )
a. Choose UDP
b. Port: 8125 (Recommended port. This port should be provided to the OverOps Administrator to publish metrics.)
c. Source Type: Select -> Metrics -> statsd
d. App Context: OverOps for Splunk(overops)
e. Index: overops
f. Validate the new port can be accessed from the server command line where the OverOps collector is installed.
nc -v -u splunk-server 8125
View the OverOps for Splunk app. ( Apps -> OverOps for Splunk )
Edit Drill-down to OverOps Root Cause Analysis: Drill-through links to the OverOps root cause analysis will need to be updated with administration access. The dashboards are Event Details, Application Drill-down, Application Drill-down-drillthrough and Continuous Reliability. In each of these dashboards, there is a table visualization.
Edit dashboards -> More details -> Edit Drilldown -> Link to custom URL
For SaaS or Hybrid model:
https://app.overops.com/tinykey/$row.RootCause|n$
or
For On-Premise model:
https://on-prem-overops-server-here/tinykey/$row.RootCause|n$
Please be aware that some visuals in the Real-Time Application State dashboard will take a couple days to populate as there are some day over day comparisons, hence the index will need to be filled with the appropriate data.
For DevOps, IT Ops, and SREs can evalutate the overall application health as visualized through key performance indicators, such as the number of unique uncaught and swallowed exceptions, and errors that are new or have resurfaced.
For SREs and QA Engineers, identify what builds and deployments have introduced new errors into your code, so you can make informed decisions on when to promote code from environment to environment.
For the VP of Engineering and Development Leads in your organization, access OverOps data inside Splunk to aggregate specific code locations where errors and exceptions are most prevalent in your code base, and correlate error volumes to application throughput for error rate analysis.
For the VP of Engineering and Development Leads to analyze errors and exceptions by infrastructure tiers, such as db, apache, google, etc, and then immediately go to root cause analysis.
Help Development Leads cut through the noise as OverOps can deduplicate billions of logged events into accurate analytics inside Splunk, without the need to parse and search through text.
As a Splunkbase app developer, you will have access to all Splunk development resources and receive a 10GB license to build an app that will help solve use cases for customers all over the world. Splunkbase has 1000+ apps from Splunk, our partners and our community. Find an app for most any data source and user need, or simply create your own with help from our developer portal.