Accept License Agreements

Thank You

Downloading S.o.S - Splunk on Splunk
MD5 checksum (sos-splunk-on-splunk_321.tgz) 35b83f352a2b1283412f5f1a79d047d3 MD5 checksum (sos-splunk-on-splunk_32.tgz) ff7849dc3117795bffbdbd23888411be MD5 checksum (sos-splunk-on-splunk_310.tgz) 68143028e67861b53135f6ad8809b584 MD5 checksum (sos-splunk-on-splunk_301.tgz) 66a174309f18a1e9c5c0c89cd2975301 MD5 checksum (sos-splunk-on-splunk_30.tgz) 65b996533f6c99475d46339aacf20637 MD5 checksum (sos-splunk-on-splunk_231.tgz) d6e9cf21484d6a178a4fd7114d488c1c MD5 checksum (sos-splunk-on-splunk_230.tgz) bfe0cb232d3e8ed713bec074559e679a MD5 checksum (sos-splunk-on-splunk_220.tgz) 4d1cfa0a676683220afbc3837d90def3 MD5 checksum (sos-splunk-on-splunk_210.tgz) f0edf58ec774d958cc21a6930d790a56 MD5 checksum (sos-splunk-on-splunk_200.tgz) 6c02ec7e7750b38ad32c05f8ec89d881 MD5 checksum (sos-splunk-on-splunk_10.tgz) dd6e70cb7fea96a7bec6791a7d5abbd9
To install your download
For instructions specific to your download, click the Details tab after closing this window.

Flag As Inappropriate

S.o.S - Splunk on Splunk

Splunk Built
IMPORTANT: As of Splunk Enterprise 6.3, the S.o.S app is End of Life. Its functionality has been replaced and superseded by the Distributed Management Console, a feature that is included with Splunk Enterprise as of version 6.2. We recommend that you migrate from S.o.S to the DMC for all your Splunk monitoring and introspection needs at your earliest convenience.


Splunk on Splunk (S.o.S) is an app that turns Splunk's diagnostic tools inward to analyze and troubleshoot problems in your Splunk environment. It contains views and tools that allow you to do the following:
* View, search and compare Splunk configuration files.
* Detect and expose errors and anomalies in your installation, including inspection of crash logs.
* Measure indexing performance and expose event processing bottlenecks.
* View details of scheduler and user-driven search activity.
* Analyze data volume metrics captured by Splunk.

IMPORTANT: As of Splunk Enterprise 6.3, the S.o.S app is End of Life.

Its functionality has been replaced and superseded by the Distributed Management Console, a feature that is included with Splunk Enterprise as of version 6.2. We recommend that you migrate from S.o.S to the DMC for all your Splunk monitoring and introspection needs at your earliest convenience.

Splunk on Splunk installation instructions for Splunk 5.x/6.x

Please consult the Splunk on Splunk User Manual and this Splunk Answer to learn about best practices to deploy the S.o.S app in a distributed environment.

1) If Sideview Utils version 1.1.7 (or later) is not installed, please install or update it before installing S.o.S.

Download Sideview Utils from Splunkbase

2) Install the S.o.S app:

If you have a distributed search environment, make sure you install S.o.S on the search-head(s) only. From the search-head the app can discover search-peers present in the distributed deployment.

3) Download and install the Splunk on Splunk add-on (S.o.S TA) on search peers to provide resource usage information to S.o.S.

This add-on provides data inputs that gather memory and CPU usage for Splunk Web, splunkd, and search processes as well as other system resource information. For more information, see the README file available with the S.o.S TAs.

Note: You do not need to install the S.o.S add-on on a Splunk instance were the S.o.S app is already installed. The S.o.S app ships with the same data inputs.

Release Notes

Version 3.2.1
Dec. 29, 2014

Bugs fixed in version 3.2.1

* [SOS-11] Fixed an issue where ps_sos.ps1 would log many "WriteError" exceptions to splunkd.log and insert incorrect values in its events.

* [SOS-12] Fixed an issue where the "Security Health Check" view would fail to show results on a Splunk Enterprise 6.2 instance.

* [SOS-19] Retired the "Bucket information" panel in the "Cluster Master View" as it was dispatching potentially unsafe rest-based searches against the Cluster Master's buckets endpoint.

* [SOS-39] The search command - and by extension, the "Security Health Check" view - now appropriately scopes its results to the instance picked by the user.

* [SOS-40] Fixed an issue where the "cluster" command would fail to show event cluster counts due to a change in internal behavior.

* [SOS-113][SOS-117][SOS-141] Forwarder instances will no longer be listed in the "Host" pulldown of the "Search Usage Patterns, "Scheduler Activity" and "Search Activity" views.

Version 3.2
May 6, 2014

* Full support for Splunk Enterprise 6.1

* NEW VIEW: Search > Search Activity
Provides deep insight into instance-scoped search workload, expressed as search concurrency, resource usage or aggregate search time. These metrics can be grouped by various relevant search properties: mode (historical vs. real-time), type (ad hoc vs. scheduled), user...

* NEW VIEW: Resource Usage > Indexes Disk Usage and Properties
Allows a deployment-wide or instance-scoped view of index disk usage and other properties. Can be scoped to one or all indexes.

* NEW VIEW: Deployment Status > Warnings and Errors > Security Health Check
A series of checks against security settings in your Splunk Enterprise installation.

* NEW VIEW: Indexing > Index Replication > Cluster Service Activity
Shows service activity in a Cluster in great detail, allowing to better understand maintenance and repair operations undertaken by the Cluster Master and its peers.

* 24 bugs fixed! See the RELEASE-NOTES file for full details.

Version 3.1.0
Sept. 30, 2013

* New features for the Deployment Topology view
Data overlays for instance status and resource usage (CPU/Memory).

* NEW VIEW - Search > Search-head Pooling Performance
Check the usage and performance of the NFS shared storage device central to search-head pooling deployments. Compare performance metrics both at the storage (NFS) and application (Splunk) levels.

* NEW VIEW - Indexing > Metrics > License Usage - Today
Get a license usage report for the current day and a history of license warnings for the current license window. (Applies to Splunk 4.3.x and 5.x only)

* NEW VIEW - Indexing > Metrics > License Usage - Last 30 Days
Get a daily license usage report for the past 30 days and break it down by pool, indexer, source, sourcetype or host. (Applies to Splunk 4.3.x and 5.x only)

* NEW VIEW - Indexing > Index Replication > Bucket Fix-up Activity
Monitor the status and progress of bucket fix-up operations in a cluster.

* 10 bugs fixed! See the README file for full details.

Version 3.0.1
June 9, 2013

Bugs fixed in version 3.0.1

* [SUP-723] Fixed an issue where scheduled searches "sos_splunk_instances_info" and "sos_refresh_splunk_servers_cache" would run several times per minute instead of at their scheduled time on a pooled search-head running Splunk 5.0.3. Note that the root cause of this problem is core Splunk bug SPL-68970.

* [SUP-720] Fixed an issue where the Home view would be caught in a reload loop after S.o.S was installed or upgraded on a pooled search-head running Splunk 5.0.3.

* [SUP-716] File $SPLUNK_HOME/var/log/splunk/sos_ftr.log is now explicitly sourcetyped.

* [SUP-715] Our invocations of the "btool" command with the "--debug" flag no longer cause logs to be appended to $SPLUNK_HOME/var/log/splunk/btool.log.

* [SUP-701] Fixed an issue where the Data Inputs > Tailing Processor view would fail to display when scoped to instances running Windows, showing instead an error banner stating "Invalid header received from stream generating script tpstatusquery".

Version 3.0
May 6, 2013

Bugs fixed in version 3.0

* [SUP-692] Fixed an issue where the in-product app browser wouldn't be scoped
to the Sideview Utils app during the installation workflow.

* [SUP-668] There is now a scheduled search populating the "splunk_forwarders_cache.csv" lookup table with forwarder information.

* [SUP-657] Added a spec file describing the "splunk_servers_cache.csv" lookup table.

* [SUP-630] Created a macro to qualify searches based on their search ID.

* [SUP-627] Fixed an issue where the scripted input would no longer print out full process arguments when executed by Splunk 5.x on Solaris.

* [SUP-619] Metrics: Fixed an issue where the license usage chart would improperly show a "license_audit" pool for a license self-master.

* [SUP-616] Fixed an issue with the ps_sos.ps1 scripted input where memory usage would sometimes be recorded as a negative value.

* [SUP-596] Metrics: Fixed an issue where the license usage chart would not show multiple pools.

* [SUP-578] Retired the "Distributed Searches Memory Usage" view.

* [SUP-573] A new scripted input is now available to monitor the I/O usage of pooled search-heads on the shared NFS device:

* [SUP-565] Fixed an issue where the ps_sos.ps1 scripted input would not run on an instance part of a search-head pool.

* [SUP-541] Updated the app icon.

* [SUP-540] Updated the app screenshot displayed on Splunkbase.

* [SUP-530] Splunk File Descriptor Usage: The time stamp of the data sample used to populate the view is now shown.

* [SUP-475] Dispatch Directory Inspector: Added a search box to filter results.

* [SUP-474] Dispatch Directory Inspector: Added some statistical aggregations at the top of the view.

Version 2.3.1
Dec. 6, 2012

Bugs fixed in version 2.3.1

* [SUP-606] Splunk CPU/Memory Usage: Resolved a problem where the memory usage charts would fail to report the memory usage of certain search processes.

* [SUP-600] Metrics: Fixed an issue with the license reporting panel, which would show inaccurate numbers when multiple license pools are defined.

* [SUP-599] Resolved a problem where the host "tag" for instances listed in the "Server to query" pulldown would not be properly determined on Splunk 5.x.

* [SUP-595] Indexing Performance: Fixed an issue where no data points would be drawn when "Last 15mn" is selected from the time picker.

* [SUP-589] Data Inputs Overview: Fixed an issue where this view would show no results when running on Splunk 5.x.

* [SUP-587] Splunk CPU/Memory Usage: Renamed the "splunkd" series to "splunkd service".

* [SUP-585] Metrics: Ensured that internal indexes and sourcetypes are no longer excluded from indexing volume reports.

* [SUP-584] Metrics: Fixed an issue where excessive division for indexing volume metrics would lead to inaccurate reporting.

* [SUP-583] Metrics: Fixed an issue where outgoing network throughput would be inaccurate by one order of magnitude when a split-by clause was used.

* [SUP-582] Fixed an issue where an improper value for the "count" parameter of the "rest" command would cause a red error banner.

* [SUP-558] Added an outputs.conf file with configuration that, if enabled, ensures that _internal events are forwarded from search-head to indexers.

* [SUP-556] Fixed an issue where the "level" parameter of the Messages module would cause a red error banner on certain versions of Splunk.

* [SUP-555] Resolved an issue where the "Server to query" pulldown on the Home view was not sorting hosts properly.

* [SUP-554] Forwarders are now excluded by the searches of the Distributed Indexing view.

* [SUP-547] Added a panel to the Indexing Performance view to expose subtask- level CPU time usage metrics for the indexer pipe which are new in 5.x.

* [SUP-545] Adapted the searches against events generated by the ps_sos.* scripted inputs to the new splunkd process command line format in 5.x.

* [SUP-527] Updated the build2version.csv lookup with information for the latest Splunk releases.

Version 2.3.0
Aug. 29, 2012

Bugs fixed in version 2.3

* [SUP-538] Inputs Overview: Fixed a bug where the drilldown to file monitor input details would break due to a regular expression not supporting Windows paths.

* [SUP-537] Home: Fixed a bug that caused the search powering the "A glimpse of your Splunk instance" panel to mismatch field values across hosts.

* [SUP-532] Configuration File Comparator: General uncluttering and visual sanitization of this view.

* [SUP-528] Distributed Indexing Performance: Set the height of the charts to a sensible default value.

* [SUP-526] Scheduler Activity: Fixed wrong total execution count reported in the "Scheduler Activity" and "Execution Count by App/SavedSearch Name" panels.

* [SUP-524] Scheduler Activity: Fixed a field extraction that was causing a NULL series to appear in the "Execution Count by App/SavedSearch Name" panel.

* [SUP-521] Splunk CPU/Memory Resource Usage: Updated the search strings in the in-view help.

* [SUP-507] Documented the search strings used for the Data Inputs Overview and Dispatch Directory Inspector in the in-view help.

* [SUP-505] Fixed a typo in the scripted input.

* [SUP-503] Entries in the "Server to query" pulldown are now sorted based on the role of the Splunk instance: search-heads > search peers > forwarders.

* [SUP-478] In the Errors view, improved chart readability by moving legends underneath the charting area.

Version 2.2.0
July 14, 2012

Version 2.1.0
Jan. 11, 2012

2 bugs and 4 new features in this version! Check the CHANGELOG file for details.

Version 2.0.0
Dec. 16, 2011

New features for 2.0:

> Centralized Splunk instance troubleshooting
> Tracking Splunk resource usage
> Improved searches and data representation
> Improved help panels and troubleshooting documentation
> Improved visual theme

Version 1.0
Aug. 15, 2011


Subscribe Share

Splunk Certification Program

Splunk's App Certification program uses a specific set of criteria to evaluate the level of quality, usability and security your app offers to its users. In addition, we evaluate the documentation and support you offer to your app's users.

Are you a developer?

As a Splunkbase app developer, you will have access to all Splunk development resources and receive a 50GB license to build an app that will help solve use cases for customers all over the world. Splunkbase has 1000+ apps and add-ons from Splunk, our partners and our community. Find an app or add-on for most any data source and user need, or simply create your own with help from our developer portal.

Follow Us:
© 2005-2017 Splunk Inc. All rights reserved.
Splunk®, Splunk>®, Listen to Your Data®, The Engine for Machine Data®, Hunk®, Splunk Cloud™, Splunk Light™, SPL™ and Splunk MINT™ are trademarks and registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners.