icon/x Created with Sketch.

Splunk Cookie Policy

We use our own and third-party cookies to provide you with a great online experience. We also use these cookies to improve our products and services, support our marketing campaigns, and advertise to you on our website and other websites. Some cookies may continue to collect information after you have left our website. Learn more (including how to update your settings) here.
Accept Cookie Policy

We are working on something new...

A Fresh New Splunkbase
We are designing a New Splunkbase to improve search and discoverability of apps. Check out our new and improved features like Categories and Collections. New Splunkbase is currently in preview mode, as it is under active development. We welcome you to navigate New Splunkbase and give us feedback.
Splunkbase will be undergoing a scheduled migration and will be unavailable on Saturday, Oct 1, 2022, from 11AM to 3PM PDT

Accept License Agreements

This app is provided by a third party and your right to use the app is in accordance with the license provided by that third-party licensor. Splunk is not responsible for any third-party apps and does not provide any warranty or support. If you have any questions, complaints or claims with respect to this app, please contact the licensor directly.

Thank You

Downloading S.o.S - Splunk on Splunk
SHA256 checksum (sos-splunk-on-splunk_321.tgz) e090d3fda466fb868f382d42e8e195003e859d10ac1f3da87cc1557fa136840b SHA256 checksum (sos-splunk-on-splunk_32.tgz) dc014510994de9e27c852208963483cfd082c5118cd9686051730395b4c18609 SHA256 checksum (sos-splunk-on-splunk_310.tgz) 5d5187603ec43786514c10c1045bba1de2fe590e63bab20648cd53a9d68ad8d0 SHA256 checksum (sos-splunk-on-splunk_301.tgz) b8891be4613d2df7587b6aee5a4e8bf279b9396b23fb18104adbfbdb0e00b866 SHA256 checksum (sos-splunk-on-splunk_30.tgz) e59ca3696b9014646519264a46f631f9d7e22ac8add75f2c6828f061dc635554 SHA256 checksum (sos-splunk-on-splunk_231.tgz) 553def8e65646d717ff5c25c2f9654e07d4deb2b7f6b609d8120dfa81a7e607c SHA256 checksum (sos-splunk-on-splunk_230.tgz) 3b4b2ca62af82286fdb67d7c6e551d5c418c2afdd40a2d4a55b1cb87221fc607 SHA256 checksum (sos-splunk-on-splunk_220.tgz) ab3a69ed3ef9ab6114e4ea425ae915f3f920b667473e1732f8802ee4ddf7659c SHA256 checksum (sos-splunk-on-splunk_210.tgz) 6eda190a0751f74fac7ef55f28346117dc0e35dd4a6e26c0cd9d4c9f7f7be628 SHA256 checksum (sos-splunk-on-splunk_200.tgz) 750a3fd81bc00f0276edd9aaca7ade3fbc2e19e59110f04275cda73ba265dea0 SHA256 checksum (sos-splunk-on-splunk_10.tgz) e8528781aa023785ae91de889173a733b041ba30031c017ce9f47a7002ef454c
To install your download
For instructions specific to your download, click the Details tab after closing this window.

Flag As Inappropriate


S.o.S - Splunk on Splunk

Splunk Labs
This app has been archived. Learn more about app archiving.
This app is NOT supported by Splunk. Please read about what that means for you here.
On April 8, 2019, this app has been deprecated and reached its End of Life on July 7, 2019. For more information about the end of availability and support for this app, see https://www.splunk.com/blog/2019/03/18/end-of-availability-splunk-built-apps-and-add-ons.html?April.

IMPORTANT: As of Splunk Enterprise 6.3, the S.o.S app is End of Life. Its functionality has been replaced and superseded by the Distributed Management Console, a feature that is included with Splunk Enterprise as of version 6.2. We recommend that you migrate from S.o.S to the DMC for all your Splunk monitoring and introspection needs at your earliest convenience.


Splunk on Splunk (S.o.S) is an app that turns Splunk's diagnostic tools inward to analyze and troubleshoot problems in your Splunk environment. It contains views and tools that allow you to do the following:
* View, search and compare Splunk configuration files.
* Detect and expose errors and anomalies in your installation, including inspection of crash logs.
* Measure indexing performance and expose event processing bottlenecks.
* View details of scheduler and user-driven search activity.
* Analyze data volume metrics captured by Splunk.

IMPORTANT: As of Splunk Enterprise 6.3, the S.o.S app is End of Life.

Its functionality has been replaced and superseded by the Distributed Management Console, a feature that is included with Splunk Enterprise as of version 6.2. We recommend that you migrate from S.o.S to the DMC for all your Splunk monitoring and introspection needs at your earliest convenience.

Splunk on Splunk installation instructions for Splunk 5.x/6.x

Please consult Splunk Answers to learn about best practices to deploy the S.o.S app in a distributed environment.

1) If Sideview Utils version 1.1.7 (or later) is not installed, please install or update it before installing S.o.S.

Download Sideview Utils from Splunkbase

2) Install the S.o.S app:

If you have a distributed search environment, make sure you install S.o.S on the search-head(s) only. From the search-head the app can discover search-peers present in the distributed deployment.

3) Download and install the Splunk on Splunk add-on (S.o.S TA) on search peers to provide resource usage information to S.o.S.

This add-on provides data inputs that gather memory and CPU usage for Splunk Web, splunkd, and search processes as well as other system resource information. For more information, see the README file available with the S.o.S TAs.

Note: You do not need to install the S.o.S add-on on a Splunk instance were the S.o.S app is already installed. The S.o.S app ships with the same data inputs.

Release Notes

Version 3.2.1
Dec. 29, 2014

Bugs fixed in version 3.2.1

  • [SOS-11] Fixed an issue where ps_sos.ps1 would log many "WriteError" exceptions to splunkd.log and insert incorrect values in its events.

  • [SOS-12] Fixed an issue where the "Security Health Check" view would fail to show results on a Splunk Enterprise 6.2 instance.

  • [SOS-19] Retired the "Bucket information" panel in the "Cluster Master View" as it was dispatching potentially unsafe rest-based searches against the Cluster Master's buckets endpoint.

  • [SOS-39] The securityinfo.py search command - and by extension, the "Security Health Check" view - now appropriately scopes its results to the instance picked by the user.

  • [SOS-40] Fixed an issue where the "cluster" command would fail to show event cluster counts due to a change in internal behavior.

  • [SOS-113][SOS-117][SOS-141] Forwarder instances will no longer be listed in the "Host" pulldown of the "Search Usage Patterns, "Scheduler Activity" and "Search Activity" views.

Version 3.2
May 6, 2014
  • Full support for Splunk Enterprise 6.1

  • NEW VIEW: Search > Search Activity
    Provides deep insight into instance-scoped search workload, expressed as search concurrency, resource usage or aggregate search time. These metrics can be grouped by various relevant search properties: mode (historical vs. real-time), type (ad hoc vs. scheduled), user...

  • NEW VIEW: Resource Usage > Indexes Disk Usage and Properties
    Allows a deployment-wide or instance-scoped view of index disk usage and other properties. Can be scoped to one or all indexes.

  • NEW VIEW: Deployment Status > Warnings and Errors > Security Health Check
    A series of checks against security settings in your Splunk Enterprise installation.

  • NEW VIEW: Indexing > Index Replication > Cluster Service Activity
    Shows service activity in a Cluster in great detail, allowing to better understand maintenance and repair operations undertaken by the Cluster Master and its peers.

  • 24 bugs fixed! See the RELEASE-NOTES file for full details.

Version 3.1.0
Sept. 30, 2013
  • New features for the Deployment Topology view
    Data overlays for instance status and resource usage (CPU/Memory).

  • NEW VIEW - Search > Search-head Pooling Performance
    Check the usage and performance of the NFS shared storage device central to search-head pooling deployments. Compare performance metrics both at the storage (NFS) and application (Splunk) levels.

  • NEW VIEW - Indexing > Metrics > License Usage - Today
    Get a license usage report for the current day and a history of license warnings for the current license window. (Applies to Splunk 4.3.x and 5.x only)

  • NEW VIEW - Indexing > Metrics > License Usage - Last 30 Days
    Get a daily license usage report for the past 30 days and break it down by pool, indexer, source, sourcetype or host. (Applies to Splunk 4.3.x and 5.x only)

  • NEW VIEW - Indexing > Index Replication > Bucket Fix-up Activity
    Monitor the status and progress of bucket fix-up operations in a cluster.

  • 10 bugs fixed! See the README file for full details.

Version 3.0.1
June 9, 2013

Bugs fixed in version 3.0.1

  • [SUP-723] Fixed an issue where scheduled searches "sos_splunk_instances_info" and "sos_refresh_splunk_servers_cache" would run several times per minute instead of at their scheduled time on a pooled search-head running Splunk 5.0.3. Note that the root cause of this problem is core Splunk bug SPL-68970.

  • [SUP-720] Fixed an issue where the Home view would be caught in a reload loop after S.o.S was installed or upgraded on a pooled search-head running Splunk 5.0.3.

  • [SUP-716] File $SPLUNK_HOME/var/log/splunk/sos_ftr.log is now explicitly sourcetyped.

  • [SUP-715] Our invocations of the "btool" command with the "--debug" flag no longer cause logs to be appended to $SPLUNK_HOME/var/log/splunk/btool.log.

  • [SUP-701] Fixed an issue where the Data Inputs > Tailing Processor view would fail to display when scoped to instances running Windows, showing instead an error banner stating "Invalid header received from stream generating script tpstatusquery".

Version 3.0
May 6, 2013

Bugs fixed in version 3.0

  • [SUP-692] Fixed an issue where the in-product app browser wouldn't be scoped
    to the Sideview Utils app during the installation workflow.

  • [SUP-668] There is now a scheduled search populating the "splunk_forwarders_cache.csv" lookup table with forwarder information.

  • [SUP-657] Added a spec file describing the "splunk_servers_cache.csv" lookup table.

  • [SUP-630] Created a macro to qualify searches based on their search ID.

  • [SUP-627] Fixed an issue where the ps_sos.sh scripted input would no longer print out full process arguments when executed by Splunk 5.x on Solaris.

  • [SUP-619] Metrics: Fixed an issue where the license usage chart would improperly show a "license_audit" pool for a license self-master.

  • [SUP-616] Fixed an issue with the ps_sos.ps1 scripted input where memory usage would sometimes be recorded as a negative value.

  • [SUP-596] Metrics: Fixed an issue where the license usage chart would not show multiple pools.

  • [SUP-578] Retired the "Distributed Searches Memory Usage" view.

  • [SUP-573] A new scripted input is now available to monitor the I/O usage of pooled search-heads on the shared NFS device: nfs-iostat_sos.py

  • [SUP-565] Fixed an issue where the ps_sos.ps1 scripted input would not run on an instance part of a search-head pool.

  • [SUP-541] Updated the app icon.

  • [SUP-540] Updated the app screenshot displayed on Splunkbase.

  • [SUP-530] Splunk File Descriptor Usage: The time stamp of the data sample used to populate the view is now shown.

  • [SUP-475] Dispatch Directory Inspector: Added a search box to filter results.

  • [SUP-474] Dispatch Directory Inspector: Added some statistical aggregations at the top of the view.

Version 2.3.1
Dec. 6, 2012

Bugs fixed in version 2.3.1

  • [SUP-606] Splunk CPU/Memory Usage: Resolved a problem where the memory usage charts would fail to report the memory usage of certain search processes.

  • [SUP-600] Metrics: Fixed an issue with the license reporting panel, which would show inaccurate numbers when multiple license pools are defined.

  • [SUP-599] Resolved a problem where the host "tag" for instances listed in the "Server to query" pulldown would not be properly determined on Splunk 5.x.

  • [SUP-595] Indexing Performance: Fixed an issue where no data points would be drawn when "Last 15mn" is selected from the time picker.

  • [SUP-589] Data Inputs Overview: Fixed an issue where this view would show no results when running on Splunk 5.x.

  • [SUP-587] Splunk CPU/Memory Usage: Renamed the "splunkd" series to "splunkd service".

  • [SUP-585] Metrics: Ensured that internal indexes and sourcetypes are no longer excluded from indexing volume reports.

  • [SUP-584] Metrics: Fixed an issue where excessive division for indexing volume metrics would lead to inaccurate reporting.

  • [SUP-583] Metrics: Fixed an issue where outgoing network throughput would be inaccurate by one order of magnitude when a split-by clause was used.

  • [SUP-582] Fixed an issue where an improper value for the "count" parameter of the "rest" command would cause a red error banner.

  • [SUP-558] Added an outputs.conf file with configuration that, if enabled, ensures that _internal events are forwarded from search-head to indexers.

  • [SUP-556] Fixed an issue where the "level" parameter of the Messages module would cause a red error banner on certain versions of Splunk.

  • [SUP-555] Resolved an issue where the "Server to query" pulldown on the Home view was not sorting hosts properly.

  • [SUP-554] Forwarders are now excluded by the searches of the Distributed Indexing view.

  • [SUP-547] Added a panel to the Indexing Performance view to expose subtask- level CPU time usage metrics for the indexer pipe which are new in 5.x.

  • [SUP-545] Adapted the searches against events generated by the ps_sos.* scripted inputs to the new splunkd process command line format in 5.x.

  • [SUP-527] Updated the build2version.csv lookup with information for the latest Splunk releases.

Version 2.3.0
Aug. 29, 2012

Bugs fixed in version 2.3

  • [SUP-538] Inputs Overview: Fixed a bug where the drilldown to file monitor input details would break due to a regular expression not supporting Windows paths.

  • [SUP-537] Home: Fixed a bug that caused the search powering the "A glimpse of your Splunk instance" panel to mismatch field values across hosts.

  • [SUP-532] Configuration File Comparator: General uncluttering and visual sanitization of this view.

  • [SUP-528] Distributed Indexing Performance: Set the height of the charts to a sensible default value.

  • [SUP-526] Scheduler Activity: Fixed wrong total execution count reported in the "Scheduler Activity" and "Execution Count by App/SavedSearch Name" panels.

  • [SUP-524] Scheduler Activity: Fixed a field extraction that was causing a NULL series to appear in the "Execution Count by App/SavedSearch Name" panel.

  • [SUP-521] Splunk CPU/Memory Resource Usage: Updated the search strings in the in-view help.

  • [SUP-507] Documented the search strings used for the Data Inputs Overview and Dispatch Directory Inspector in the in-view help.

  • [SUP-505] Fixed a typo in the lsof_sos.sh scripted input.

  • [SUP-503] Entries in the "Server to query" pulldown are now sorted based on the role of the Splunk instance: search-heads > search peers > forwarders.

  • [SUP-478] In the Errors view, improved chart readability by moving legends underneath the charting area.

Version 2.2.0
July 14, 2012
Version 2.1.0
Jan. 11, 2012

2 bugs and 4 new features in this version! Check the CHANGELOG file for details.

Version 2.0.0
Dec. 16, 2011

New features for 2.0:

Centralized Splunk instance troubleshooting
Tracking Splunk resource usage
Improved searches and data representation
Improved help panels and troubleshooting documentation
Improved visual theme

Version 1.0
Aug. 15, 2011

Subscribe Share

Are you a developer?

As a Splunkbase app developer, you will have access to all Splunk development resources and receive a 10GB license to build an app that will help solve use cases for customers all over the world. Splunkbase has 1000+ apps from Splunk, our partners and our community. Find an app for most any data source and user need, or simply create your own with help from our developer portal.

Follow Us:
Splunk, Splunk>,Turn Data Into Doing, Data-to-Everything, and D2E are trademarks or registered trademarks of Splunk Inc. in the United States and other countries. All other brand names,product names,or trademarks belong to their respective owners.