icon/x Created with Sketch.

Splunk Cookie Policy

We use our own and third-party cookies to provide you with a great online experience. We also use these cookies to improve our products and services, support our marketing campaigns, and advertise to you on our website and other websites. Some cookies may continue to collect information after you have left our website. Learn more (including how to update your settings) here.
Accept Cookie Policy

Accept License Agreements

This app is provided by a third party and your right to use the app is in accordance with the license provided by that third-party licensor. Splunk is not responsible for any third-party apps and does not provide any warranty or support. If you have any questions, complaints or claims with respect to this app, please contact the licensor directly.

Thank You

Downloading Alerts For Splunk Admins
SHA256 checksum (alerts-for-splunk-admins_2513.tgz) 3c4ca5274f71fb6e58d04e77a6ddb3c07ab596e632e2413088acb00762f50157 SHA256 checksum (alerts-for-splunk-admins_2511.tgz) 3ff5fa4f8e175e21c1aea306c54272062e586d8be5773fbe78a158450d8f2f6c SHA256 checksum (alerts-for-splunk-admins_2510.tgz) 229c2d3f7ad4a67ef5e3faf8b87f375542727435ac5de460e7d641f012d2ce32 SHA256 checksum (alerts-for-splunk-admins_259.tgz) 7adc3116ce8cd36abfcd8cfbb0812b2adfa14c859bb8e961eb7f323f66996a39 SHA256 checksum (alerts-for-splunk-admins_258.tgz) 694458c701d10eae4b0a7dd2de87da21ac18c5521c1666ae6d87e57117f35062 SHA256 checksum (alerts-for-splunk-admins_257.tgz) 42bc8b94ffbe3b301b5b42df76d807043c8eb3e6f3bb18f4256da4df39eea817 SHA256 checksum (alerts-for-splunk-admins_256.tgz) aff575993bdf71ac4a661dcb53fa615dd17d31150d2f6ed6ce0de4f7330b579c SHA256 checksum (alerts-for-splunk-admins_255.tgz) 7d78c7a98a7c71e3a335f0a4b1476ee5653ab1f7d7129a624151b806664f9ed6 SHA256 checksum (alerts-for-splunk-admins_254.tgz) c9c527c3c505db8cb49ef92c5588c9bdbfaeb442cff305f078230410a31cb08c SHA256 checksum (alerts-for-splunk-admins_252.tgz) 551ea6d6cc499e140588c5b18a94b76c6e1d9aeaca6ff6c4f965c7487d51b1b9 SHA256 checksum (alerts-for-splunk-admins_251.tgz) 688e90a12c8a3bd501d3c92af84f3cb119b875644ce6befc5a04d1729b2a9d91 SHA256 checksum (alerts-for-splunk-admins_250.tgz) c6fd144fc9ce80bedb5eccf4bc02d27e153caf843296691c85eef862bc1866f0 SHA256 checksum (alerts-for-splunk-admins_249.tgz) 2cc9b082dc9e96ac0b4e7943edd485de4e96f1082acd74e0610840d7aba64673 SHA256 checksum (alerts-for-splunk-admins_248.tgz) 81361da0051741bf1ca8b0f943e821869ebfbf0e60d4c38edaadc622c88b7572 SHA256 checksum (alerts-for-splunk-admins_247.tgz) 05ceb41b00308c9853065e252960bf80e4920ea437c44081e98e59af66d45ef5 SHA256 checksum (alerts-for-splunk-admins_246.tgz) fccde63f685ad4480af853dc26b5eea42e7d77e2452997309953b3311aa2e248 SHA256 checksum (alerts-for-splunk-admins_245.tgz) cea1674aa529ca6ee19610976901ae3446f4f90e7914e7078ac7888ff678546f SHA256 checksum (alerts-for-splunk-admins_243.tgz) accdf3b1262af47820e841caa68e8d3f10adac81d2916f207ddd39dac72aa7b9 SHA256 checksum (alerts-for-splunk-admins_242.tgz) 457b69fc873b941effa37899e273a3de70742ba31b2c42c478e396cfe4146d4a SHA256 checksum (alerts-for-splunk-admins_241.tgz) 5fdfc3e4cb5b05a23e95dd216f893639f6378187065c13011d7206816bc52956 SHA256 checksum (alerts-for-splunk-admins_239.tgz) 45a4c7f0b0c5da10ab55e197ba0ab9979ea0c409da9b87d9cfea4b590b77ab14 SHA256 checksum (alerts-for-splunk-admins_238.tgz) a40471869c2d8cfdbb9bc59e775cd6fff4b113ba187b2443a8ea959a54befad0 SHA256 checksum (alerts-for-splunk-admins_237.tgz) 6b19abd6af063975f17aec8b997d53b748e78e09ddca1d09a10369fc822eb24d SHA256 checksum (alerts-for-splunk-admins_236.tgz) eec515054f6a6ff02462af8de0b047c08487b30dc0179196d78895a5fcfaf0de SHA256 checksum (alerts-for-splunk-admins_235.tgz) 4706f5661385174a4320cacd5d11791dc290c8036aafba35d952c50c1b4718e3 SHA256 checksum (alerts-for-splunk-admins_234.tgz) 2e639fbb7b29d6a9f4d5fdef518fa275ce67a7b5758d0e1d7e1f74c279c342ed SHA256 checksum (alerts-for-splunk-admins_233.tgz) 790acce643faa813209d45177361ddfbf06382ea02f011c0483e1b4295386563 SHA256 checksum (alerts-for-splunk-admins_232.tgz) 6a56b5830f9f4bf598028a11bc060d5955753441afa821bb50de3139370138de SHA256 checksum (alerts-for-splunk-admins_231.tgz) ec2ceb3afc023fb9b1410aa6e24f98942fde36c64c50be5d019cc2acc79c42d2 SHA256 checksum (alerts-for-splunk-admins_230.tgz) 374c06b1d5634bfe170cad6964789aae6e15dfccac2caddd3b8258cac98a2768 SHA256 checksum (alerts-for-splunk-admins_22.tgz) bad0ae1be36b4fddc90e7fc6a0cf5353f1c4a971cbfad124bdff897d2386092b SHA256 checksum (alerts-for-splunk-admins_15.tgz) 1b617c3cc34a6d2598ec8d82dc9f4c624ad960b91fd93b459b7020a02cdf8660
To install your download
For instructions specific to your download, click the Details tab after closing this window.

Flag As Inappropriate

Alerts For Splunk Admins

Splunk AppInspect Passed
Admins: Please read about Splunk Enterprise 8.0 and the Python 2.7 end-of-life changes and impact on apps and upgradeshere.
Overview
Details
This application accompanies the Splunk conf 2017 presentation "How did you get so big? Tips and tricks for growing your Splunk installation from 50GB/day to 1TB/day"

The 2017 conf presentation is linked from the detailed notes, the overall idea behind this application is to provide a variety of alerts that detect issues or potential issues within the splunk log files and then advise via an alert that this has occurred.
This application was built as there were a variety of messages in the Splunk console and logs in Splunk that if acted upon could have prevented an issue within the environment.

In addition to the alerts there are a few dashboards that relate to troubleshooting indexer/heavy forwarder performance issues.
Many of the alerts are informational and the description and comments inside the alert explain which alerts are likely to generate the most noise.
All alerts are disabled by default so you can choose which alerts may be appropriate for your environment.

Feedback is welcome!

The conf 2017 presentation How did you get so big? Tips and tricks for growing your Splunk instance from 50GB/day to 1TB/day

The conf 2017 presentation recording is available here, and PDF only here. The powerpoint presentation is available here

Introduction

This application accompanies the Splunk conf 2017 presentation "How did you get so big? Tips and tricks for growing your Splunk installation from 50GB/day to 1TB/day"

The overall idea behind this application is to provide a variety of alerts that detect issues or potential issues within the splunk log files and then advise via an alert that this has occurred
This application was built as there were a variety of messages in the Splunk console and logs in Splunk that if acted upon could have prevented an issue within the environment

There are many potential alerts that might cause an issue so this application has all alerts disabled by default, post-installation once the required macros are configured you can enable the alerts you wish to use and add the required actions

There are also a few dashboards for investigating indexer performance, heavy forwarder queue usage and data model acceleration issues

Please note that the all alerts & dashboards were tested on Linux-based Splunk infrastructure, with AIX, Linux and Windows forwarders

If you are running your Splunk enterprise installation on Windows or have customised your installation directory you will need to customise some of the macros such as splunkadmins_splunkd_source to point to the correct splunkd log file location

Macros - required configuration

The various saved searches and dashboards use macros within their searches, you will need to update the macros to ensure the searches/dashboards work as expected
To check the contents of the macros in Splunk 7 or newer, use CTRL-SHFT-E within the search window

The macros are listed below, many expect a host=A OR host=B item to assist in narrowing down a search while others expect only a single value...note that for splunk_server values they are always lower-case and case-sensitive!

indexerhosts - a host=... list of your indexers (for example host=indexer1 OR host=indexer2)

heavyforwarderhosts - a host=... list of your heavy forwarders (for example host=heavyforwarder1 OR host=heavyforwarder2)

searchheadhosts - a host=... list of your search head(s) (for example host=searchhead1 OR host=searchhead2)

localsearchheadhosts - a host=... list of your search head(s) within the cluster that these alerts are running on

splunkenterprisehosts - a host=... list of any Splunk enterprise instance (for example host=indexer1 OR host=searchhead1 OR ...)

deploymentserverhosts - a host=... list of deployment server(s) (for example host=splunkdeploymentserver)

licensemasterhost - a host=... entry for the license master server (for example host=splunklicensemaster)

searchheadsplunkservers - a splunk_server=... list of any Splunk search head hosts (for example splunk_server=searchhead*)

splunkindexerhostsvalue - a splunk_server=... list of any Splunk indexer hosts (for example splunk_server=indexer*)

splunkadmins_splunkd_source - this defaults to source=*splunkd.log, for a slight improvement in performance you can make this a specific file such as /opt/splunk/var/log/splunk/splunkd.log

splunkadmins_splunkuf_source - this defaults to source=*splunkd.log, you may wish to narrow down this location if your splunkd logs on universal forwarders have consistent installation directories

splunkadmins_mongo_source - this defaults to source=*mongod.log, for a slight improvement in performance you can make this a specific file such as /opt/splunk/var/log/splunk/mongod.log

splunkadmins_clustermaster_oshost - a host=... entry for the cluster master server (for example host=splunkclustermaster)

The macros are used in various alerts which you can optionally enable, the alerts will raise a triggered alert only as emails are not allowed for Splunk app certification purposes
The macros are also used in the dashboards for this application

The vast majority of the alerts also have a macro(s) which you can customise to tweak the search results, for example the macro splunkadmins_weekly_truncated allows the alert, IndexerLevel - Weekly Truncated Logs Report, to be customised without changing the alert itself. This will make upgrading to a new version of this app more straightforward
I have attempted to provide an appropriate macro in any alert where I deemed it appropriate, feedback is welcome for any alert that you believe should have a macro or requires further improvement

Installation

The application is designed to work on a search head or search head cluster instance, installation on the indexing tier is not required
There are a few searches that use REST API calls which are specific to the search head cluster they run on. These alerts will have to be placed on each search head or search head cluster, alternatively any server with the required search peers will also work, the relevant alerts are:
- SearchHeadLevel - Accelerated DataModels with All Time Searching Enabled
- SearchHeadLevel - Realtime Scheduled Searches are in use
- SearchHeadLevel - Realtime Search Queries in dashboards
- SearchHeadLevel - Scheduled Searches without a configured earliest and latest time
- SearchHeadLevel - Scheduled searches not specifying an index
- SearchHeadLevel - Scheduled searches not specifying an index macro version
- SearchHeadLevel - Scheduled Searches Configured with incorrect sharing
- SearchHeadLevel - Saved Searches with privileged owners and excessive write perms
- SearchHeadLevel - User - Dashboards searching all indexes
- SearchHeadLevel - User - Dashboards searching all indexes macro version
- SearchHeadLevel - Users exceeding the disk quota (recent jobs list uses a REST call so you may need to adjust the search), the SearchHeadLevel - Users exceeding the disk quota introspection is a non-search head specific alternative

The following reports also are specific to a search head or search head cluster:
- SearchHeadLevel - Alerts that have not fired an action in X days
- SearchHeadLevel - Data Model Acceleration Completion Status
- SearchHeadLevel - Macro report
- What Access Do I Have?

The following dashboards are search head or search head cluster specific:
- Data Model Rebuild Monitor
- Data Model Status

The following reports / alert must either run on the cluster master or a server where the cluster master is a peer:
- ClusterMasterLevel - Per index status
- ClusterMasterLevel - Primary bucket count per peer

Using the application

Once the application is installed, all alerts are disabled by default and you can enable those you require or want to test in your local environment.
If you choose not to customise the macros then many searches will search for all hosts, which will make the alerts and dashboards inaccurate!

Which alerts should be enabled?

The alerts are all useful for detecting a variety of different scenarios which may or may not be applicable within your Splunk environment
The description field has an (extremely) simple way of determining if an alert will require action, there are three levels:
- Low - the alert is informational and likely relates to a potential issue, these alerts may produce false alarms
- Moderate - the alert is a warning, most likely further action will need to be taken, a moderate chance of false alarms
- High - the alert is likely relating to something that requires action and there is a very low chance that this will create false alarms

I do not have a nice way to auto-enable various alerts excluding editing the local/savedsearches.conf or via the GUI, any contribution of a setup file would be welcome here!

How is this application used?

In the current environment the vast majority of the alerts are enabled to detect issues, they raise automated tickets or email depending on the urgency of the specific alert.
There are a few environment characteristics that may require changes to the way the app is used, and feedback is welcome if there is a nicer way to structure the alerts/application
The overall assumption is that the admin(s) are not carefully watching the splunkd logs or the messages in the console of the monitoring server/Splunk servers

How is this application tested?

Before 2019 the universal forwarders in use are installed on a mix of Windows, Linux & AIX servers, in 2019 and beyond the testing scope has been vastly reduced to focus primarily on Splunk enterprise servers
All heavy forwarders, and Splunk enterprise installations are Linux based, while I expect the alerts will work with only changes to the macros.conf for a Windows based environment this remains untested
The test environment for this application has a single indexer cluster and two search head clusters

Why was this application and associated conf talk created?

Inspired by articles such as "Things I wish I knew then" and knowledge collected from various conference replays, SplunkAnswers, 200+ support tickets & nearly four years of working on a Splunk environment I decided that I would attempt to share what I have learned in an attempt to prevent others from repeating the same mistakes
There are many Splunk conf talks available on this subject in various conference replays, however my goal was to provide practical steps to implement the ideas. That is why this application exists

Which alerts are best suited to automation?

  • SearchHeadLevel - Scheduled searches not specifying an index
  • SearchHeadLevel - Scheduled Searches Configured with incorrect sharing
  • SearchHeadLevel - Splunk login attempts from users that do not have any LDAP roles
  • SearchHeadLevel - Scheduled Searches That Cannot Run
  • SearchHeadLevel - Scheduled Searches without a configured earliest and latest time
  • SearchHeadLevel - Users exceeding the disk quota
  • SearchHeadLevel - User - Dashboards searching all indexes

Are all well suited to an automated email using the sendresults command or a similar function as they involve end user configuration which the individual can change/fix

Custom search commands

Due to the current SPL not handling a particular task well, and the lookup commands not supporting regular expressions, I found that the only workable solution was to create a custom lookup command.

Two exist:
- streamfilter - based on a single (or multivalue) field name, and a single (or multivalue) field with patterns, apply the regular expression in the pattern field against the nominated field(s)
- streamfilterwildcard - identical to streamfilter except that this takes a field name with wildcards, and assumes an index-style expression, so * becomes (?i)^[^_].*$, and example* becomes (?i)^example.*$

Search help is available and these are used within the reports in this application. The Splunk python SDK version 1.6.5 is also included as this is required as part of the app, an example from the reports is:
| streamfilterwildcard pattern=indexes fieldname=indexes srchIndexesAllowed

Where indexes is a field name containing a list of wildcards (_int*, _aud*) or similar, indexes is the output field name, srchIndexesAllowed is the field name which the indexes field will be compared to.
Each entry in the pattern field will be compared to each entry in the srchIndexesAllowed field in this example

To make this command work the Splunk python SDK is bundled into the app, if the bin directory is wiped due to issues with other applications this only disables the two commands which are used in Search Queries summary non-exact match so far

KVStore Usage

Some CSV lookups are now replaced with kvstore entries due to the ability to sync the kvstore across multiple search head or search head cluster(s) via apps like TA-SyncKVStore https://splunkbase.splunk.com/app/3519/

Feedback?

Feel free to provide feedback via SplunkBase and contributions are welcome!

GitHub Location

This project is open source and hosted on github SplunkAdmins

Release Notes

2.5.12

New alerts:
SearchHeadLevel - splunk_search_messages dispatch

SearchHeadLevel - WLM aborted searches

SearchHeadLevel - dispatch metadata files may need removal

Minor changes to reports:
SearchHeadLevel - Search Queries summary exact match 73

SearchHeadLevel - Search Queries summary non-exact match 73

And macro:
splunkadmins_audit_logs_datamodel_sub

Updated alert:
SearchHeadLevel - Dashboards with all time searches set to look for earliest= in tokens and to ignore that case

Updated reports:
SearchHeadLevel - Indexer Peer Connection Failures

SearchHeadLevel - Detect searches hitting corrupt buckets

The above were updated to use splunk_search_messages sourcetype

IndexerLevel - Knowledge bundle upload stats updated to handle cascading bundle replication

2.5.11

Added notes around the log_search_messages property under [search] in limits.conf

New macros:
conf_rest_endpoint

splunkadmins_epoch

splunkadmins_audit_logs_datamodel_sub

splunkadmins_audit_logs_eventtypes_sub

splunkadmins_audit_logs_macro_sub_v8 - note this version uses mvmap so Splunk v8+, the splunkadmins_audit_logs_macro_sub still exists for pre-version 8 but can only replace 1 macro per run...

splunkadmins_audit_logs_tags_sub

New reports:
SearchHeadLevel - DataModels report

SearchHeadLevel - Tags report

SearchHeadLevel - EventTypes report

Updated dashboard troubleshooting_resource_usage_per_user_drilldown to display the correct time range for more searches

Updated reports:
IndexerLevel - RemoteSearches Indexes Stats - to summarize indexes stats

SearchHeadLevel - Scheduled searches not specifying an index macro version

SearchHeadLevel - User - Dashboards searching all indexes macro version

SearchHeadLevel - Search Queries By Type Audit Logs macro version

SearchHeadLevel - Search Queries By Type Audit Logs macro version other

SearchHeadLevel - Dashboards with all time searches set

To use the new macro splunkadmins_audit_logs_macro_sub_v8

Upated reports:
SearchHeadLevel - Search Queries summary exact match 73

SearchHeadLevel - Search Queries summary non-exact match 73

To use the new macros splunkadmins_audit_logs_macro_sub_v8, splunkadmins_audit_logs_eventtypes_sub, splunkadmins_audit_logs_datamodel_sub, splunkadmins_audit_logs_tags_sub

2.5.10

Updated to Splunk python SDK 1.6.13 (previous 2.5.9 did not include this update)

New alerts:
AllSplunkLevel - TailReader Ignoring Path

ForwarderLevel - Channel churn issues

SearchHeadLevel - Dashboards with all time searches set

New reports:
SearchHeadLevel - audit logs showing all time searches

Updated reports:
SearchHeadLevel - Macro report to use the new macro

SearchHeadLevel - Search Queries summary exact match 73 to use the new macro

SearchHeadLevel - Search Queries summary non-exact match 73 to use the new macro

New macros:
splunkadmins_splunk_server_name

2.5.9

New alerts:
AllSplunkLevel - Unexpected termination of a Splunk process windows

AllSplunkLevel - Unexpected termination of a Splunk process unix

IndexerLevel - strings_metadata triggering bucket rolling

New reports:
ForwarderLevel - Data dropping duration

SearchHeadLevel - Lookup CSV size

New dashboards:
lookup_audit

New macro:
mylookups (7.3.3+ only)

New nav menu items:
Hyperlink to https://github.com/silkyrich/cluster_health_tools

Updated to Splunk python SDK 1.6.12
Set python.version = python3 within inputs.conf.spec as per appinspect requirement

2.5.8

New alerts:
ClusterMasterLevel - excess buckets on master

Updated alerts:
ForwarderLevel - Splunk HEC issues - corrected criteria for newer Splunk versions and added more matching in

SearchHeadLevel - SHC Captain unable to establish common bundle - to remove special character from comment

Renamed alert:
IndexerLevel - Buckets are been frozen due to index sizing to IndexerLevel - Buckets have being frozen due to index sizing (as requested by woodcock)

New reports:
SearchHeadLevel - Dashboards using special characters

SearchHeadLevel - SavedSearches using special characters

2.5.7

Moved lib directory to bin/lib (as this does not distribute to the indexers otherwise, sent feedback on https://dev.splunk.com/enterprise/docs/python/sdk-python/howtousesplunkpython/howtocreatemodpy/ so this gets updated)

New macro:

base64decode this macro requires decrypt or a similar app to be useful but the searches utilising this will work fine without it...

New reports:
SearchHeadLevel - platform_stats.audit metrics searches

SearchHeadLevel - platform_stats.audit metrics users

SearchHeadLevel - platform_stats.audit metrics api

The above 3 replace SearchHeadLevel - platform_stats.audit metrics which is now removed.

New reports continued:

IndexerLevel - RemoteSearches Indexes Stats

SearchHeadLevel - DataModel Fields

SearchHeadLevel - Dashboard refresh intervals

SearchHeadLevel - Dashboards using depends and running searches in the background

SearchHeadLevel - Summary searches using realtime search scheduling

SearchHeadLevel - Searches dispatched as owner by other users

Updated reports:

SearchHeadLevel - Search Queries summary exact match

SearchHeadLevel - Search Queries summary non-exact match

Minor tweaks to the regex for both the above

SearchHeadLevel - Search Queries summary exact match 73

SearchHeadLevel - Search Queries summary non-exact match 73

The above now attempt to handle append, join, appendcols, multisearch

Also updated reports:

SearchHeadLevel - platform_stats.remote_searches metrics populating search to ignore pretypeahead/copybuckets searches, and default acceleration searches

SearchHeadLevel - platform_stats.user_stats.introspection metrics populating search to include indexer cluster as a field

SearchHeadLevel - Scheduled Searches That Cannot Run to handle additional failure scenarios

Updated streamfilter.py, lookup_watcher.py and streamfilterwildcard.py so they include the libraries from bin/lib

Older releases

Refer to the README file for the full history of release notes

Other

Icons made by Freepik from www.flaticon.com is licensed by Creative Commons BY 3.0

Release Notes

Version 2.5.13
Sept. 15, 2020

2.5.13 is identical to 2.5.12 and includes an extra lookup file to pass appinspect

New alerts:
SearchHeadLevel - splunk_search_messages dispatch
SearchHeadLevel - WLM aborted searches
SearchHeadLevel - dispatch metadata files may need removal

Minor changes to reports:
SearchHeadLevel - Search Queries summary exact match 73
SearchHeadLevel - Search Queries summary non-exact match 73

And macro:
splunkadmins_audit_logs_datamodel_sub

Updated alert:
SearchHeadLevel - Dashboards with all time searches set to look for earliest= in tokens and to ignore that case

Updated reports:
SearchHeadLevel - Indexer Peer Connection Failures
SearchHeadLevel - Detect searches hitting corrupt buckets

The above were updated to use `splunk_search_messages` sourcetype

IndexerLevel - Knowledge bundle upload stats updated to handle cascading bundle replication

Version 2.5.11
July 20, 2020

Added notes around the log_search_messages property under [search] in limits.conf

New macros:
conf_rest_endpoint
splunkadmins_epoch
splunkadmins_audit_logs_datamodel_sub
splunkadmins_audit_logs_eventtypes_sub
splunkadmins_audit_logs_macro_sub_v8 - note this version uses mvmap so Splunk v8, the splunkadmins_audit_logs_macro_sub still exists for pre-version 8 but can only replace 1 macro per run...
splunkadmins_audit_logs_tags_sub

New reports:
SearchHeadLevel - DataModels report
SearchHeadLevel - Tags report
SearchHeadLevel - EventTypes report

Updated dashboard troubleshooting_resource_usage_per_user_drilldown to display the correct time range for more searches

Updated reports:
IndexerLevel - RemoteSearches Indexes Stats - to summarize indexes stats
SearchHeadLevel - Scheduled searches not specifying an index macro version
SearchHeadLevel - User - Dashboards searching all indexes macro version
SearchHeadLevel - Search Queries By Type Audit Logs macro version
SearchHeadLevel - Search Queries By Type Audit Lo

Version 2.5.10
June 17, 2020

Updated to Splunk python SDK 1.6.13 (previous 2.5.9 did not include this update)

New alerts:
AllSplunkLevel - TailReader Ignoring Path
ForwarderLevel - Channel churn issues
SearchHeadLevel - Dashboards with all time searches set

New reports:
SearchHeadLevel - audit logs showing all time searches

Updated reports:
SearchHeadLevel - Macro report` to use the new macro
SearchHeadLevel - Search Queries summary exact match 73` to use the new macro
SearchHeadLevel - Search Queries summary non-exact match 73` to use the new macro

New macros:
splunkadmins_splunk_server_name

Version 2.5.9
May 6, 2020

New alerts:
AllSplunkLevel - Unexpected termination of a Splunk process windows
AllSplunkLevel - Unexpected termination of a Splunk process unix
IndexerLevel - strings_metadata triggering bucket rolling

New reports:
ForwarderLevel - Data dropping duration
SearchHeadLevel - Lookup CSV size

New dashboards:
lookup_audit

New macro:
mylookups (7.3.3+ only)

New nav menu items:
Hyperlink to https://github.com/silkyrich/cluster_health_tools

Updated to Splunk python SDK 1.6.12
Set python.version = python3 within inputs.conf.spec as per appinspect requirement

Version 2.5.8
March 8, 2020

New alerts:
ClusterMasterLevel - excess buckets on master

Updated alerts:
ForwarderLevel - Splunk HEC issues - corrected criteria for newer Splunk versions and added more matching in
SearchHeadLevel - SHC Captain unable to establish common bundle - to remove special character from comment

Renamed alert:
IndexerLevel - Buckets are been frozen due to index sizing to IndexerLevel - Buckets have being frozen due to index sizing (as requested by woodcock)

New reports:
SearchHeadLevel - Dashboards using special characters
SearchHeadLevel - SavedSearches using special characters

Version 2.5.7
Jan. 28, 2020

Moved lib directory to bin/lib (as this does not distribute to the indexers otherwise, sent feedback on https://dev.splunk.com/enterprise/docs/python/sdk-python/howtousesplunkpython/howtocreatemodpy/ so this gets updated)

New macro:
base64decode this macro requires decrypt or a similar app to be useful but the searches utilising this will work fine without it...

New reports:
SearchHeadLevel - platform_stats.audit metrics searches
SearchHeadLevel - platform_stats.audit metrics users
SearchHeadLevel - platform_stats.audit metrics api

The above 3 replace `SearchHeadLevel - platform_stats.audit metrics` which is now removed.

New reports continued:
IndexerLevel - RemoteSearches Indexes Stats
SearchHeadLevel - DataModel Fields
SearchHeadLevel - Dashboard refresh intervals
SearchHeadLevel - Dashboards using depends and running searches in the background
SearchHeadLevel - Summary searches using realtime search scheduling
SearchHeadLevel - Searches dispatched as owner by other users

Various report updates...

Version 2.5.6
Dec. 31, 2019

Further updates to the new reports from 2.5.5 relating to platform stats, improved accuracy with identifying dashboard usage vs ad-hoc searches
Lookup Watcher now imports six from lib directory (allows this to work on older Splunk versions)
Minor update to props.conf for splunk:search:info as in 7.3 auto-finalized messages are now INFO level

Updated SearchHeadLevel - platform_stats access summary to include searches triggered (which are often coming from dashboard usage)

New report:
SearchHeadLevel - platform_stats.remote_searches metrics populating search

Updated reports:
IndexerLevel - platform_stats.counters hosts
IndexerLevel - platform_stats.counters hosts 24hour
IndexerLevel - platform_stats.indexers totalgb measurement
SearchHeadLevel - SHC conf log summary
SearchHeadLevel - platform_stats.audit metrics
SearchHeadLevel - platform_stats.user_stats.introspection metrics populating search
SearchHeadLevel - platform_stats access summary

New macro:
search_type_from_sid

Version 2.5.5
Dec. 28, 2019

Lookup Watcher now imports six from lib directory (allows this to work on older Splunk versions)
Minor update to props.conf for splunk:search:info as in 7.3 auto-finalized messages are now INFO level

Various new summary reports that record platform level metrics/stats

New alert:
SearchHeadLevel - SHC Captain unable to establish common bundle

New reports:
IndexerLevel - platform_stats.counters hosts
IndexerLevel - platform_stats.counters hosts 24hour
IndexerLevel - platform_stats.indexers totalgb measurement
SearchHeadLevel - SHC conf log summary
SearchHeadLevel - platform_stats.audit metrics
SearchHeadLevel - platform_stats.user_stats.introspection metrics populating search
SearchHeadLevel - platform_stats access summary

Updated dashboard:
indexer_max_data_queue_sizes_by_name

New macro:
search_head_cluster

Version 2.5.4
Nov. 15, 2019

Identical to 2.5.3, re-release due to SplunkBase issue

Changes for python3 compatability
Updated python SDK to 1.6.11 (from 1.6.6)

Lookup files are now included (zero sized)

New macros:
splunkadmins_audit_logs_macro_sub
splunkadmins_remote_macros (this macro requires Webtools Add-on)
splunkadmins_remote_roles (this macro requires Webtools Add-on)

New reports:
SearchHeadLevel - Search Queries summary exact match 73
SearchHeadLevel - Search Queries summary exact match 73 by user (uses Search Queries summary exact match 73 as base)
SearchHeadLevel - Search Queries summary exact match 73 by index (uses Search Queries summary exact match 73 as base)
SearchHeadLevel - Search Queries summary non-exact match 73

Updated alerts:
IndexerLevel - Time format has changed multiple log types in one sourcetype
IndexerLevel - Timestamp parsing issues combined alert

Updated dashboard:
issues_per_sourcetype

Full details in the README.md or details tab

Version 2.5.2
July 20, 2019

New modular input - Lookup Watcher - details in the README.md file
Introduced a new sub-menu in the navigation menu for Search Head Level "Recommended (externally hosted)" with links to external dashboards

Updated reports:
SearchHeadLevel - Search Queries By Type Audit Logs
SearchHeadLevel - Search Queries By Type Audit Logs macro version
SearchHeadLevel - Search Queries By Type Audit Logs macro version other

To reduce the number of unknown queries

Updated reports:
SearchHeadLevel - Search Queries summary exact match
SearchHeadLevel - Search Queries summary non-exact match

To improve the statistics around indexes found

Please note that if you are upgrading from a version pre 2.5.0 the bin/splunklib directory can be deleted from this app ($SPLUNK_HOME/etc/apps/SplunkAdmins/bin/splunklib is no longer required)

If you like this app you may also be interested in VersionControl For Splunk https://splunkbase.splunk.com/app/4355/

Version 2.5.1
July 2, 2019

Updated alert - `SearchHeadLevel - Scheduled Searches That Cannot Run` to find more results
Updated dashboard `issues per sourcetype` to handle message becoming event_message in newer Splunk versions (7.1 or 7.2)
Updated macros `splunkadmins_shutdown_list`, `splunkadmins_shutdown_keyword`, `splunkadmins_shutdown_time`, `splunkadmins_transfer_captain_times` to handle message becoming event_message in newer Splunk versions (7.1 or 7.2)

Updated python files streamfilter/streamfilterwildcard to import lib relative to the current app name

Updated many alerts/reports to handle the message field becoming event_message in newer Splunk versions (7.1 or 7.2), full list in the details tab or the README.md file

Please note that if you are upgrading from a version pre 2.5.0 the bin/splunklib directory can be deleted from this app ($SPLUNK_HOME/etc/apps/SplunkAdmins/bin/splunklib is no longer required)

If you like this app you may also be interested in VersionControl For Splunk https://splunkbase.splunk.com/app/4355/

Version 2.5.0
April 29, 2019

New macro - splunkadmins_shutdown_keyword
New report - IndexerLevel - Knowledge bundle upload stats
Updated alert - AllSplunkEnterpriseLevel - Replication Failures with new criteria and excluded shutdowns
Updated alert - AllSplunkEnterpriseLevel - Splunk Scheduler skipped searches and the reason to handle another skipped scenario
Updated alert - AllSplunkEnterpriseLevel - Splunk Servers with resource starvation with new comments
Updated alert - SearchHeadLevel - Detect MongoDB errors with update to handle tstats issue in Splunk (issue #3 in github)
Moved splunklib from bin to lib directory as per new appinspect recommendations

Please note that if you are upgrading from an older version the bin/splunklib directory can be deleted from this app ($SPLUNK_HOME/etc/apps/SplunkAdmins/bin/splunklib is no longer required)

If you like this app you may also be interested in VersionControl For Splunk https://splunkbase.splunk.com/app/4355/

Version 2.4.9
March 1, 2019

Change summary:
New alert - ForwarderLevel - Splunk HEC issues
New dashboard - Lookups in use finder
New macro - splunkadmins_license_usage_source
New report - IndexerLevel - Maximum memory utilisation per search
New report - SearchHeadLevel - Lookup updates within SHC
New report - SearchHeadLevel - Maximum memory utilisation per search
New report - SearchHeadLevel - Detect Excessive Search Use - Dashboard - Automated
Updated alert - AllSplunkEnterpriseLevel - Replication Failures to match more results
Updated alert - ForwarderLevel - Splunk HTTP Listener Overwhelmed comment/description update
Updated alert - SearchHeadLevel - Detect MongoDB errors` to include " W " based on git feedback
Updated dashboard - Rolled buckets by index - to no longer hardcode Linux paths to the license usage log
Updated dashboard - Heavy Forwarders Max Data Queue Sizes by name to use the thruput in the metrics.log

If you like this app you may also be interested in VersionControl For Splunk https://splunkbase.splunk.com/app/4355/

Version 2.4.8
March 1, 2019

Change summary:
New alert - ForwarderLevel - Splunk HEC issues
New dashboard - Lookups in use finder
New macro - splunkadmins_license_usage_source
New report - IndexerLevel - Maximum memory utilisation per search
New report - SearchHeadLevel - Lookup updates within SHC
New report - SearchHeadLevel - Maximum memory utilisation per search
New report - SearchHeadLevel - Detect Excessive Search Use - Dashboard - Automated
Updated alert - AllSplunkEnterpriseLevel - Replication Failures to match more results
Updated alert - ForwarderLevel - Splunk HTTP Listener Overwhelmed comment/description update
Updated dashboard - Rolled buckets by index - to no longer hardcode Linux paths to the license usage log
Updated dashboard - Heavy Forwarders Max Data Queue Sizes by name to use the thruput in the metrics.log

If you like this app you may also be interested in VersionControl For Splunk https://splunkbase.splunk.com/app/4355/

Version 2.4.7
Feb. 1, 2019

Introduced an updated navigation menu to navigate around the alerts, reports and dashboards available in the app
Created a number of reports from Splunk audit logs around user search activity
New dashboards for troubleshooting poor user behaviour (Detect excessive search usage)

Change summary
2.4.7
New README (README.md replaces README)
New dashboard Detect excessive search usage
New dashboard Cluster Master Jobs
New dashboard Knowledge Objects by app (and drilldown dashboard)
New report - IndexerLevel - Corrupt buckets via DBInspect
New report - SearchHeadLevel - Detect changes to knowledge objects
New report - SearchHeadLevel - Detect changes to knowledge objects directory
New report - SearchHeadLevel - Detect changes to knowledge objects non-directory

If you like this app you may also be interested in VersionControl For Splunk https://splunkbase.splunk.com/app/4355/

Version 2.4.6
Dec. 22, 2018

Minor changes since 2.4.5

Introduced an updated navigation menu to navigate around the alerts, reports and dashboards available in the app
Created a number of reports from Splunk audit logs around user search activity

Change summary
2.4.6
New alert - AllSplunkLevel - Data Loss on shutdown
New macro - whataccessdoihave - can be used with | `whataccessdoihave` by users
New report - SearchHeadLevel - Dashboard load times
New report - SearchHeadLevel - Scheduled searches status
Updated dashboard - Troubleshooting Resource Usage Per User Drilldown - now uses search_et/search_lt
Upgraded Splunk python SDK to 1.6.6, if this causes problems remove the bin directory this only disables the "Search Queries summary non-exact match" report

2.4.5
New dashboard - Troubleshooting Resource Usage Per User (&drilldown)
New commands - streamfilter, streamfilterwildcard
New reports around search query info

If you like this app you may also be interested in VersionControl For Splunk https://splunkbase.splunk.com/app/4355/

Version 2.4.5
Nov. 6, 2018

Introduced an updated navigation menu to navigate around the alerts, reports and dashboards available in the app
Changed label of all dashboards to have Dashboard - ... this is just to make the navigation menu work as expected
Created a number of reports from Splunk audit logs around user search activity
Updated and corrected a number of alerts/reports, full details in the release notes

Change summary
New dashboard - Troubleshooting Resource Usage Per User
New dashboard - Troubleshooting Resource Usage Per User Drilldown
New command - streamfilter
New command - streamfilterwildcard
New reports around search query information

Version 2.4.3
Sept. 6, 2018

No functional changes from 2.4.1/2.4.2, commenting out sendresults in 1 alert and removed 2 harmless errors to obtain the appinspect badge

Introduced an updated navigation menu to navigate around the alerts, reports and dashboards available in the app
Changed label of all dashboards to have Dashboard - ... this is just to make the navigation menu work as expected

New alert IndexerLevel - Buckets changes per day
New alert IndexerLevel - Timestamp parsing issues combined alert
New report SearchHeadLevel - Audit log search example only
Updated alert IndexerLevel - Future Dated Events that appeared in the last week to +10y instead of +20y
Updated alert IndexerLevel - Indexer Queues May Have Issues - to work with multiple pipelines
Updated alert IndexerLevel - Buckets rolling more frequently than expected with an improved regex
Updated alert SearchHeadLevel - Captain Switchover Occurring - to ignore manual captain transfers
Corrected alert SearchHeadLevel - Determine query scan density with a relevant query

Version 2.4.2
Sept. 6, 2018

No functional changes from 2.4.1, commenting out sendresults in 1 alert to obtain the appinspect badge

Introduced an updated navigation menu to navigate around the alerts, reports and dashboards available in the app
Changed label of all dashboards to have Dashboard - ... this is just to make the navigation menu work as expected

New alert IndexerLevel - Buckets changes per day
New alert IndexerLevel - Timestamp parsing issues combined alert
New report SearchHeadLevel - Audit log search example only
Updated alert IndexerLevel - Future Dated Events that appeared in the last week to +10y instead of +20y
Updated alert IndexerLevel - Indexer Queues May Have Issues - to work with multiple pipelines
Updated alert IndexerLevel - Buckets rolling more frequently than expected with an improved regex
Updated alert SearchHeadLevel - Captain Switchover Occurring - to ignore manual captain transfers
Corrected alert SearchHeadLevel - Determine query scan density with a relevant query

Version 2.4.1
Aug. 25, 2018

Introduced an updated navigation menu to navigate around the alerts, reports and dashboards available in the app
Changed label of all dashboards to have Dashboard - ... this is just to make the navigation menu work as expected

New alert IndexerLevel - Buckets changes per day
New alert IndexerLevel - Timestamp parsing issues combined alert
New report SearchHeadLevel - Audit log search example only
Updated alert IndexerLevel - Future Dated Events that appeared in the last week to +10y instead of +20y
Updated alert IndexerLevel - Indexer Queues May Have Issues - to work with multiple pipelines
Updated alert IndexerLevel - Buckets rolling more frequently than expected with an improved regex
Updated alert SearchHeadLevel - Captain Switchover Occurring - to ignore manual captain transfers
Corrected alert SearchHeadLevel - Determine query scan density with a relevant query

Version 2.3.9
Aug. 2, 2018

Please note that this release renames the alert IndexerLevel - ERROR from linebreaker to IndexerLevel - Data parsing error
Also note that the app certification program is now over, a new badge for "app inspect certified" will appear in September

Creating new reports around users exceeding the disk quota, updates to a few other reports/alerts.

Update summary:
Updated alert SearchHeadLevel - Users exceeding the disk quota to include username
Updated report ForwarderLevel - Forwarders connecting to a single endpoint for extended periods (and UF level version)
Renamed alert IndexerLevel - ERROR from linebreaker to IndexerLevel - Data parsing error
New report SearchHeadLevel - Users exceeding the disk quota introspection
New report SearchHeadLevel - Users exceeding the disk quota introspection cleanup

Version 2.3.8
June 19, 2018

New reports for diagnosing forwarder issues, alerts around bucket corruption and peer connection failures
New dashboards for troubleshooting sourcetypes or buckets rolled per day
Updated all alerts with an investigationQuery to use index=* explicitly rather than assume the admin has all indexes listed in the indexes searched by default list
Updated alert ClusterMasterLevel - Per index status to 5 minute intervals for certification purposes

Refer to the details tab or README for more information

Version 2.3.7
June 10, 2018

New reports for diagnosing forwarder issues, alerts around bucket corruption and peer connection failures
New dashboards for troubleshooting sourcetypes or buckets rolled per day
New reports around forwarders connected to indexers or HF's for extended periods
New dashboard - Issues per sourcetype, a combination of timestamp parsing, future based and past data searches to look at a single problematic sourcetype
New dashboard - Rolled buckets by index, a dashboard to assist with determing which index is rolling the most buckets

Updated all alerts with an investigationQuery to use index=* explicitly rather than assume the admin has all indexes listed in the indexes searched by default list

Version 2.3.6
June 8, 2018

New reports for diagnosing forwarder issues, alerts around bucket corruption and peer connection failures
New dashboards for troubleshooting sourcetypes or buckets rolled per day
New reports around forwarders connected to indexers or HF's for extended periods
New dashboard - Issues per sourcetype, a combination of timestamp parsing, future based and past data searches to look at a single problematic sourcetype
New dashboard - Rolled buckets by index, a dashboard to assist with determing which index is rolling the most buckets

Updated all alerts with an investigationQuery to use index=* explicitly rather than assume the admin has all indexes listed in the indexes searched by default list

Version 2.3.5
May 15, 2018

Minor tweaks to support 7.1
Updated IndexerLevel - Cold data location approaching size limits to handle only maxTotalDataSizeMB been set
Updated Future Dated Events that appeared in the last week to use +10y and 7.1 rejects +20y
Corrected AllSplunkEnterpriseLevel - TCP or SSL Config Issue to remove extra ( symbol
Corrected SearchHeadLevel - User - Dashboards searching all indexes macro version to refer to correct lookup name
Corrected dashboard for troubleshooting indexer CPU to handle standalone server
Inclusion of alternative app icons to work in 7.1
Updated SearchHeadLevel - Scheduled searches not specifying an index to exclude 1 additional type of search
Updated SearchHeadLevel - KVStore Or Conf Replication Issues Are Occurring to detect a disconnected member scenario
Updated Troubleshooting indexer CPU & drilldown dashboards to include commmas and the search head field (to make it easier to update to search head instead of indexer hosts)

Note 2.3.4 was not released

Version 2.3.4
May 3, 2018

Update summary:
Updated SearchHeadLevel - Scheduled searches not specifying an index to exclude 1 additional type of search
Updated SearchHeadLevel - KVStore Or Conf Replication Issues Are Occurring to detect a disconnected member scenario
Updated Troubleshooting indexer CPU & drilldown dashboards to include commas and the search head field (to make it easier to update to search head instead of indexer hosts)

Version 2.3.3
March 21, 2018

Update summary:
New alert SearchHeadLevel - Disabled modular inputs are running
Updated SearchHeadLevel - Detect MongoDB errors to timechart to have no limit on the number of hosts involved
Updated the shutdown macros to find one additional scenario

Version 2.3.2
March 6, 2018

Due to resourcing issues on the search heads this includes a few warnings/errors related to performance issues

Update summary:
New alert AllSplunkEnterpriseLevel - Splunk Servers with resource starvation
New alert IndexerLevel - S2SFileReceiver Error
New alert SearchHeadLevel - Captain Switchover Occurring
Various miscellaneous updates (refer to README)
Corrected AllSplunkEnterpriseLevel - sendmodalert errors to not show random savedsearch_names when no match is found
Corrected SearchHeadLevel - Alerts that have not fired an action in X days to only show alerts relevant to the current search head/cluster

Version 2.3.1
Feb. 22, 2018

Update summary:
New alert AllSplunkEnterpriseLevel - Non-existent roles are assigned to users
New alert IndexerLevel - Index not defined
New alert IndexerLevel - Search Failures
New alert SearchHeadLevel - Saved Searches with privileged owners and excessive write perms (detect 1 way of accessing data outside your level of access)
New report SearchHeadLevel - Macro report (required by "macro version" alerts)
New alerts for detecting macro usage within saved searches/dashboards
Various minor updates
Corrected AllSplunkLevel - Unable To Distribute to Peer
Corrected IndexerLevel - Failures To Parse Timestamp Correctly (excluding breaking issues) to correctly exclude broken events & to handle newer 7.0.2 errors

Version 2.3.0
Feb. 7, 2018

Attempt to reduce false alarms and improve investigationQuery searches
Created macros for shutdown events for indexers/search heads/enterprise servers for excluding false alarms related to restarts

Update summary:
New alert ClusterMasterLevel - Per index status
New macro splunkadmins_shutdown_list
New macro splunkadmins_shutdown_time
New report ClusterMasterLevel - Primary bucket count per peer
Updated various alerts to use new shutdown macros
Updated SearchHeadLevel - Scheduled Searches That Cannot Run - to detect errors in splunkd related to saved searches
Corrected SearchHeadLevel - User - Dashboards searching all indexes - a newline resulted in it working in search but not via the scheduler!

Version 2.2
Jan. 31, 2018

Attempted to reduce false alarms and made further improvements to investigationQueries, the idea behind the "investigationQuery" is that you can copy and paste the output into a search window and see results relevant to the particular alert

Macros now exist for shutdown events for indexers/search heads/enterprise servers this helps reduce false alarms related to restart events
In the previous release macros were added to the majority of alerts, this should remove the requirement for the alerts to be customised locally and ensure upgrades are more straightforward

Update summary (see README for more details):
New macro splunkadmins_shutdown_list
New macro splunkadmins_shutdown_time
Updated various alerts to use the above macros

Version 1.5
Nov. 27, 2017

Updated Splunk Alert Failures alert and the Time format has changed alerts to have more clear output via email
Includes changes from 1.4 (and 1.3 which was the application icon), two new alerts LicenseMaster - Duplicated License Situation, DeploymentServer - Unsupported attribute within DS config
Simplified "Scheduled Searches without a configured earliest and latest time", and "Scheduled searches not specifying an index"
Created a macro splunkadmins_splunkd_source for Windows users or others using non-standard Splunk installation directories
Updated the README

1,516
Installs
7,708
Downloads
Share Subscribe LOGIN TO DOWNLOAD

Subscribe Share

AppInspect Tooling

Splunk AppInspect evaluates Splunk apps against a set of Splunk-defined criteria to assess the validity and security of an app package and components.

Are you a developer?

As a Splunkbase app developer, you will have access to all Splunk development resources and receive a 10GB license to build an app that will help solve use cases for customers all over the world. Splunkbase has 1000+ apps and add-ons from Splunk, our partners and our community. Find an app or add-on for most any data source and user need, or simply create your own with help from our developer portal.

Follow Us:
© 2005-2020 Splunk Inc. All rights reserved.
Splunk®, Splunk>®, Listen to Your Data®, The Engine for Machine Data®, Hunk®, Splunk Cloud™, Splunk Light™, SPL™ and Splunk MINT™ are trademarks and registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners.