Search Log Processing
Admins: Please read about Splunk Enterprise 8.0 and the Python 2.7 end-of-life changes and impact on apps and upgradeshere.
Overview
For comparing indexer performance, understanding operators, or log levels (INFO .. CRIT), you have to leverage search.log. Indexing it isn't scalable though. This app will, every five minutes, launch a scripted input that will review the local dispatch directory for any search.log files, and the parse out the details into a JSON blob that will be put in index=_internal.
Details so far:
* Per Search Peer: # of Results
* Per Search Peer: Amount of time reported
* Time to set up search peers
* Count by log level of the search (e.g., INFO / WARN / ERROR / FATAL)
* Count by log operator of the search (e.g., “SearchOperator:kv” vs LMConfig)
* Time taken per operation (e.g., how many slow operations were there, how many fast ones — this can help identify particularly expensive regex / eval, etc.)
* SearchID
* Time Run
Overall, this should allow some trending information, and specifically to be able to detect an indexer that is slower than the rest.
Details so far:
* Per Search Peer: # of Results
* Per Search Peer: Amount of time reported
* Time to set up search peers
* Count by log level of the search (e.g., INFO / WARN / ERROR / FATAL)
* Count by log operator of the search (e.g., “SearchOperator:kv” vs LMConfig)
* Time taken per operation (e.g., how many slow operations were there, how many fast ones — this can help identify particularly expensive regex / eval, etc.)
* SearchID
* Time Run
Overall, this should allow some trending information, and specifically to be able to detect an indexer that is slower than the rest.
Release Notes
Version 1.0
Jan. 22, 2016
*** BETA RELEASE ***
Tested in a few environments, but not extensively and no dashboards are yet present. Look for a 2.0 release in the next few weeks with dashboards. If you run it and would like me to build dashboards for you, please email me.