This app contains six dashboards, made to compliment the Monitoring Console during maintenance activities involving data sources for pre- and post-validations.
With the exception of "Poll Instance", all dashboards are portable, allowing their source XML to be copy-and-pasted into a new dashboard on another Splunk instance.
Shows high-level trends of indexing rates, deployment clients, Splunk Stream forwarders, distinct data source counts, and average data source throughput.
Click the Indexing Rate, Events Per Second, HTTP Event Collector Data Received, Deployment Clients Phoning Home, or Stream Forwarders Phoning Home values to expand the metric to the timechart below. The Indexing Rate timechart contains distinct values over time for Indexes, Sourcetypes, Sources, and Hosts to identify changes to ongoing trends. The Deployment Clients and Stream Forwarders Phoning Home values are the distinct count for the given time span, with a minimum span to compensate for adjusted phone home intervals above default.
Click the Show Filters link on the top to expose the time picker.
Reviews internal logs and metrics for troubleshooting common forwarder issues. Check this dashboard when performing maintenance activities on forwarders. Includes forwarder information, throughput metrics, stops and starts, index/sourcetype/source details, health status, resource usage, splunkd logs, deployment server messages, and indexer discovery messages.
Filter by a single forwarder at a time, or a batch of forwarders during maintenance or triage.
Reviews internal logs and metrics for troubleshooting common data source issues. Check this dashboard when deploying configuration changes to data sources. Includes tstats data over time, as well as detected issues with permissions, enqueuing, timestamp parsing, line breaking, aggregation, future timestamps, and time disparity.
Tick the Overlay checkbox in the Tstats Events panels to see distinct values over time for indexes, sourcetypes, sources, and/or hosts to identify changes to ongoing trends.
Tick the Sparklines checkbox in the Tstats Details panel to review individual index, sourcetype, source, and/or host trends beside the other split-by fields.
Reviews REST and tstats data for troubleshooting common data model issues, similar to the Data Source Troubleshooter but for identifying issues with a particular data model. Works best when installed on a search head which includes the underlying data models (i.e. Splunk_SA_CIM) and associated knowledge objects to identify constrained data (i.e. CIM-compliant tags).
Reviews REST API data from the Deployment Server(s), extending the information given by Forwarder Management. Includes a similar interface to Forwarder Management for monitoring multiple Deployment Servers, with many filters and extended information of both Deployment Clients and Servers.
A bit unique relative to the other dashboards, the Poll Instance dashboard pulls information from a remote Splunk instance for diagnostic information, such as from an unpeered Splunk Enterprise instance or a Universal Forwarder. Use this dashboard to troubleshoot issues such as file inputs not ingesting or to validate current configuration settings.
This dashboard requires configuration before use. See the Installation section below regarding setup remote Splunk credentials.
This generator command powers the Poll Instance dashboard, and may be used in your own SPL to query REST API endpoints of Splunk Enterprise and Universal Forwarder instances which are not search peers of this instance. Returns a table similar in formatting to the "| rest" command.
Syntax:
| pollinstance splunk_host=<string> [splunk_port=<int>] account=<string> object=<string> [uri]
splunk_host - the host/IP of the target Splunk instance to poll information fromsplunk_port - the management port of the target Splunk instance (optional, default is 8089)account - the account name given to the username/password combination, as saved in this app's Configuration dashboardobject - determines what output to return from the instance; choose one of the following:rest - Perform a GET operation from the given REST API endpoint (requires uri to be specified)all - Returns information from all the objects listed belowinfo - System high-level infosettings - System high-level settingsmessages - System messagesconfs - Current configuration valuesinputstatus - Input status (tailing processor, exec processor, modular inputs, TCP/UDP)apps - Splunk Appsdata - Data inputs and outputs (TCP cooked/raw, UDP, forward servers)kvstore - KV Storecluster - Indexer Clustershcluster - Search Head Clusterdeployment - Deployment Clientslicenser - License Slavessearch - Distributed search peershealth - Splunkd Healthstatus - Introspection (CPU/memory/disk utilization, partitions, splunkd processes)Examples:
Poll data from remote Splunk instance "splunk.mycorp.com" REST endpoint "/services/server/status", using credentials from account "Main_IDXC":
| pollinstance splunk_host="splunk.mycorp.com" account="Main_IDXC" object="rest" /services/server/status
Poll all objects from remote Splunk instance "10.50.1.22" on non-standard port "8091", using credentials from account "Tampa_UFs":
| pollinstance splunk_host="10.50.1.22" splunk_port=8091 account="Tampa_UFs" object="all"
Install this app on a search head, preferably on the Splunk Enterprise instance hosting the distributed Monitoring Console. App will not fully function without peering with the Indexing tier and Deployment Server.
For usage of the Data Model Troubleshooter, consider either installing this app on the search head with the defined data models (i.e. an Enterprise Security search head), or copying this dashboard to that Splunk instance.
For usage of the Poll Instance dashboard and | pollinstance command, the Splunk instance hosting this app must have the admin credentials of the remote Splunk instance's stored locally, as well as IP connectivity to the remote Splunk instance's management port (default TCP/8089). Use the associated Configuration dashboard within this app to add any remote Splunk instance's admin credentials into the encrypted storage passwords service, saved as an Account. Save one Account for each username/password combination used with the Poll Instance dashboard or | pollinstance command, then poll the remote instances as necessary during troubleshooting for information not provided by internal logs. See Splunk docs on how to modify admin credentials on your remote Splunk Enterprise and Universal Forwarder instances.
Data Collection Monitor
by Joe Misner
http://tools.misner.net/
Kindly submit identified bugs, comments, and suggestions to the developer contact in the sidebar. Thank you!
New Features:
- Added HTTP Event Collector and Stream Forwarder metrics to the Overview dashboard
Fixes:
- Indexing Rate metric overlays on the Overview dashboard are now displaying all distinct counts
New Features:
- "| pollinstance" generator command, to poll a remote Splunk instance, such as an unpeered Splunk Enterprise instance or a Universal Forwarder, for diagnostic information
- "Poll Instance" dashboard, which utilizes "| pollinstance" to retrieve splunkd health, messages, instance information, apps, resource usage, input status, and configurations
- "Overview" dashboard, to review high-level trends of indexing rates, deployment clients, distinct data source counts, and average data source throughput
- "Data Model Troubleshooter" dashboard, to review internal logs and metrics related to data models and accelerations
- Sparkline option for Tstats Details panels on troubleshooter dashboards, to help visualize individual data sources over time
Fixes:
- Numerous additions, changes, and fixes to the Forwarder Troubleshooter and Data Source Troubleshooter dashboards
Initial release
Splunk AppInspect evaluates Splunk apps against a set of Splunk-defined criteria to assess the validity and security of an app package and components.
As a Splunkbase app developer, you will have access to all Splunk development resources and receive a 10GB license to build an app that will help solve use cases for customers all over the world. Splunkbase has 1000+ apps and add-ons from Splunk, our partners and our community. Find an app or add-on for most any data source and user need, or simply create your own with help from our developer portal.