Usage

This app contains six dashboards, made to compliment the Monitoring Console during maintenance activities involving data sources for pre- and post-validations.

With the exception of "Poll Instance", all dashboards are portable, allowing their source XML to be copy-and-pasted into a new dashboard on another Splunk instance.

Overview dashboard

Shows high-level trends of indexing rates, deployment clients, Splunk Stream forwarders, distinct data source counts, and average data source throughput.

Click the Indexing Rate, Events Per Second, HTTP Event Collector Data Received, Deployment Clients Phoning Home, or Stream Forwarders Phoning Home values to expand the metric to the timechart below. The Indexing Rate timechart contains distinct values over time for Indexes, Sourcetypes, Sources, and Hosts to identify changes to ongoing trends. The Deployment Clients and Stream Forwarders Phoning Home values are the distinct count for the given time span, with a minimum span to compensate for adjusted phone home intervals above default.

Click the Show Filters link on the top to expose the time picker.

Forwarder Troubleshooter dashboard

Reviews internal logs and metrics for troubleshooting common forwarder issues. Check this dashboard when performing maintenance activities on forwarders. Includes forwarder information, throughput metrics, stops and starts, index/sourcetype/source details, health status, resource usage, splunkd logs, deployment server messages, and indexer discovery messages.

Filter by a single forwarder at a time, or a batch of forwarders during maintenance or triage.

Data Source Troubleshooter dashboard

Reviews internal logs and metrics for troubleshooting common data source issues. Check this dashboard when deploying configuration changes to data sources. Includes tstats data over time, as well as detected issues with permissions, enqueuing, timestamp parsing, line breaking, aggregation, future timestamps, and time disparity.

Tick the Overlay checkbox in the Tstats Events panels to see distinct values over time for indexes, sourcetypes, sources, and/or hosts to identify changes to ongoing trends.

Tick the Sparklines checkbox in the Tstats Details panel to review individual index, sourcetype, source, and/or host trends beside the other split-by fields.

Data Model Troubleshooter dashboard

Reviews REST and tstats data for troubleshooting common data model issues, similar to the Data Source Troubleshooter but for identifying issues with a particular data model. Works best when installed on a search head which includes the underlying data models (i.e. Splunk_SA_CIM) and associated knowledge objects to identify constrained data (i.e. CIM-compliant tags).

Deployment Clients dashboard

Reviews REST API data from the Deployment Server(s), extending the information given by Forwarder Management. Includes a similar interface to Forwarder Management for monitoring multiple Deployment Servers, with many filters and extended information of both Deployment Clients and Servers.

Poll Instance dashboard

A bit unique relative to the other dashboards, the Poll Instance dashboard pulls information from a remote Splunk instance for diagnostic information, such as from an unpeered Splunk Enterprise instance or a Universal Forwarder. Use this dashboard to troubleshoot issues such as file inputs not ingesting or to validate current configuration settings.

This dashboard requires configuration before use. See the Installation section below regarding setup remote Splunk credentials.

| pollinstance command

This generator command powers the Poll Instance dashboard, and may be used in your own SPL to query REST API endpoints of Splunk Enterprise and Universal Forwarder instances which are not search peers of this instance. Returns a table similar in formatting to the "| rest" command.

Syntax:

| pollinstance splunk_host=<string> [splunk_port=<int>] account=<string> object=<string> [uri]

• splunk_host - the host/IP of the target Splunk instance to poll information from
• splunk_port - the management port of the target Splunk instance (optional, default is 8089)
• account - the account name given to the username/password combination, as saved in this app's Configuration dashboard
• object - determines what output to return from the instance; choose one of the following:
  • rest - Perform a GET operation from the given REST API endpoint (requires uri to be specified)
  • all - Returns information from all the objects listed below
  • info - System high-level info
  • settings - System high-level settings
  • messages - System messages
  • confs - Current configuration values
  • inputstatus - Input status (tailing processor, exec processor, modular inputs, TCP/UDP)
  • apps - Splunk Apps
  • data - Data inputs and outputs (TCP cooked/raw, UDP, forward servers)
  • kvstore - KV Store
  • cluster - Indexer Cluster
  • shcluster - Search Head Cluster
  • deployment - Deployment Clients
  • licenser - License Slaves
  • search - Distributed search peers
  • health - Splunkd Health
  • status - Introspection (CPU/memory/disk utilization, partitions, splunkd processes)

Examples:

Poll data from remote Splunk instance "splunk.mycorp.com" REST endpoint "/services/server/status", using credentials from account "Main_IDXC":

| pollinstance splunk_host="splunk.mycorp.com" account="Main_IDXC" object="rest" /services/server/status

Poll all objects from remote Splunk instance "10.50.1.22" on non-standard port "8091", using credentials from account "Tampa_UFs":

| pollinstance splunk_host="10.50.1.22" splunk_port=8091 account="Tampa_UFs" object="all"

Installation

Install this app on a search head, preferably on the Splunk Enterprise instance hosting the distributed Monitoring Console. App will not fully function without peering with the Indexing tier and Deployment Server.

For usage of the Data Model Troubleshooter, consider either installing this app on the search head with the defined data models (i.e. an Enterprise Security search head), or copying this dashboard to that Splunk instance.

For usage of the Poll Instance dashboard and | pollinstance command, the Splunk instance hosting this app must have the admin credentials of the remote Splunk instance's stored locally, as well as IP connectivity to the remote Splunk instance's management port (default TCP/8089). Use the associated Configuration dashboard within this app to add any remote Splunk instance's admin credentials into the encrypted storage passwords service, saved as an Account. Save one Account for each username/password combination used with the Poll Instance dashboard or | pollinstance command, then poll the remote instances as necessary during troubleshooting for information not provided by internal logs. See Splunk docs on how to modify admin credentials on your remote Splunk Enterprise and Universal Forwarder instances.

About

Data Collection Monitor
by Joe Misner
http://tools.misner.net/

Kindly submit identified bugs, comments, and suggestions to the developer contact in the sidebar. Thank you!