This app contains nine dashboards, made to compliment the Monitoring Console during maintenance activities involving data sources and the Forwarding tier for pre- and post-validations.
With the exception of the "Poll Instance" and "Data Source Monitoring" dashboards, all dashboards are portable, allowing their source XML to be copy-and-pasted into a new dashboard on another Splunk instance. (To make a dashboard portable, search and replace the dcm_internal_index macro with index=_internal, and replace the dcm_audit_index macro with index=_audit)
Shows high-level trends of indexing rates, deployment clients, Splunk Stream forwarders, distinct data source counts, and average data source throughput.
Click the Indexing Rate, Events Per Second, HTTP Event Collector Data Received, Forwarding Splunk Instances, Deployment Clients Phoning Home, Stream Forwarders Phoning Home, Instances Reporting Red/Yellow Data Forwarding Health, or SC4S Sources Forwarding Data values to expand the metric to the timechart below. The Events Per Second timechart contains distinct values over time for Indexes, Sourcetypes, Sources, and Hosts to identify changes to ongoing trends. The Deployment Clients and Stream Forwarders Phoning Home values are the distinct count for the given time span, with a minimum span to compensate for adjusted phone home intervals above default.
Click the Show Filters link on the top to expose the time picker, and to display indexers by instance for smaller deployments, or by site for larger deployments.
Reviews internal logs and metrics for troubleshooting common forwarder issues. Check this dashboard when performing maintenance activities on forwarders. Includes forwarder information, throughput metrics, stops and starts, index/sourcetype/source details, health status, resource usage, splunkd logs, deployment server messages, and indexer discovery messages.
Although this dashboard was made primarily for troubleshooting use cases around forwarding, it may also be used to investigate many issues involving both Splunk Enterprise and Universal Forwarder instances. This is as long as their internal logs are forwarding to your indexers, to allow this app to review for issues.
Filter by a single forwarder (Splunk instance) at a time, or a batch of forwarders during maintenance or triage.
Reviews internal logs and metrics for troubleshooting common data source issues. Check this dashboard when deploying configuration changes to data sources. Includes tstats data over time, as well as detected issues with permissions, enqueuing, timestamp parsing, line breaking, aggregation, future timestamps, and time disparity.
Overlays are added to Tstats Events panels to see distinct values over time for indexes, sourcetypes, sources, and hosts, to help identify changes in ongoing trends.
Tick the Sparklines checkbox in the Tstats Details panel to review individual index, sourcetype, source, and/or host trends beside the other split-by fields. Adding these sparklines may cause these dashboard searches to consume lots of memory -- make sure the Splunk Enterprise instance running this app has a healthy amount of memory available or memory tracker configured properly.
Reviews REST and tstats data for troubleshooting common data model issues, similar to the Data Source Troubleshooter but for identifying issues with a particular data model. Works best when installed on a search head which includes the underlying data models (i.e. Splunk_SA_CIM) and associated knowledge objects to identify constrained data (i.e. CIM-compliant tags). For example, on the search head hosting the Splunk Enterprise Security (ES) app.
Reviews events by sourcetype and host over time, optionally monitoring and alerting on anomalies such as missing data. Use this dashboard to review a particular data source for issues such as spikes in data ingestion or outages.
Optionally, visit the Data Source Monitor Configuration dashboard to input sourcetypes and hosts for active monitoring. This feature requires installation of the Machine Learning Toolkit (MLTK), alerting on anomalous values by using the DensityFunction algorithm. See the Configuration section for details.
Reviews a particular data source (either a sourcetype or a host) for anomalous values. This dashboard works alongside the Data Source Monitoring dashboard during troubleshooting.
Reviews Splunk Connect for Syslog (SC4S) indexed fields and metrics for investigating common syslog ingestion issues. Use this dashboard if you have SC4S configured in your deployment, and need to troubleshoot issues related to SC4S syslog collection.
To review data in the metrics panels, the SC4S environment variable "SC4S_DEST_SPLUNK_SC4S_METRICS_HEC" should be properly configured to forward metrics data. See the SC4S documentation for the Splunk data source.
Reviews REST API data from the Deployment Server(s), extending the information given by Forwarder Management. Includes a similar interface to Forwarder Management, but with the addition of monitoring multiple Deployment Servers, and many filters and extended information of both Deployment Clients and Servers.
This feature is for use wherever the Data Collection Monitor app is installed on-premise, such as on the Monitoring Console or Deployment Server, per Installation instructions. It will not work on the Splunk Cloud Search Head.
A bit unique relative to the other dashboards, the Poll Instance dashboard pulls information from a remote Splunk instance for diagnostic information, such as from an unpeered Splunk Enterprise instance or a Universal Forwarder. Use this dashboard to troubleshoot issues such as file inputs not ingesting or to validate current configuration settings.
This dashboard requires configuration before use. See the Installation section below regarding setup remote Splunk credentials.
This feature is for use wherever the Data Collection Monitor app is installed on-premise, such as on the Monitoring Console or Deployment Server, per Installation instructions. It will not work on the Splunk Cloud Search Head.
This generator command powers the Poll Instance dashboard, and may be used in your own SPL to query REST API endpoints of Splunk Enterprise and Universal Forwarder instances. This is most useful for instances which are not search peers of this instance, which is a requirement of the "| rest" command. To help with compatibility, "| pollinstance" also returns a table similar in formatting to "| rest".
Syntax:
| pollinstance splunk_host=<string> [splunk_port=<int>] account=<string> object=<string> [uri]
splunk_host - the host/IP of the target Splunk instance to poll information fromsplunk_port - the management port of the target Splunk instance (optional, default is 8089)account - the account name given to the username/password combination, as saved in this app's Configuration dashboardobject - determines what output to return from the instance; choose one of the following:rest - Perform a GET operation from the given REST API endpoint (requires uri to be specified)all - Returns information from all the objects listed belowinfo - System high-level infosettings - System high-level settingsmessages - System messagesconfs - Current configuration valuesinputstatus - Input status (tailing processor, exec processor, modular inputs, TCP/UDP)apps - Splunk Appsdata - Data inputs and outputs (TCP cooked/raw, UDP, forward servers)kvstore - KV Storecluster - Indexer Clustershcluster - Search Head Clusterdeployment - Deployment Clientslicenser - License Slavessearch - Distributed search peershealth - Splunkd Healthstatus - Introspection (CPU/memory/disk utilization, partitions, splunkd processes)Examples:
Poll data from remote Splunk instance "splunk.mycorp.com" REST endpoint "/services/server/status", using credentials from account "Main_IDXC":
| pollinstance splunk_host="splunk.mycorp.com" account="Main_IDXC" object="rest" /services/server/status
Poll all objects from remote Splunk instance "10.50.1.22" on non-standard port "8091", using credentials from account "Tampa_UFs":
| pollinstance splunk_host="10.50.1.22" splunk_port=8091 account="Tampa_UFs" object="all"
In a Splunk On-Premise Deployment, install this app on a search head, preferably on the Splunk Enterprise instance hosting the Monitoring Console configured in distributed mode. App will not fully function without peering with the Indexing tier and Deployment Server.
In a Splunk Cloud Deployment, install this app both on the Splunk Cloud search head, as well as on the on-premise Deployment Server. The "Deployment Client" and "Poll Instance" dashboards located under the "On-Premises Dashboards" menu within the app must be used from the on-premise Deployment Server web GUI, while all other dashboards work from within the Splunk Cloud web GUI.
If using an index other than the default _internal to collect Splunkd logs and _audit to collect Splunk audit logs, set the dcm_internal_index and dcm_internal_audit macros appropriately.
For usage of the Data Model Troubleshooter dashboard, consider either also installing this app on the search head with the defined data models (i.e. a Splunk Enterprise Security search head), or copying this particular dashboard to that Splunk instance.
For usage of the Data Source Monitoring dashboard's alerting functionality, this app requires installation of the Machine Learning Toolkit (MLTK) and Python for Scientific Computing (Cloud or Linux, Windows) apps from Splunkbase, in order to check for anomalous event ingestion rates by data source (sourcetype or host). After installing these apps, navigate in the Data Collection Monitor app to Configuration > Data Source Monitor Configuration to tell Splunk which sourcetypes and hosts to monitor for anomalies. These monitored sourcetypes and hosts accept wildcards (*), and may also be configured from this app's appropriately named lookup files.
You have a choice of configuring a monitor for 1hr or 24hr monitoring, which corresponds to the interval which MLTK will be training it's DensityFunction model, as well as how often Splunk checks for related anomalies. Typically, using 1hr for more noisy logs and 24hr for more sporadic logs works best. After configuration, navigate to Search > Reports, and manually execute any "Baseline Model Gen" searches where alerts were configured (i.e. if a sourcetype was configured for 24hr monitoring, execute the search "Events By Sourcetype Per Day - Baseline Model Gen"). This will initially train the MLTK model using 60 days of prior data, after which the "Model Gen" searches will run on a schedule to periodically update the model in the future.
Be default, alerts are sent to Splunk's Triggered Alerts, and are also displayed on the bottom of the Data Source Monitoring dashboard. To forward alerts to another location, such as to an e-mail address or to a case management solution such as Alert Manager, edit the Anomaly in Events By X alerts within this app by navigating to Search > Alerts > Edit Alert.
This feature is for use wherever the Data Collection Monitor app is installed on-premise, such as on the Monitoring Console or Deployment Server, per Installation instructions. It will not work on the Splunk Cloud Search Head.
For usage of the Poll Instance dashboard and | pollinstance command, the Splunk instance hosting this app must have the admin credentials of the remote Splunk instance stored locally, as well as IP connectivity to the remote Splunk instance's management port (default TCP/8089). Use the associated Configuration dashboard within this app to add the remote Splunk instance's admin credentials into the encrypted storage passwords service, saved as an Account. Save one Account for each username/password combination used with the Poll Instance dashboard or | pollinstance command, then poll the remote instances as necessary (useful during troubleshooting for information not provided by internal logs). See Splunk docs on how to modify admin credentials on your remote Splunk Enterprise and Universal Forwarder instances.
Data Collection Monitor
by Joe Misner
http://tools.misner.net/
Kindly submit identified bugs, comments, and suggestions to the developer contact in the sidebar. Thank you!
New Features:
- Created new Data Source Monitor and Data Source Outliers dashboards, for monitoring data sources for anomalous event counts (spikes or drops) using MLTK, and alerting when an issue was found
- Added new Time Disparity panels to the Forwarder Troubleshooter dashboard
- Added SC4S Status Messages panel to the SC4S Metrics dashboard
Fixes:
- Data Collect Overview dashboard:
--- Default view for indexers is now by Instance instead of by Site
--- Changed order of single-value panels
--- Changed language from "Red/Yellow" to "Poor"
--- Corrected EPS panel to align with index-time instead of event-time
--- Changed indexer site detection logic, now using internal logs instead of REST calls for Splunk Cloud compatibility
--- Updated forwarder health drilldown panels
- Forwarder Troubleshooter dashboard:
--- Fixed the starting and stopping panel's SPL logic to work with recent versions of Splunk
--- Fixed the Forwarder Info panel SPL logic to properly display when no forwarders are found
App Updates:
- Splunk SDK for Python updated to v1.6.16
- Project updated to Splunk Add-on Builder v4.0.0
New Features:
- Created new SC4S Metrics dashboard, for Splunk deployments using Splunk Connect for Syslog
- Data Collection Overview dashboard:
--- Graph for Splunkd Health on Data Forwarding
--- Graph for SC4S
--- Option to show indexers by site
--- New panels below select categories
- Forwarder Troubleshooter dashboard:
--- Added radio button to switch between Clustered Logs and Individual Logs to optionally show event frequency by each individual host
--- Added log_level filter to Clustered Messages
- Data Source Troubleshooter dashboard:
--- Added Span dropdown to tables
- Data Model Troubleshooter dashboard:
--- Updated views for accelerated versus non-accelerated data
- Poll Instance dashboard:
--- Created hyperlink from Host field value to the Forwarder Troubleshooter dashboard, with input value populated
Many other additions, bug fixes, name changes, and spelling adjustments
New Features:
- Added HTTP Event Collector and Stream Forwarder metrics to the Overview dashboard
Fixes:
- Indexing Rate metric overlays on the Overview dashboard are now displaying all distinct counts
New Features:
- "| pollinstance" generator command, to poll a remote Splunk instance, such as an unpeered Splunk Enterprise instance or a Universal Forwarder, for diagnostic information
- "Poll Instance" dashboard, which utilizes "| pollinstance" to retrieve splunkd health, messages, instance information, apps, resource usage, input status, and configurations
- "Overview" dashboard, to review high-level trends of indexing rates, deployment clients, distinct data source counts, and average data source throughput
- "Data Model Troubleshooter" dashboard, to review internal logs and metrics related to data models and accelerations
- Sparkline option for Tstats Details panels on troubleshooter dashboards, to help visualize individual data sources over time
Fixes:
- Numerous additions, changes, and fixes to the Forwarder Troubleshooter and Data Source Troubleshooter dashboards
Initial release
As a Splunkbase app developer, you will have access to all Splunk development resources and receive a 10GB license to build an app that will help solve use cases for customers all over the world. Splunkbase has 1000+ apps from Splunk, our partners and our community. Find an app for most any data source and user need, or simply create your own with help from our developer portal.