icon/x Created with Sketch.

Splunk Cookie Policy

We use our own and third-party cookies to provide you with a great online experience. We also use these cookies to improve our products and services, support our marketing campaigns, and advertise to you on our website and other websites. Some cookies may continue to collect information after you have left our website. Learn more (including how to update your settings) here.
Accept Cookie Policy

We are working on something new...

A Fresh New Splunkbase
We are designing a New Splunkbase to improve search and discoverability of apps. Check out our new and improved features like Categories and Collections. New Splunkbase is currently in preview mode, as it is under active development. We welcome you to navigate New Splunkbase and give us feedback.
Log4Shell Vulnerability: Information and guidance for you. Get resources.

Accept License Agreements

This app is provided by a third party and your right to use the app is in accordance with the license provided by that third-party licensor. Splunk is not responsible for any third-party apps and does not provide any warranty or support. If you have any questions, complaints or claims with respect to this app, please contact the licensor directly.

Thank You

Downloading Data Collection Monitor
SHA256 checksum (data-collection-monitor_040.tgz) bd1e0bbbed9d543a2181980a9b1640040a1cb01e374df57c9d02d5ffc0be2d7c SHA256 checksum (data-collection-monitor_030.tgz) 53f92596649edef6ced18c7b61e7317f7cdcc7279bc2318d9cbfde8d43a85a9d SHA256 checksum (data-collection-monitor_021.tgz) 2f309f6ced46c2c7e2b1793307d9cb5db98e012b630d904389f969a9e2ea0d05 SHA256 checksum (data-collection-monitor_020.tgz) 1bf1876a1bc8300d3d3e77177deb78bb8859a59408b48e76f508a80e6ef3b05b SHA256 checksum (data-collection-monitor_010.tgz) d9ea63521af4d30e7c46eedcbed1ed7bc7ef7ea0f415a59e7b7cc6e0b3df1beb
To install your download
For instructions specific to your download, click the Details tab after closing this window.

Flag As Inappropriate

splunk

Data Collection Monitor

Splunk Cloud
Overview
Details
Data Collection Monitor works alongside the Monitoring Console, giving greater visibility into the current status of data collection from Splunk Universal Forwarders, Heavy Forwarders, and other data sources.

This app is a utility for Splunk administrators, which contains dashboards for performing validations before and after maintenance activities related to work performed at the data collection layer.

NOTE 10/7/21 -- If using version 3.0.0+ of the optional dependency Python for Scientific Computing used with the new Data Source Monitoring dashboard, make sure you are using version 5.3.0+ of Machine Learning Toolkit to avoid issues with model generation.

Usage

This app contains nine dashboards, made to compliment the Monitoring Console during maintenance activities involving data sources and the Forwarding tier for pre- and post-validations.

With the exception of the "Poll Instance" and "Data Source Monitoring" dashboards, all dashboards are portable, allowing their source XML to be copy-and-pasted into a new dashboard on another Splunk instance. (To make a dashboard portable, search and replace the dcm_internal_index macro with index=_internal, and replace the dcm_audit_index macro with index=_audit)

Data Collection Overview dashboard

Shows high-level trends of indexing rates, deployment clients, Splunk Stream forwarders, distinct data source counts, and average data source throughput.

Click the Indexing Rate, Events Per Second, HTTP Event Collector Data Received, Forwarding Splunk Instances, Deployment Clients Phoning Home, Stream Forwarders Phoning Home, Instances Reporting Red/Yellow Data Forwarding Health, or SC4S Sources Forwarding Data values to expand the metric to the timechart below. The Events Per Second timechart contains distinct values over time for Indexes, Sourcetypes, Sources, and Hosts to identify changes to ongoing trends. The Deployment Clients and Stream Forwarders Phoning Home values are the distinct count for the given time span, with a minimum span to compensate for adjusted phone home intervals above default.

Click the Show Filters link on the top to expose the time picker, and to display indexers by instance for smaller deployments, or by site for larger deployments.

Forwarder Troubleshooter dashboard

Reviews internal logs and metrics for troubleshooting common forwarder issues. Check this dashboard when performing maintenance activities on forwarders. Includes forwarder information, throughput metrics, stops and starts, index/sourcetype/source details, health status, resource usage, splunkd logs, deployment server messages, and indexer discovery messages.

Although this dashboard was made primarily for troubleshooting use cases around forwarding, it may also be used to investigate many issues involving both Splunk Enterprise and Universal Forwarder instances. This is as long as their internal logs are forwarding to your indexers, to allow this app to review for issues.

Filter by a single forwarder (Splunk instance) at a time, or a batch of forwarders during maintenance or triage.

Data Source Troubleshooter dashboard

Reviews internal logs and metrics for troubleshooting common data source issues. Check this dashboard when deploying configuration changes to data sources. Includes tstats data over time, as well as detected issues with permissions, enqueuing, timestamp parsing, line breaking, aggregation, future timestamps, and time disparity.

Overlays are added to Tstats Events panels to see distinct values over time for indexes, sourcetypes, sources, and hosts, to help identify changes in ongoing trends.

Tick the Sparklines checkbox in the Tstats Details panel to review individual index, sourcetype, source, and/or host trends beside the other split-by fields. Adding these sparklines may cause these dashboard searches to consume lots of memory -- make sure the Splunk Enterprise instance running this app has a healthy amount of memory available or memory tracker configured properly.

Data Model Troubleshooter dashboard

Reviews REST and tstats data for troubleshooting common data model issues, similar to the Data Source Troubleshooter but for identifying issues with a particular data model. Works best when installed on a search head which includes the underlying data models (i.e. Splunk_SA_CIM) and associated knowledge objects to identify constrained data (i.e. CIM-compliant tags). For example, on the search head hosting the Splunk Enterprise Security (ES) app.

Data Source Monitoring

Reviews events by sourcetype and host over time, optionally monitoring and alerting on anomalies such as missing data. Use this dashboard to review a particular data source for issues such as spikes in data ingestion or outages.

Optionally, visit the Data Source Monitor Configuration dashboard to input sourcetypes and hosts for active monitoring. This feature requires installation of the Machine Learning Toolkit (MLTK), alerting on anomalous values by using the DensityFunction algorithm. See the Configuration section for details.

Data Source Outliers

Reviews a particular data source (either a sourcetype or a host) for anomalous values. This dashboard works alongside the Data Source Monitoring dashboard during troubleshooting.

SC4S Metrics dashboard

Reviews Splunk Connect for Syslog (SC4S) indexed fields and metrics for investigating common syslog ingestion issues. Use this dashboard if you have SC4S configured in your deployment, and need to troubleshoot issues related to SC4S syslog collection.

To review data in the metrics panels, the SC4S environment variable "SC4S_DEST_SPLUNK_SC4S_METRICS_HEC" should be properly configured to forward metrics data. See the SC4S documentation for the Splunk data source.

Deployment Clients dashboard

Reviews REST API data from the Deployment Server(s), extending the information given by Forwarder Management. Includes a similar interface to Forwarder Management, but with the addition of monitoring multiple Deployment Servers, and many filters and extended information of both Deployment Clients and Servers.

This feature is for use wherever the Data Collection Monitor app is installed on-premise, such as on the Monitoring Console or Deployment Server, per Installation instructions. It will not work on the Splunk Cloud Search Head.

Poll Instance dashboard

A bit unique relative to the other dashboards, the Poll Instance dashboard pulls information from a remote Splunk instance for diagnostic information, such as from an unpeered Splunk Enterprise instance or a Universal Forwarder. Use this dashboard to troubleshoot issues such as file inputs not ingesting or to validate current configuration settings.

This dashboard requires configuration before use. See the Installation section below regarding setup remote Splunk credentials.

This feature is for use wherever the Data Collection Monitor app is installed on-premise, such as on the Monitoring Console or Deployment Server, per Installation instructions. It will not work on the Splunk Cloud Search Head.

| pollinstance command

This generator command powers the Poll Instance dashboard, and may be used in your own SPL to query REST API endpoints of Splunk Enterprise and Universal Forwarder instances. This is most useful for instances which are not search peers of this instance, which is a requirement of the "| rest" command. To help with compatibility, "| pollinstance" also returns a table similar in formatting to "| rest".

Syntax:

| pollinstance splunk_host=<string> [splunk_port=<int>] account=<string> object=<string> [uri]
  • splunk_host - the host/IP of the target Splunk instance to poll information from
  • splunk_port - the management port of the target Splunk instance (optional, default is 8089)
  • account - the account name given to the username/password combination, as saved in this app's Configuration dashboard
  • object - determines what output to return from the instance; choose one of the following:
    • rest - Perform a GET operation from the given REST API endpoint (requires uri to be specified)
    • all - Returns information from all the objects listed below
    • info - System high-level info
    • settings - System high-level settings
    • messages - System messages
    • confs - Current configuration values
    • inputstatus - Input status (tailing processor, exec processor, modular inputs, TCP/UDP)
    • apps - Splunk Apps
    • data - Data inputs and outputs (TCP cooked/raw, UDP, forward servers)
    • kvstore - KV Store
    • cluster - Indexer Cluster
    • shcluster - Search Head Cluster
    • deployment - Deployment Clients
    • licenser - License Slaves
    • search - Distributed search peers
    • health - Splunkd Health
    • status - Introspection (CPU/memory/disk utilization, partitions, splunkd processes)

Examples:

Poll data from remote Splunk instance "splunk.mycorp.com" REST endpoint "/services/server/status", using credentials from account "Main_IDXC":

| pollinstance splunk_host="splunk.mycorp.com" account="Main_IDXC" object="rest" /services/server/status

Poll all objects from remote Splunk instance "10.50.1.22" on non-standard port "8091", using credentials from account "Tampa_UFs":

| pollinstance splunk_host="10.50.1.22" splunk_port=8091 account="Tampa_UFs" object="all"

Installation

In a Splunk On-Premise Deployment, install this app on a search head, preferably on the Splunk Enterprise instance hosting the Monitoring Console configured in distributed mode. App will not fully function without peering with the Indexing tier and Deployment Server.

In a Splunk Cloud Deployment, install this app both on the Splunk Cloud search head, as well as on the on-premise Deployment Server. The "Deployment Client" and "Poll Instance" dashboards located under the "On-Premises Dashboards" menu within the app must be used from the on-premise Deployment Server web GUI, while all other dashboards work from within the Splunk Cloud web GUI.

Configuration

If using an index other than the default _internal to collect Splunkd logs and _audit to collect Splunk audit logs, set the dcm_internal_index and dcm_internal_audit macros appropriately.

Data Source Troubleshooter

For usage of the Data Model Troubleshooter dashboard, consider either also installing this app on the search head with the defined data models (i.e. a Splunk Enterprise Security search head), or copying this particular dashboard to that Splunk instance.

Data Source Monitoring

For usage of the Data Source Monitoring dashboard's alerting functionality, this app requires installation of the Machine Learning Toolkit (MLTK) and Python for Scientific Computing (Cloud or Linux, Windows) apps from Splunkbase, in order to check for anomalous event ingestion rates by data source (sourcetype or host). After installing these apps, navigate in the Data Collection Monitor app to Configuration > Data Source Monitor Configuration to tell Splunk which sourcetypes and hosts to monitor for anomalies. These monitored sourcetypes and hosts accept wildcards (*), and may also be configured from this app's appropriately named lookup files.

You have a choice of configuring a monitor for 1hr or 24hr monitoring, which corresponds to the interval which MLTK will be training it's DensityFunction model, as well as how often Splunk checks for related anomalies. Typically, using 1hr for more noisy logs and 24hr for more sporadic logs works best. After configuration, navigate to Search > Reports, and manually execute any "Baseline Model Gen" searches where alerts were configured (i.e. if a sourcetype was configured for 24hr monitoring, execute the search "Events By Sourcetype Per Day - Baseline Model Gen"). This will initially train the MLTK model using 60 days of prior data, after which the "Model Gen" searches will run on a schedule to periodically update the model in the future.

Be default, alerts are sent to Splunk's Triggered Alerts, and are also displayed on the bottom of the Data Source Monitoring dashboard. To forward alerts to another location, such as to an e-mail address or to a case management solution such as Alert Manager, edit the Anomaly in Events By X alerts within this app by navigating to Search > Alerts > Edit Alert.

Poll Instance

This feature is for use wherever the Data Collection Monitor app is installed on-premise, such as on the Monitoring Console or Deployment Server, per Installation instructions. It will not work on the Splunk Cloud Search Head.

For usage of the Poll Instance dashboard and | pollinstance command, the Splunk instance hosting this app must have the admin credentials of the remote Splunk instance stored locally, as well as IP connectivity to the remote Splunk instance's management port (default TCP/8089). Use the associated Configuration dashboard within this app to add the remote Splunk instance's admin credentials into the encrypted storage passwords service, saved as an Account. Save one Account for each username/password combination used with the Poll Instance dashboard or | pollinstance command, then poll the remote instances as necessary (useful during troubleshooting for information not provided by internal logs). See Splunk docs on how to modify admin credentials on your remote Splunk Enterprise and Universal Forwarder instances.

About

Data Collection Monitor
by Joe Misner
http://tools.misner.net/

Kindly submit identified bugs, comments, and suggestions to the developer contact in the sidebar. Thank you!

Release Notes

Version 0.4.0
Oct. 4, 2021

New Features:
- Created new Data Source Monitor and Data Source Outliers dashboards, for monitoring data sources for anomalous event counts (spikes or drops) using MLTK, and alerting when an issue was found
- Added new Time Disparity panels to the Forwarder Troubleshooter dashboard
- Added SC4S Status Messages panel to the SC4S Metrics dashboard

Fixes:
- Data Collect Overview dashboard:
--- Default view for indexers is now by Instance instead of by Site
--- Changed order of single-value panels
--- Changed language from "Red/Yellow" to "Poor"
--- Corrected EPS panel to align with index-time instead of event-time
--- Changed indexer site detection logic, now using internal logs instead of REST calls for Splunk Cloud compatibility
--- Updated forwarder health drilldown panels
- Forwarder Troubleshooter dashboard:
--- Fixed the starting and stopping panel's SPL logic to work with recent versions of Splunk
--- Fixed the Forwarder Info panel SPL logic to properly display when no forwarders are found

Version 0.3.0
Aug. 5, 2021

App Updates:
- Splunk SDK for Python updated to v1.6.16
- Project updated to Splunk Add-on Builder v4.0.0

New Features:
- Created new SC4S Metrics dashboard, for Splunk deployments using Splunk Connect for Syslog
- Data Collection Overview dashboard:
--- Graph for Splunkd Health on Data Forwarding
--- Graph for SC4S
--- Option to show indexers by site
--- New panels below select categories
- Forwarder Troubleshooter dashboard:
--- Added radio button to switch between Clustered Logs and Individual Logs to optionally show event frequency by each individual host
--- Added log_level filter to Clustered Messages
- Data Source Troubleshooter dashboard:
--- Added Span dropdown to tables
- Data Model Troubleshooter dashboard:
--- Updated views for accelerated versus non-accelerated data
- Poll Instance dashboard:
--- Created hyperlink from Host field value to the Forwarder Troubleshooter dashboard, with input value populated

Many other additions, bug fixes, name changes, and spelling adjustments

Version 0.2.1
Sept. 11, 2020

New Features:
- Added HTTP Event Collector and Stream Forwarder metrics to the Overview dashboard

Fixes:
- Indexing Rate metric overlays on the Overview dashboard are now displaying all distinct counts

Version 0.2.0
Sept. 2, 2020

New Features:
- "| pollinstance" generator command, to poll a remote Splunk instance, such as an unpeered Splunk Enterprise instance or a Universal Forwarder, for diagnostic information
- "Poll Instance" dashboard, which utilizes "| pollinstance" to retrieve splunkd health, messages, instance information, apps, resource usage, input status, and configurations
- "Overview" dashboard, to review high-level trends of indexing rates, deployment clients, distinct data source counts, and average data source throughput
- "Data Model Troubleshooter" dashboard, to review internal logs and metrics related to data models and accelerations
- Sparkline option for Tstats Details panels on troubleshooter dashboards, to help visualize individual data sources over time

Fixes:
- Numerous additions, changes, and fixes to the Forwarder Troubleshooter and Data Source Troubleshooter dashboards

Version 0.1.0
June 11, 2020

Initial release


Subscribe Share

Are you a developer?

As a Splunkbase app developer, you will have access to all Splunk development resources and receive a 10GB license to build an app that will help solve use cases for customers all over the world. Splunkbase has 1000+ apps from Splunk, our partners and our community. Find an app for most any data source and user need, or simply create your own with help from our developer portal.

Follow Us:
Splunk, Splunk>,Turn Data Into Doing, Data-to-Everything, and D2E are trademarks or registered trademarks of Splunk Inc. in the United States and other countries. All other brand names,product names,or trademarks belong to their respective owners.