icon/x Created with Sketch.

Splunk Cookie Policy

We use our own and third-party cookies to provide you with a great online experience. We also use these cookies to improve our products and services, support our marketing campaigns, and advertise to you on our website and other websites. Some cookies may continue to collect information after you have left our website. Learn more (including how to update your settings) here.
Accept Cookie Policy

Accept License Agreements

This app is provided by a third party and your right to use the app is in accordance with the license provided by that third-party licensor. Splunk is not responsible for any third-party apps and does not provide any warranty or support. If you have any questions, complaints or claims with respect to this app, please contact the licensor directly.

Thank You

Downloading Unified Forwarder Monitoring App for Splunk
SHA256 checksum (unified-forwarder-monitoring-app-for-splunk_202.tgz) b24a2eab3f82c10e555d79ae08aba8c8dcbf92f83353abbcff2b84b70fd38a7b SHA256 checksum (unified-forwarder-monitoring-app-for-splunk_112.tgz) 5030296a803d1857495999d1f47535174e30f37771207f69810a48bb487fa18c SHA256 checksum (unified-forwarder-monitoring-app-for-splunk_110.tgz) 0b157128c43fce9a2238da7bb0f832c58d5844192fa9b0e5a7443527456b3e88 SHA256 checksum (unified-forwarder-monitoring-app-for-splunk_100.tgz) 2a3ea1d2693f1e341fabef18781fb35eb92080a41be83dad4bd683a644887cc4
To install your download
For instructions specific to your download, click the Details tab after closing this window.

Flag As Inappropriate

Unified Forwarder Monitoring App for Splunk

Splunk AppInspect Passed
Admins: Please read about Splunk Enterprise 8.0 and the Python 2.7 end-of-life changes and impact on apps and upgradeshere.
The Unified Forwarder Monitoring App for Splunk (UFMA) brings all of your forwarder monitoring needs into a single app. Combining and enhancing views from both DMC Forwarder Monitoring and Deployment Server Forwarder Management, as well as adding in Stream forwarder monitoring, brings greater insight into what is on and what is going on with both forwarders and deployment servers in your environment. UFMA will help you clean up unnecessary deployment add-ons and server classes, identify problem forwarders, and help with operational day-to-day of Splunk.

Unified Forwarder Monitoring and Alerting App For Splunk (UFMA)


The UMFA App for Splunk is a forwarder monitoring app bringing together and enhancing multiple
disjointed monitoring functions already in use in Splunk. This monitoring tool will provide details on
any system that has sent an event to your indexer(s) including reporting on full instances of Splunk,
Universal Forwarders, Heavy Forwarders, and even independent Stream forwarders. It also brings deployment
server information to your forwarder monitoring. At the click of a button see volume and connection stats,
find out exactly which apps and server classes are deployed to a forwarder, or see if there are inactive
apps or server classes that may be cleaned from a deployment server. This app aims to be a one stop shop
for your forwarder monitoring needs.


Questions or feature requests may be sent to Madison.Moss@jhuapl.edu and support may depend on availability.
Support is to only include bug fixes in the case any feature or process does not work as described within
documentation. In this case I will work to resolve the issue and update the app with the fix in place. Feature
requests may be sent but are not guaranteed in any future releases of the app.


The UFMA App for Splunk is an app that provides searches and visualizations and should be installed on a
search head. The app will work in a standalone, distributed, or clustered environment. On a standalone
search head place the app in $SPLUNK_HOME/etc/apps/ and refresh Splunk
(https://<your_searchHead>:8000/en-US/debug/refresh) or install from file directly from the web GUI. In
a clustered environement place the app in $SPLUNK_HOME/etc/shcluster/apps/ on the Deployer and apply
the cluster bundle to the search head cluster (splunk apply shcluster-bundle -target <shcluster-captain>:8089).
After installation it is recommended to manually run "Rebuild Asset Table" under All Configurations to get
an initial table of forwarders.


Users of UFMA must have permissions to view _internal index and access to REST capabilities. The recommended
role is "admin". In order to enrich your data with deployment server information your deployment server(s)
must be added as search peers on the search head where UFMA will be run. This is not required for forwarder
alerting but for functionality is recommended. Follow Splunk recommended requirements for OS and hardware

App Components

Many components in the app have customized drilldowns for filtering and to provide quicker deep dives.

Forwarders - Forwarder Summary

The Forwarder Summary dashboard is a complete overview of everything forwarding in your environment. It
will provide everything from a total count of forwarders to event stats and sending volume for individual
forwarders for the time range selected. This view may help you identify your most active forwarders, if your
forwarders are load balancing properly, or if a once active forwarder is not active at all. It will also tell
you exactly where your forwarders are checking into if you use multiple deployment servers. Please note any
deployment server information will appear as "N/A" if a forwarder is not configured to use a deployment server
or the deployment server it is checking into has not been added as a search peer. Other information may appear
as "N/A" if no data has been received for the forwarder within the selected time range.

Forwarders - Forwarder Detail

The Forwarder Detail dashboard is a deep dive into a single forwarder in your environment. It will provide
forwarding statistics for a forwarder for the selected time range as well as provide deployment server details
as to exactly which apps and server classes are on the forwarder. Please note, if an app has been deployed
directly to the forwarder and not through a deployment server the app will not appear in this information. It
also provides and overview of which sourcetypes the forwarder has sent over the selected time range. New in 2.0.0
there is now a hidden panel that will appear if _introspection has been enabled for the selected forwarder. This
will provide a deep dive into how Splunk is utilizing system resources and allow you to optimize forwarder
performance. Look for things like if entire CPU cores are consumed by Splunk and see whether those resources are
going to scripted input, registry monitoring (baselining can be CPU intensive), or other Splunk actions. Details
on how to enable this additional monitoring are available on the Forwarder Resource Usage Dashboard.

Forwarders - Forwarder Resource Usage

The Forwarder Resource Usage dashboard adds even more granularity into monitoring your forwarders. It dives into
resource usage of different Splunk components and helps enable you to look for anomalous systems that Splunk
may be consuming more CPU or RAM on for whatever reason. Drill down into different systems and it will direct you
to the Forwarder Detail dashboard for more details on a specific system.

Deployment Server Summary

The Deployment Server Summary provides details on apps, server classes, and every forwarder checking into your
deployment server(s). See what apps or server classes are deployed and which ones are inactive and might be able
to be cleaned from the deployment server. Also quickly be able to tell which forwarder and app or server class
is deployed to.

All Configurations - Rebuild Asset Table

Tired of having to completely rebuild your asset table in the Monitoring Console and losing missing forwarders
that may still be important? Now you have the option to selectively remove missing forwarders from your asset
table while retaining your alerting for other missing forwarders. Select the default "All" to completely rebuild
the table or use the multiselect to pick missing forwarders to remove from the current list. Please note if a
selected forwarder has recorded activity within that timerange selected it will still appear in the table. Also,
the total forwarder number on this page may differ from that on the Forwarders - Forwarder Summary page as this
page accounts for all GUIDs which may have duplicate hostnames. The Forwarders - Forwarder Summary page will dedup
over hostname only taking the latest/active GUID.

All Configurations - Reports

A collection of all saved reports for UFMA. By default there is a single scheduled search, "UFMA - Complete Asset
List",set to run every 5 minutes to update the forwarder asset table. You may adjust these settings to your
environment's needs. Larger environments may want to reduce the frequency in which the report runs.

All Configurations - Alerts

A collection of all alerts for UFMA. By default there is a single alert, "UFMA - ALERT - Missing Forwarders",
checking for any missing forwarder set to run every 5 minutes. The alert action is currently set to list in
triggered alerts. You may update this to your needs such as email alerting. This alert corresponds with the
scheduled report "UFMA - Complete Asset List". A forwarder is considered missing if it has not connected to an
indexer in the last 15 minutes. Using this combination of search and alert you should know if a forwarder is
offline in about 20 minutes or less from the time the forwarder stops connecting. You may adjust this time by
changing the "900" seconds used in "UFMA - Complete Asset List" to alter the 15 minute missing classification and
may change the run frequency of the scheduled search and alert to reduce or increase the additional up to 5 minutes
of lag in reporting. Please do take into account the "UFMA - Complete Asset List" search can be an intensive search
and may need to be adjusted to work well depending the size of your environment. You may customize your forwarder
alerting further by filtering what you want to report on. Some examples below will help layout how to exclude
specific forwarder hostnames or forwarders checking into specific deployment servers from alerting on missing
forwarders. You may disable the default alert and create your own or modify the existing alert to insert your
environment details.

Custom Alert - Exclude Deployment Server(s)
| inputlookup ufma_asset_list
| search status="missing" deployment_server!=<deploymentServer> ... deployment_server!=<deploymentServerN>
| eval last_connected = strftime(last_connected, "%m/%d/%Y %H:%M:%S %z")
| fields hostname forwarder_type version last_connected deployment_server

Custom Alert - Exclude Hostname(s)
| inputlookup ufma_asset_list
| search status="missing" hostname!=<hostname1> hostname!=<hostname2> ... hostname!=<hostnameN>
| eval last_connected = strftime(last_connected, "%m/%d/%Y %H:%M:%S %z")
| fields hostname forwarder_type version last_connected deployment_server

All Configurations - Dashboards

A collection of all views contained in UFMA


A basic search function to use within the UFMA context


If you look at the macros contained within UFMA you will see a variety of macros used to drive the app. One macro
called "exclude_hosts" contains some dummy hosts by default but can be editted if you would like to exclude certain
hosts from being populated in your asset table. A use case for this may be where certain VMs are continuously spun
up and destroyed. Rather than having a quickly growing asset table that needs to be rebuilt to remove these system
they can be called out in this macro to exclude them from the start.

Release Notes

Version 2.0.2
Oct. 17, 2019

- Added optional forwarder resource usage statistic. This utilizes _introspection that is enabled by default on full instances of Splunk but must be enabled for Universal Forwarders. If _introspection is enabled for a forwarder it will appear in the new Forwarder Resource Usage Dashboard. Clicking on a forwarder there or viewing the forwarder using the Forwarder Details dashboard will display a hidden panel that contains more granular information on how Splunk is using CPU and RAM resources on a given system.
- Removed base search from some dynamic drop downs allowing it to be populated only by lookup table. This enables much quicker population of the dropdown, but only systems available in the ufma_asset_list will be available. This list is updated every ~5 minutes.
- 2.0.2 Adjusted /static permissions for AppInspect

Version 1.1.2
June 8, 2018

- Corrections to lookup for certification

Version 1.1.0
May 31, 2018

- Tweaked sort command to remove limits and allow for more than 10,000 forwarders.
- Note that "Forwarder Count" in Rebuild Asset Table dashboard may differ than the count in Forwarder Summary. This is because of duplicate hostnames that may be a result of universal forwarders being reinstalled on a system causing a new GUID. The orwarder Summary only shows the most recently GUID for a duplicate hostname.

Version 1.0.0
Nov. 29, 2017


Subscribe Share

AppInspect Tooling

Splunk AppInspect evaluates Splunk apps against a set of Splunk-defined criteria to assess the validity and security of an app package and components.

Are you a developer?

As a Splunkbase app developer, you will have access to all Splunk development resources and receive a 10GB license to build an app that will help solve use cases for customers all over the world. Splunkbase has 1000+ apps and add-ons from Splunk, our partners and our community. Find an app or add-on for most any data source and user need, or simply create your own with help from our developer portal.

Follow Us:
© 2005-2020 Splunk Inc. All rights reserved.
Splunk®, Splunk>®, Listen to Your Data®, The Engine for Machine Data®, Hunk®, Splunk Cloud™, Splunk Light™, SPL™ and Splunk MINT™ are trademarks and registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners.