Broken Hosts App for Splunk
Overview
Details
If the arrival of the final log for the index/sourcetype/host combination is later than expected, the Broken Hosts App will send an alert. This allows for quick status detection of the hosts and fast issue resolution.
The Broken Hosts App for Splunk is the app for monitoring missing data in Splunk. The app’s three main objectives include:
1. Alerting when data is missing from Splunk in order to determine the cause.
2. Utilizing saved searches to facilitate rapid detection of the missing data.
3. Creating dashboards for visualization to help with further investigations.
* Latest documentation can be found here:
https://brokenhosts.hurricanelabs.com
Latest documentation can be found here:
Release Notes
Version 4.0.4
- updated bh_stats_gen search to fix a bug that might cause false positives
- set eventtypes to be local to the app instead of global
Version 4.0.3
- updated AutoSort to allow for arbitrary fields
- update investigation panel to show a timechart of splunk restarts
- fixed type in app.conf that was preventing successful vetting
Version 4.0.2
- Fixed a bug with Google Chrome 70+
- Revamped architecture
-- Decouple stats generation from alert generation
-- Eventtype-based aggregations and suppressions
- Additional investigation dashboards
- KV Store auto-sort functionality (enabled by default) to prevent false positive matches
Version 3.3.6
v3.3.6
- Row reordering feature added to 'Configure Broken Hosts Lookup' page. Can drag rows using the 'Comments' column.
- 'Add New Suppression' button added to top right to make more visible.
- Ability to Copy formatted row data to clipboard
- Added expectedTime_tmp for backup purposes.
- In edge cases where KV Store is being updated after a row-reorder on Configure page and user refreshes, KV Store data could be lost. For this reason, every change made backs up the current version to a expectedTime_tmp KV Store first
- On initial load of the table it will check if expectedTime is empty, if it is it will then check expectedTime_tmp for data and use that as a backup in case the KV Store was emptied. If both are empty then it is assumed this is a new install and the user has an option to add default values to the KV Store.
Version 3.3.5
v3.3.5
- updated the savedsearch to account for sourcetype rewrites
KNOWN ISSUE:
Since kvstore doesn't allow reordering, use this process if a line needs to be moved:
1) go to the broken hosts search and run:
| inputlookup expectedTime | outputlookup expectedTime.csv
(this will populate the CSV with contents of the kvstore)
2) go to the lookup editor app:
- open the expectedTime.csv
- reorder the rows as necessary
- remove _key column
3) go to the splunk search window and run:
| inputlookup expectedTime.csv | outputlookup expectedTime
IMPORTANT: Update Instructions:
- v3.3.3 and greater uses a KV Store instead of a lookup file. Once the app is updated, you will need to populate the KV Store.
- This will only need to be done one time:
1. Run the following search which will dump all the results from the lookup into the KV Store:
| inputlookup expectedTime.csv | outputlookup expectedTime
2. Go to the new "Configure Broken Hosts Lookup" dashboard to check if data is populating on dashboard.
Version 3.3.4
v3.3.4
- Removed unnecessary inputs.conf
Version 3.3.2
v3.3.2:
- fixed a bug where the the broken hosts dashboard would show the wrong value for "Time Since Last Event"
- updated the app to work if the app directory is renamed
- updated the order of fields in the broken hosts dashboard
- reordered default expectedTime lookup table to be alphabetical
- added "cim_modactions" index to the default suppressions
- added cisco:ios default suppression
- added pan_config and pan:config default suppressions
Version 3.3.1
v3.3.1:
- bug fixes for splunk certification
-- scale icon sizes down to splunk approved sizes
v3.3.0:
- updated savedsearch to include any hosts that are sending logs from the future
- added the ability to add custom search additions to make the search more flexible
- added dashboard panel to show suppressed items
- updated dashboard panels to show currently broken items, and all items from the future
- added sparkline to the dashboard panels
Version 3.2.0
Modified the savedsearch to use 'tstats' instead of 'metadata' to allow use of sourcetype for tuning.
Updated the savedsearch schedule to run every 30 minutes (because tstats takes longer than metadata).
Updated the savedsearch suppression to suppress for 2 hours instead of 1.
Updated the savedsearch suppression to include sourcetype.
Updated expectedTime lookup table to add a 'sourcetype' column.
Updated first_time script to add 'sourcetype' column to lookup table.
Added Broken Hosts dashboard.
Updated documentation to include Broken Hosts dashboard information.
Added app nav color.
Version 3.1.0
RELEASE NOTES:
v3.1:
- Added setup page with default contact and default allowable lateness
v3.0:
- Another major rewrite
- Added the ability to suppress an item
- Added the ability to send different items to different contacts
v2.2:
- fixed issue with the index exclusions in the search
- reversed the order of the release notes, putting new version at the top
v2.1:
- wildcard in lookup table instead of empty quoted string
- app is visible (to allow the "run" button on the saved search to work)
- initial lookup table is now named with .sample extention to not over-write any previous tuning
v2.0: complete re-write of the app from scratch
- uses dbinspect and metadata commands to make this search much faster
- uses a lookup table to make tuning a breeze