Meta Woot!
Overview
Details
The app maintains a near real-time state table of host, sourcetype and index metadata. Meta Woot! is accurate at scale and allows users to instantly report on host, sourcetype and/or index together. The app includes summary based event count trending, correlation of event volumes against license and includes compliance reporting on both data latency and indexing.
Meta Woot!
For support, please email support@discoveredintelligence.ca
New Changes in Version 3.0
- Major change made to how Meta Woot! stores keys in the KV store to eliminate issues with duplicate keys
- Summarized data now also keeps track of Splunk Server
IF YOU ARE UPGRADING, YOU MUST RUN THE 'Migrate Meta Woot Key' SEARCH FOLLOWING UPGRADE
Upgrades from earliest versions to Version 3
If you are upgrading from Version 1 or 2 to version 3 you must run the search named "Migrate Meta Woot Key" one time following the upgrade. This search will modify the key used for the KV store and remediate an issue where keys could potentially be duplicated.
- If you do not run this search, then any existing records will not be updated going forward until you do.
- You only need to run this search one time, then you can disable it.
Overview
The app leverages a tstats search to populate/update a KV store based lookup on a periodic basis with accurate metadata, specific to host, sourcetype and index. In addition, each time the lookup is updated, the rows being updated are written to a summary, which will allow for event count trending, latency and license usage over time by host, sourcetype and index.
Why is this useful?
You can use the data generated by Meta Woot! to:
- Instantly correlate hosts with sourcetype and index
- Quickly identify data sources that are logging behind / ahead of time.
- Identify hosts that might be sending one sourcetype but not another
- Create metrics to measure whether data sources have stopped coming in
- Accurately measure event count volumes by splunk server, index, sourcetype and host
- Calculate license volumes by index, sourcetype, host and even individual events!
- See all the hosts associated with each sourcetype and all the sourcetypes associated with each index
Why not just use tstats or the metadata command?
- The metadata command is not guaranteed to be accurate at scale
- The metadata command is silo'd - meaning you cannot correlate host, sourcetype and index together
- The tstats command takes time to run over longer time frames - especially when focused on index time
How Does It work?
The app is focused on 'Index Time' data. Essentially every time the search runs, it looks back over a certain index time period. One search runs every 15 mins and looks back over a 15 minute index time range (20m@m to -5m@m). Another search runs every 5 mins and looks back over a 5 minute index time range (-10m@m to -5m@m). There is also now a 30 mins search as well. To take into account some processing delays in index time, we leave a 5 minute buffer. In addition to this rolling time window the search has outer time parameters of -4d and +24h. This is to account for latent data or data logging with future timestamps. This sounds inefficient, but the search runs pretty fast and the index time criteria helps focus the search down.
We chose to focus on index time, as we can then have a rolling window with some degree of accuracy. For example, if a host sends data into Splunk right now, but the data is 4 hours late; the index time would be the time it hits Splunk (i.e. now), so we would catch it. The only time we would not pick up the data is if data comes in that has an event timestamp more than four (4) days behind or in the future. If this case applies to your data, then you can adjust the time range accordingly - although you should really fix your data sources!
Each time the search is run, the events for that specific Index time window are gathered. The data, together with event counts is then summarized using the collect command to a summary index. The search then continues and updates the KV store with the latest data. Existing records are updated and any new records are added.
What Does the Meta Woot KV Lookup contain?
Meta Woot generates a KV state table of every host, sourcetype, index combination that has been seen in Splunk. It is continually updated every time the search runs. The following fields are in the lookup:
host - the hostname
sourcetype - the sourcetype
index - the index
hash - an md5 hash of host.sourcetype.index
recentTime - the index time
lastTime - the last timestamp for that data set
firstTime - the first timestamp for that data set that Meta Woot! has seen
lastUpdated - the last time the row in the KV store was updated
What does the summary index contain?
The summary index contains snapshots of the data found each time the search runs. For example, if you choose to run the 15 minute search, then there will be a summary dump each 15 mins of the data identified within that 15 minute index time windows. In addition, we capture event counts for each host, sourcetype, index combination. The event counts allow you to trend data counts over time. You would simply do something like | timechart span=15m sum(count) as count by host etc.
The summary index that is used can be set by amending the meta_woot_summary macro.
The sourcetype for the summary data is meta_woot.
host - the hostname
sourcetype - the sourcetype
index - the index
recentTime - the index time
lastTime - the last timestamp for that data set
firstTime - the first timestamp for that data set that Meta Woot! has seen
count - the event count
_time - the time that the dump was created
Using the default summary index, you can search for this data using index=summary sourcetype=meta_woot
What does the Meta Woot License Usage Data Model do?
In version 2.0 of Meta Woot! we have included a Data Model that maps and accelerates the internal license data in Splunk. This data is then correlated with event volumes to calculate license volumes per event and host. The volume by host in the license internal data by itself is typically not accurate in large environments, so integrating this data and correlating with event volumes allows us to fairly accurately calculate license volumes by host and even by individual event. Pretty cool! In order for this to work, Meta Woot! must be installed on a Search Head that has visibility into the license logs. Typically we would encourage installing Meta Woot! on the same host as your DMC environment.
Are there dashboards/reports?
Yes, there are seven dashboards
- Meta Woot! Search - leverages the KV store data to report on the hosts, sourcetypes and indexes
- Meta Woot! Trends - leverages the summary data to show event count trends over time
- Meta Woot! Latency Compliance - leverages the summary and KV store data to report on data latency
- Meta Woot! Indexing Compliance - leverages the summary and KV store to report on data sources that have stopped sending data
- Meta Woot! First Time Report - Reports on data sources that have been seen for the first time over a selected past timeframe
- Meta Woot! License Volume Usage - leverages the license usage data model to show license usage by sourcetype, index and over time
- Meta Woot! License Event Usages - uses the KV store, license usage data model and summary data to show license usage per host and event
Important Stuff
Requirements:
- Only works on Splunk 6.3 and above
- You must have a 64 bit version of Splunk installed
- In large environments, ensure that you have hardware that meets Splunk's requirements - the KV store can get big if you have a lot of hosts, sourcetypes and indexes.
- For the license model and reporting to work, Meta Woot! must be installed on a Search Head that is able to search the internal license data
Other Points:
- In some large environments the 30 min search may be more efficient than the search that runs every 5 mins. The selection of the right search will depend on how real-time you need the data to be and how long the search takes to run.
- Default search now looks back over 4 days, instead of 10 days, for better scalability in large environments
- It is recommended that Meta Woot! be installed on the same SH that you run the Splunk Monitoring Console (DMC)
How do I install this?
The app is simple to install.
1. Download the app from Splunkbase
2. Install the app on a search head or search head cluster.
3. Go to Settings --> "Searches, reports and alerts" and enable the "Generate Meta Woot Server GUID Lookup" search and then RUN THE SEARCH (click on run, under actions), to initially generate the lookup
4. Go to Settings --> "Data models" and enable the acceleration for the Meta Woot License Usage data model. Default acceleration range is 7 days, but feel free to select a longer period.
5. Go to Settings --> "Searches, reports and alerts" and enable ONE of the "Generate Meta Woot Every X Min" searches. All searches are disabled by default. The 15 min search is typically a good one to use.
4. (optional) If you wish to have the summary data write into a different index, go the macro page and edit the meta_woot_summary macro.
5. (optional) If you wish to filter out certain indexes, sourcetypes, splunk_servers or hosts, go to the macro page and edit the meta_woot_index_filter, meta_woot_sourcetype_filter, meta_woot_splunk_server_filter and/or meta_woot_host_filter macros.
6. (optional) If you wish to have the summary data read from a different index, go to the macro page and edit the meta_woot_read_summary macro. Defaults to use the meta_woot_summary macro.
7. (optional) If you wish for Meta Woot! to clean out records that have not been updated for X seconds, enable the "Remove Meta Woot Stale Data" search. You can define X in seconds by editing the meta_woot_stale_data_secs macro. The default time is 2592000 secs, which is 30 days.
8. Restart the Splunk SH after installing.
That's it - the search will run and the KV store will populate after the search has run for the first time.
How to Set Default Compliance Thresholds
The default thresholds for both the Meta Woot! Latency Compliance and the Meta Woot! Indexing Compliance can be changed. Simply edit the default values for the text box inputs. These values will automatically be passed down to the rest of the panels within the dashboard.
For Meta Woot! Latency Compliance edit: “Medium Latency (mins)” and “High Latency (mins)”
For Meta Woot! Indexing Compliance edit: “Delayed Thresholds (mins)” and “Late Threshold (mins)”
How to Change Cell Value Colors
If the colors used for cell highlighting need to be changed, the hex values can be edited in the CSS file for the appropriate class. The CSS file is located:
$SPLUNK_HOME/etc/apps/meta_woot-master/appserver/static/metawoot_cell_highlighting.css
For support, to request feature enhancements or simply to give us your feedback - please contact us at support@discoveredintelligence.ca
Release Notes
Version 3.0.0
## New Changes in Version 3.0 ############################
- Major change made to how Meta Woot! stores keys in the KV store to eliminate issues with duplicate keys
- Summarized data now also keeps track of Splunk Server
## IMPORTANT - Upgrading from earlier versions to version 3 #########################
If you are upgrading from Version 2 to version 3 you must run the search named "Migrate Meta Woot Key" one time following the upgrade. This migration search will modify the key used for the KV store and remediate an issue where keys could potentially be duplicated.
- If you do not run this search, then any existing records will not be updated going forward until you do.
- You only need to run this search one time, then you can disable it.
Version 2.1.1
## New Changes in Version 2.1 ############################
- New Meta Woot! First Time Report dashboard provides insight into new data sources or hosts over a selected timeframe
- New Remove Meta Woot! Stale Data search allows you to remove entries from the Meta Woot KV store that have not been updated for your selected timeframe
- New Splunk Server Filter macro allows you to filter out specific Splunk servers
- Various bug fixes and tweaks
Version 2.1
Version 2.0.4
Minor bug fixes
- Removed indexing compliance line chart which was not functioning correctly.
- Corrected table to read "No Latency" when latency is truly 0.00 seconds
- Added summariesonly=t to licence event dashboard for performance improvements
Version 2.0.2
Fixed minor bug in 6.5.x versions, causing the latency dashboards not to load.
Version 2.00
Version 1.12
Bug Fixes:
- Fixed issue with first detected date not being update if earlier timestamped data starts coming in
- Expanded the timeframe that the main search looks over to 10 days in order to pick up sources that are latent by days not hours.
New coolness:
- Splunk Certified! and Splunk Cloud Certified!
- Built in the ability to filter out indexes, sourcetypes and hosts that you don't want meta_woot indexing or reporting on. All you need to do is edit the respective macros.
- Added a new Meta Woot Compliance dashboard to report on data source latency that is beyond acceptable bounds. Also allows for reporting on hosts that are no longer sending data within a specified timeframe.
Version 1.11
Bug Fixes:
- Fixed issue with first detected date not being update if earlier timestamped data starts coming in
- Expanded the timeframe that the main search looks over to 10 days in order to pick up sources that are latent by days not hours.
New coolness:
- Built in the ability to filter out indexes, sourcetypes and hosts that you don't want meta_woot indexing or reporting on. All you need to do is edit the respective macros.
- Added a new Meta Woot Compliance dashboard to report on data source latency that is beyond acceptable bounds. Also allows for reporting on hosts that are no longer sending data within a specified timeframe.
Version 1.9
No functional changes - modified permissions for app certification process
Version 1.8
Version 1.8
- Force a restart upon first install (this was required)
- Fixed issue with the no filter drop down on the search dashboard not working
- The scheduled searches are now disabled on install by default - simply enable the search required.
- Documentation updated
Version 1.7
- Fixed typo on trends dashboard
- Modified default time to 24hrs on trends board and default span to 30 mins
- Added a 30 min search for those that want a less frequent udpate
- Modified owner of searches to admin
- Added latency filtering to the search dashboard to identify hosts logging ahead/behind index time
- Added drill down from search to trending dashboard
This will likely be the last mini-update for a bit, barring any major bugs or issues. If you have enhancement requests, please send them along to support@discoveredintelligence.ca
Version 1.6
Version 1.5
Modifications to the searches to leverage index time over event time - this massively improves accuracy and takes into account latent data or future timestamped data.
Addition of documentation.
Smaller changes to dashboards and other tweaks following testing and feedback.
Works with Splunk version 6.3 and above only
Version 1.3
Version 1.3
Adds macro to set the summary index name
Works on version 6.3 and above.
Version 1.2
Version 1.2
Compatible with Splunk 6.3 and above only due to use of KV where filtering introduced with 6.3.
Version 1.1
Version 1.1
Choose whether the populating search runs every 5 mins or every 15 mins.
Just activate one of the searches.
Version 1.0
Version 1.0
Works on 6.2 and above.