For support, please email support@discoveredintelligence.ca
Splunk Cloud no longer allows for searching of REST API endpoints. As a result one of the searches named "Generate Meta Woot Server GUID Lookup" does not function. Please change or clone the search and edit the search syntax as follows:
index=_internal (host=idx* OR host=sh*) source="*metrics.log" instance_guid=* component=Metrics group=instance earliest=-6h
| stats latest(instance_guid) AS guid by server_name
| rename server_name AS splunk_server | outputlookup meta_woot_server_guid
This will be updated in a future release.
If you are upgrading from version 2 to version 4 you must run the search named "Migrate Meta Woot Key" one time following the upgrade. This search will modify the key used for the KV store and remediate an issue where keys could potentially be duplicated. Important: You do not need to run this if you are already on version 3 or higher.
- If you do not run this search, then any existing records will not be updated going forward until you do.
- You only need to run this search one time, then you can disable it.
The app leverages a tstats search to populate/update a KV store based lookup on a periodic basis with accurate metadata, specific to host, sourcetype and index. In addition, each time the lookup is updated, the rows being updated are written to a summary, which will allow for event count trending, latency and license usage over time by host, sourcetype and index.
You can use the data generated by Meta Woot! to:
- Instantly correlate hosts with sourcetype and index
- Quickly identify data sources that are logging behind / ahead of time.
- Identify hosts that might be sending one sourcetype but not another
- Create metrics to measure whether data sources have stopped coming in
- Accurately measure event count volumes by splunk server, index, sourcetype and host
- Calculate license volumes by index, sourcetype, host and even individual events!
- See all the hosts associated with each sourcetype and all the sourcetypes associated with each index
- Estimate license costs associated with your data sources and hosts
The app is simple to install.
1. Download the app from Splunkbase
2. Install the app on a search head or search head cluster.
3. Go to the search manager page and enable the "Generate Meta Woot Server GUID Lookup" search and RUN THE SEARCH to initially generate the lookup. (NOTE: See known issue at top - this must be edited for Splunk Cloud)
4. Go to the Data Models page and enable the acceleration for the Meta Woot License Usage data model. Default acceleration range is 7 days, but feel free to select a longer period.
5. Go to the search manager page and enable ONE of the "Generate Meta Woot Every X Min" searches. All searches are disabled by default. The 15 min search is typically a good one to use.
6. (optional) If you wish to have the summary data write into a different index, go the macro page and edit the meta_woot_summary macro.
7. (optional) If you wish to filter out certain indexes, sourcetypes, splunk_servers or hosts, go to the macro page and edit the meta_woot_index_filter, meta_woot_sourcetype_filter, meta_woot_splunk_server_filter and/or meta_woot_host_filter macros.
8. (optional) If you wish to have the summary data read from a different index, go to the macro page and edit the meta_woot_read_summary macro. Defaults to use the meta_woot_summary macro.
9. (optional) If you wish for Meta Woot! to clean out records that have not been updated for X seconds, enable the "Remove Meta Woot Stale Data" search. You can define X in seconds by editing the meta_woot_stale_data_secs macro. The default time is 2592000 secs, which is 30 days.
10. (optional) If you wish to enable accurate license cost reporting, go the macro page and edit the meta_woot_license_cost_per_gb and update with the approximate cost you are paying per Gb.
11. Restart the Splunk SH after installing.
That's it - the search will run and the KV store will populate after the search has run for the first time.
The app is focused on 'Index Time' data. Essentially every time the search runs, it looks back over a certain index time period. One search runs every 15 mins and looks back over a 15 minute index time range (20m@m to -5m@m). Another search runs every 5 mins and looks back over a 5 minute index time range (-10m@m to -5m@m). There is also now a 30 mins search as well. To take into account some processing delays in index time, we leave a 5 minute buffer. In addition to this rolling time window the search has outer time parameters of -4d and +24h. This is to account for latent data or data logging with future timestamps. This sounds inefficient, but the search runs pretty fast and the index time criteria helps focus the search down.
We chose to focus on index time, as we can then have a rolling window with some degree of accuracy. For example, if a host sends data into Splunk right now, but the data is 4 hours late; the index time would be the time it hits Splunk (i.e. now), so we would catch it. The only time we would not pick up the data is if data comes in that has an event timestamp more than four (4) days behind or in the future. If this case applies to your data, then you can adjust the time range accordingly - although you should really fix your data sources!
Each time the search is run, the events for that specific Index time window are gathered. The data, together with event counts is then summarized using the collect command to a summary index. The search then continues and updates the KV store with the latest data. Existing records are updated and any new records are added.
Meta Woot generates a KV state table of every host, sourcetype, index combination that has been seen in Splunk. It is continually updated every time the search runs. The following fields are in the lookup:
host - the hostname
sourcetype - the sourcetype
index - the index
hash - an md5 hash of host.sourcetype.index
recentTime - the index time
lastTime - the last timestamp for that data set
firstTime - the first timestamp for that data set that Meta Woot! has seen
lastUpdated - the last time the row in the KV store was updated
The summary index contains snapshots of the data found each time the search runs. For example, if you choose to run the 15 minute search, then there will be a summary dump each 15 mins of the data identified within that 15 minute index time windows. In addition, we capture event counts for each host, sourcetype, index combination. The event counts allow you to trend data counts over time. You would simply do something like | timechart span=15m sum(count) as count by host etc.
The summary index that is used can be set by amending the meta_woot_summary macro.
The sourcetype for the summary data is meta_woot.
host - the hostname
sourcetype - the sourcetype
index - the index
recentTime - the index time
lastTime - the last timestamp for that data set
firstTime - the first timestamp for that data set that Meta Woot! has seen
count - the event count
_time - the time that the dump was created
Using the default summary index, you can search for this data using index=summary sourcetype=meta_woot
The Meta Woot License Usage Data Model maps and accelerates the internal license data in Splunk. This data is then correlated with event volumes to calculate license volumes per event and host. The volume by host in the license internal data by itself is typically not accurate in large environments, so integrating this data and correlating with event volumes allows us to fairly accurately calculate license volumes by host and even by individual event. Pretty cool! In order for this to work, Meta Woot! must be installed on a Search Head that has visibility into the license logs. Typically we would encourage installing Meta Woot! on the same host as your DMC environment.
Yes, there are seven dashboards
- Meta Woot! Search - leverages the KV store data to report on the hosts, sourcetypes and indexes
- Meta Woot! Trends - leverages the summary data to show event count trends over time
- Meta Woot! Latency Compliance - leverages the summary and KV store data to report on data latency
- Meta Woot! Indexing Compliance - leverages the summary and KV store to report on data sources that have stopped sending data
- Meta Woot! First Time Report - Reports on data sources that have been seen for the first time over a selected past timeframe
- Meta Woot! License Volume Usage - leverages the license usage data model to show license usage by sourcetype, index and over time
- Meta Woot! License Event Usages - uses the KV store, license usage data model and summary data to show license usage per host and event
Requirements:
- Only works on Splunk 6.3 and above
- You must have a 64 bit version of Splunk installed
- In large environments, ensure that you have hardware that meets Splunk's requirements - the KV store can get big if you have a lot of hosts, sourcetypes and indexes.
- For the license model and reporting to work, Meta Woot! must be installed on a Search Head that is able to search the internal license data
Other Points:
- In some large environments the 30 min search may be more efficient than the search that runs every 5 mins. The selection of the right search will depend on how real-time you need the data to be and how long the search takes to run.
- Default search now looks back over 4 days, instead of 10 days, for better scalability in large environments
- It is recommended that Meta Woot! be installed on the same SH that you run the Splunk Monitoring Console (DMC)
The default thresholds for both the Meta Woot! Latency Compliance and the Meta Woot! Indexing Compliance can be changed. Simply edit the default values for the text box inputs. These values will automatically be passed down to the rest of the panels within the dashboard.
For Meta Woot! Latency Compliance edit: “Medium Latency (mins)” and “High Latency (mins)”
For Meta Woot! Indexing Compliance edit: “Delayed Thresholds (mins)” and “Late Threshold (mins)”
If the colors used for cell highlighting need to be changed, the hex values can be edited in the CSS file for the appropriate class. The CSS file is located:
$SPLUNK_HOME/etc/apps/meta_woot/appserver/static/metawoot_cell_highlighting.css
For support, to request feature enhancements or simply to give us your feedback - please contact us at support@discoveredintelligence.ca
Fixes:
- Updates for jQuery 3.5
As a Splunkbase app developer, you will have access to all Splunk development resources and receive a 10GB license to build an app that will help solve use cases for customers all over the world. Splunkbase has 1000+ apps from Splunk, our partners and our community. Find an app for most any data source and user need, or simply create your own with help from our developer portal.