Shuttl provides data archive management for Splunk. It supports backend storage solutions such as: ApacheHDFS, Amazon S3, or NFS attached storage. Shuttl works on the bucket level, and leverages the standard Splunk mechanism for archiving data based on total data size or time expiration. Use of Shuttl eliminates the need for Splunk users to implement their own homegrown solution for bulk-moving data to storage backends.
In addition to Archiving, Shuttl is useful for both compliance needs of data retention, as well as improving performance of Splunk. Shuttl also supports archiving the data in CSV format, and therefore, when data is moved to HDFS, it opens up the data to other tools such as Apache Hive and Hadoop Map Reduce to do further data processing and analysis.
For more information see the following blog articles:
Source code is available here: <https: github.com="" splunk="" splunk-shuttl="">
Quickstart Guide is available here: <https: github.com="" splunk="" splunk-shuttl="" wiki="" quickstart-guide="">
Setup video is available here: <http: www.youtube.com="" watch?v="OP7IYNVR5ms">
For feedback, please email shuttl-dev at splunk.com.
Now comes with both coldToFrozen AND warmToCold transfer retries!
This means that all failed transfers/shuttl's will be retried periodically (default every 60 secs), instead of "everytime any bucket is shuttl'ed".
User happiness is expected to go up by 137%!
Automatically retries to transfer coldToFrozen failures every 60 secs (configurable)
* Fixed bug where if an indexer (which is a search peer) does not have Shuttl installed, the resulting error would result in missing values in the UI.
* Further tests on Hadoop 1.1.1
Splunk Shuttl 0.8.0 release is another major milestone release.
The main new feature is distributed Shuttl operations from the Splunk Search Head. This means that all Shuttl operations will operate on the entire cluster, and not on a per-indexer basis.
* Listing of archived buckets
* Thawing of archived buckets
* Flushing of archived buckets
There is no special configuration that needs to be done on the Search Head. Shuttl when installed on a Search Head will query the Search Head for all Search Peers (the Search Peers should all have Shuttl installed of course), and all Shuttl operations will be distributed operations.
Extensive testing has been done for distributed failure scenarios. However, if problems are encountered, please report them ASAP for them to be addressed.
Shuttl version 0.7.2 represents a significant leap in functionality. Thanks to our robust user base for the feature request ideas. Now, Shuttl is easier to setup, supports a new backend (Glacier), reduced archive latency, and full support for the latest Splunk 5.0.
In summary, new features include:
- Amazon Glacier storage! Amazon Glacier is an extremely low-cost storage service that provides secure and durable storage for data archiving and backup.
- Splunk 5.0 Clustering Support. Since Splunk replicates buckets, when archiving, any archiving script will by default get duplicate buckets. Shuttl will dedup buckets and not waste storage space.
- Shuttl now will move data when buckets are moved to cold. This means significantly reduced latency times, and data that is shuttled is also still available for search by Splunk!
- New simpler and easier configuration! and a script to test out backends, to verify the configuration is correct, without going through the whole exercise of indexing data and forcing a bucket roll.
* Shuttl tested on CDH3. There is a wirelevel incompatibility with CDH and Apache, so there's documents on how to configure Shuttl to work with CDH3 on the github wiki. https://github.com/splunk/splunk-shuttl/wiki/System-Requirements
* Fix spurious error message when a thaw request is issued while thaw is in progress. No duplicates happen, and all requests are serviced.
* Simplified the failed bucket transfer dashboard.
* When doing thaw operation, the UI can "return" and list a table, before the thaw action completes - Fix is to do thaw operation asynchronously, so the UI returns immediately, while job happens in background.
* Amazon S3 tested, and works with no code change - slight display issue for bucket sizes in the UI, bug filed.
* Support for Flushing of Thawed Splunk Buckets via the UI
Known Issues being worked on:
* Quirks in the Thaw request UI
* S3 is not yet tested
* Failed buckets dashboard may report false failures
Splunk AppInspect evaluates Splunk apps against a set of Splunk-defined criteria to assess the validity and security of an app package and components.
As a Splunkbase app developer, you will have access to all Splunk development resources and receive a 10GB license to build an app that will help solve use cases for customers all over the world. Splunkbase has 1000+ apps and add-ons from Splunk, our partners and our community. Find an app or add-on for most any data source and user need, or simply create your own with help from our developer portal.