Amazon Kinesis Modular Input
Overview
Details
Splunk Amazon Kinesis Modular Input v1.3.4
Overview
This is a Splunk Modular Input Add-On for indexing records from Amazon Kinesis.
What is Kinesis ?
http://aws.amazon.com/kinesis/
Dependencies
- Splunk 5.0+
- Java Runtime 1.7+
- Supported on Windows, Linux, MacOS, Solaris, FreeBSD, HP-UX, AIX
Setup
- Optionally set your JAVA_HOME environment variable to the root directory of your JRE installation.If you don't set this , the input will look for a default installed java executable on the path.
- Untar the release to your $SPLUNK_HOME/etc/apps directory
- Restart Splunk
- If you are using a Splunk UI Browse to
Settings -- Data Inputs -- Amazon Kinesisto add a new Input stanza via the UI - If you are not using a Splunk UI (ie: you are running on a Universal Forwarder) , you need to add a stanza to inputs.conf directly as per the specification in
README/inputs.conf.spec. Theinputs.conffile should be placed in alocaldirectory under an App or User context.
Activation Key
You require an activation key to use this App. Visit http://www.baboonbones.com/#activation to obtain a non-expiring key
Logging
Any log entries/errors will get written to $SPLUNK_HOME/var/log/splunk/splunkd.log
These are also searchable in Splunk : index=_internal error kinesis.py
JVM Heap Size
The default heap maximum is 64MB.
If you require a larger heap, then you can alter this in $SPLUNK_HOME/etc/apps/kinesis_ta/bin/kinesis.py on line 95
JVM System Properties
You can declare custom JVM System Properties when setting up new input stanzas.
Note : these JVM System Properties will apply to the entire JVM context and all stanzas you have setup
Customized Message Handling
The way in which the Modular Input processes the received Kinesis records is enitrely pluggable with custom implementations should you wish.
To do this you code an implementation of the com.splunk.modinput.kinesis.AbstractMessageHandler class and jar it up.
Ensure that the necessary jars are in the $SPLUNK_HOME/etc/apps/kinesis_ta/bin/lib directory.
If you don't need a custom handler then the default handler com.splunk.modinput.kinesis.DefaultMessageHandler will be used.
Code examples are on GitHub : https://github.com/damiendallimore/SplunkModularInputsJavaFramework/tree/master/kinesis/src/com/splunk/modinput/kinesis
Troubleshooting
- JAVA_HOME environment variable is set or "java" is on the PATH for the user's environment you are running Splunk as
- You are using Splunk 5+
- You are using a 1.7+ Java Runtime
- You are running on a supported operating system
- Look for any errors in $SPLUNK_HOME/var/log/splunk/splunkd.log
- Run this command as the same user that you are running Splunk as and observe console output : "$SPLUNK_HOME/bin/splunk cmd python ../etc/apps/kinesis_ta/bin/kinesis.py --scheme"
Contact
This project was initiated by Damien Dallimore , damien@baboonbones.com
Release Notes
Version 1.3.4
cosmetic fixes
Version 1.3.3
cosmetic fixes
Version 1.3.2
updated docs
Version 1.3.1
added trial key functionality
Version 1.3
docs updated
Version 1.2
minor manager xml ui tweak for 7.1
Version 1.1
Added an activation key requirement , visit http://www.baboonbones.com/#activation to obtain a free,non-expiring key
Docs updated
Splunk 7.1 compatible
Version 1.0.3
Added JSON Object parsing for Cloudwatch to the GZIP handler
Version 1.0.2
tweaks to gzip handler
Version 1.0.1
Pushed default charset decoding out of the main message processing flow and into custom handling , so custom handlers that you implement should in theory be able to process any binary or text payload.
Version 1.0
Can now pass the raw payload bytes to your custom message handler ie: if you want to decode binary data
Added a custom GZIP decoder , com.splunk.modinput.kinesis.GZIPDataRecordDecoderHandler
Version 0.9
Tweaked the HEC transport.
Added a new custom handler that allows you to declare the fieldnames in the JSON that hold the time and host values of the event.
message_handler_impl = com.splunk.modinput.kinesis.JSONBodyWithFieldExtraction
message_handler_params = timefield=foo,hostfield=goo
Version 0.8
Added support to optional output to Splunk via a HEC (HTTP Event Collector) endpoint
Version 0.7
Enabled TLS1.2 support by default.
Made the core Modular Input Framework compatible with latest Splunk Java SDK
Please use a Java Runtime version 7+
If you need to use SSLv3 , you can turn this on in bin/kinesis.py
SECURE_TRANSPORT = "tls"
#SECURE_TRANSPORT = "ssl"
Version 0.6
Added a custom message handler that just dumps the JSON body.
message_handler_impl = com.splunk.modinput.kinesis.JSONOnlyMessageHandler
Version 0.5
Initial Beta release