splunk
Amazon Kinesis Modular Input
Overview
Details
This is a Splunk Modular Input Add-On for indexing records from Amazon Kinesis.
The Python code in this App is dual 2.7/3 compatible.
This version of the App enforces Python 3 for execution of the modular input script when running on Splunk 8+ in order to satisfy Splunkbase AppInspect requirements.
If running this App on Splunk versions prior to 8 , then Python 2.7 will get executed.
For details of the support we offer for our Apps , browse to : https://www.baboonbones.com/#support
This App is fully AppInspect passed for installing in your own hosted Splunk Enterprise environments or on your own hosted Splunk Forwarders and forwarding the collected data into Splunk Cloud.
If you want to install this App directly in Splunk Cloud , for our Commercially supported customers on our Ultra plan ( https://www.baboonbones.com/#support ) we provide custom private builds of any of our Apps tailored to the customer's specific requirements and configuration , and these custom private builds can then be uploaded as your own private Splunk Cloud Apps.
For customers that don't have support, we also offer the option to perform ad-hoc custom private builds of any of our Apps.
Browse to https://www.baboonbones.com/#cloud for more details of our Cloud options and offerings.
The Python code in this App is dual 2.7/3 compatible.
This version of the App enforces Python 3 for execution of the modular input script when running on Splunk 8+ in order to satisfy Splunkbase AppInspect requirements.
If running this App on Splunk versions prior to 8 , then Python 2.7 will get executed.
For details of the support we offer for our Apps , browse to : https://www.baboonbones.com/#support
This App is fully AppInspect passed for installing in your own hosted Splunk Enterprise environments or on your own hosted Splunk Forwarders and forwarding the collected data into Splunk Cloud.
If you want to install this App directly in Splunk Cloud , for our Commercially supported customers on our Ultra plan ( https://www.baboonbones.com/#support ) we provide custom private builds of any of our Apps tailored to the customer's specific requirements and configuration , and these custom private builds can then be uploaded as your own private Splunk Cloud Apps.
For customers that don't have support, we also offer the option to perform ad-hoc custom private builds of any of our Apps.
Browse to https://www.baboonbones.com/#cloud for more details of our Cloud options and offerings.
Release Notes
Version 1.7.5
June 18, 2022
-
in order to support huge modular input configuration xml strings when users want to run a very large number of inputs on 1 single instance of this app , we had to change the way that the child java process is invoked. Previously the xml string was passed as a program argument which could break the max argument size in the Linux kernel. Now we changed the logic to pass the xml string to the java process via the STDIN pipe.
-
The app performs periodic socket pings to the splunkd management port to determine if splunkd is still alive and if splunkd is not responding , usually because it has exited or is not network reachable, then the app self exits it's running java process.The default timeout is now 300 seconds. You can change this timeout value in bin/kinesis.py by setting the
SPLUNKD_TIMEOUT_SECSvariable. -
upgraded internal logging libraries to Log4j2 v2.17.2
Version 1.7.4
Dec. 23, 2021
fixed a bug in getting credentials from storage/passwords
Version 1.7.3
Dec. 23, 2021
upgraded internal logging libraries to Log4j2 v2.17.0
Version 1.7.2
Dec. 15, 2021
upgraded internal logging libraries to Log4j2 v2.16.0
Version 1.7.1
Dec. 11, 2021
upgraded internal logging libraries to Log4j2 v2.15.0
Version 1.7.0
Oct. 19, 2021
- general updates to meet latest Cloud Vetting requirements
- moved aws_secret_access_key out of inputs.conf , browse to the
Setup Credentialsmenu tab and enter any AWS Secret Access Key(s) you require. - activation key is now setup globally via a menu tab
- removed the HEC output option, default is now stdout
Version 1.6.3
Oct. 6, 2020
upgraded logging functionality
Version 1.6.2
Sept. 29, 2020
docs update
Version 1.6.1
Aug. 30, 2020
added a setup page to encrypt any credentials you require in your configuration
Version 1.6
Aug. 23, 2020
enforced python3 for execution of the modular input script.If you require Python2.7 , then download a prior version (such as 1.5).
Version 1.5
May 16, 2020
Dual Python 2.7 and 3+ compatibility.
App will run on :
Splunk Enterprise versions back to Splunk 5 where there is only a Python 2.7 runtime shipped
Splunk Enterprise version 8 where there is both a Python 2.7 and Python 3+ runtime shipped
Future versions of Splunk Enterprise where there is only a Python 3+ runtime
Version 1.4
Dec. 30, 2019
added JAXB dependencies for JRE 9+
fixed Splunk 8 compatibility for manager.xml file
Version 1.3.4
June 25, 2019
cosmetic fixes
Version 1.3.3
May 10, 2019
cosmetic fixes
Version 1.3.2
April 23, 2019
updated docs
Version 1.3.1
April 19, 2019
added trial key functionality
Version 1.3
March 28, 2019
docs updated
Version 1.2
June 3, 2018
minor manager xml ui tweak for 7.1
Version 1.1
May 27, 2018
Added an activation key requirement , visit http://www.baboonbones.com/#activation to obtain a free,non-expiring key
Docs updated
Splunk 7.1 compatible
Version 1.0.3
April 21, 2016
Added JSON Object parsing for Cloudwatch to the GZIP handler
Version 1.0.2
April 16, 2016
tweaks to gzip handler
Version 1.0.1
April 13, 2016
Pushed default charset decoding out of the main message processing flow and into custom handling , so custom handlers that you implement should in theory be able to process any binary or text payload.
Version 1.0
April 12, 2016
Can now pass the raw payload bytes to your custom message handler ie: if you want to decode binary data
Added a custom GZIP decoder , com.splunk.modinput.kinesis.GZIPDataRecordDecoderHandler
Version 0.9
Nov. 10, 2015
Tweaked the HEC transport.
Added a new custom handler that allows you to declare the fieldnames in the JSON that hold the time and host values of the event.
message_handler_impl = com.splunk.modinput.kinesis.JSONBodyWithFieldExtraction
message_handler_params = timefield=foo,hostfield=goo
Version 0.8
Sept. 22, 2015
Added support to optional output to Splunk via a HEC (HTTP Event Collector) endpoint
Version 0.7
Feb. 11, 2015
Enabled TLS1.2 support by default.
Made the core Modular Input Framework compatible with latest Splunk Java SDK
Please use a Java Runtime version 7+
If you need to use SSLv3 , you can turn this on in bin/kinesis.py
SECURE_TRANSPORT = "tls"
SECURE_TRANSPORT = "ssl"
Version 0.6
Sept. 12, 2014
Added a custom message handler that just dumps the JSON body.
message_handler_impl = com.splunk.modinput.kinesis.JSONOnlyMessageHandler
Version 0.5
Aug. 30, 2014
Initial Beta release