This is a Splunk Add-On for receiving data via a number of different data protocols.
Yes we do. And by all means use those. But if you want to perform some custom data handling and pre-processing of the received data before it gets indexed (above and beyond what you can accomplish using Splunk conf files) , then this Modular Input presents another option for you.
Furthermore , this Modular Input also implements several other protocols for sending data to Splunk.
This Modular Input utilizes VERTX.IO version 2.1.4 under the hood.http://vertx.io/vertx2/manual.html#what-is-vertx.
This framework provides for an implementation that is :
The way in which the Modular Input processes the received raw data is entirely pluggable with custom implementations should you wish.
This allows you to :
To do this you code a Vertx "Verticle" to handle the received data. http://vertx.io/vertx2/manual.html#verticle
These data handlers can be written in numerous JVM languages. http://vertx.io/vertx2/manual.html#polyglot
You then place the handler in the $SPLUNK_HOME/etc/apps/protocol_ta/bin/datahandlers directory.
On the Splunk config screen for the Modular Input there is a field where you can then specify the name of this handler to be applied.
If you don't need a custom handler then the default handler com.splunk.modinput.protocolverticle.DefaultHandlerVerticle will be used.
To get started , you can refer to the default handler examples in the datahandlers directory.
Note : experimental Nashorn support is included for js and coffee (requires Java 8). To use the Nashorn JS/Coffee engine rather than the default Rhino engine , then edit protocol_ta/bin/vertx_conf/langs.properties
This is provisioned using your own Java Keystore that you can create using the keytool utility that is part of the JDK.
Client certificate based authentication can be enabled for the TLS/SSL channels you setup.
Any required Vertx modules , such as various language modules for the polyglot functionality (JS , Scala , Groovy etc...) will be dynamically downloaded from online repositorys and installed in your protocol_ta/bin/vertx_modules directory.
You can edit your repository locations in protocol_ta/bin/vertx_conf/repos.txt
Due to the nature of the async/event driven/non blocking architecture , the out of the box default settings may just well suffice for you.
But there are some other parameters that you can tune to take more advantage of your underlying computing resource(ie: cpu cores) available to you.
These are the "server_verticle_instances" and "handler_verticle_instances" params.
Refer to http://vertx.io/vertx2/core_manual_java.html#specifying-number-of-instances for an explanation of how increasing the number of instances may help you.
You can also tune the TCP accept queue settings (also requires OS tweaks) , particularly if you are receiving lots of connections within a short time span.
By default data will be output to STDOUT in Modular Input Stream XML format.
However you can bypass this if you wish and declare that data is output to a Splunk TCP port or via Splunk's HTTP Event Collector.
Settings -- Data Inputs -- Protocol Data Inputsto add a new Input stanza via the UI
inputs.conffile should be placed in a
localdirectory under an App or User context.
You require an activation key to use this App. Visit http://www.baboonbones.com/#activation to obtain a non-expiring key
Any log entries/errors will get written to $SPLUNK_HOME/var/log/splunk/splunkd.log
These are also searchable in Splunk :
index=_internal error protocol.py
The default heap maximum is 256MB.
If you require a larger heap, then you can alter this in $SPLUNK_HOME/etc/apps/protocol_ta/bin/protocol.py on line 95
You can declare custom JVM System Properties when setting up new input stanzas.
Note : these JVM System Properties will apply to the entire JVM context and all stanzas you have setup
This project was initiated by Damien Dallimore , firstname.lastname@example.org
Search/Replace (with chars or a hash) Custom Data Handler Example
added trial key functionality
minor manager xml ui tweak for 7.1
Added an activation key requirement , visit http://www.baboonbones.com/#activation to obtain a free,non-expiring key
Splunk 7.1 compatible
Added the latest jython jar to the main classpath because the jython language module that
is dynamically installed is missing some useful jython modules ie:json
Added an example handler for decompressing gzip content
Minor HEC data handling tweaks
Added support to optional output to Splunk via a HEC (HTTP Event Collector) endpoint
Enabled TLS1.2 support by default.
Made the core Modular Input Framework compatible with latest Splunk Java SDK
Please use a Java Runtime version 7+
If you need to use SSLv3 , you can turn this on in bin/protocol.py
SECURE_TRANSPORT = "tls"
#SECURE_TRANSPORT = "ssl"
Abstracted the output transport logic out into verticles.
So you can choose from STDOUT (default for Modular Inputs) or bypass this and output
data to Splunk over other transports ie: TCP.
This also makes it easy to add other output transports in the future.
Futhermore , this makes the implementation of custom data handlers much cleaner as you don't have
worry out output transport logic or formatting Modular Input Stream XML for STDOUT transports.
Added langs.properties and repos.txt to the classpath
Initial beta release
Splunk AppInspect evaluates Splunk apps against a set of Splunk-defined criteria to assess the validity and security of an app package and components.
As a Splunkbase app developer, you will have access to all Splunk development resources and receive a 50GB license to build an app that will help solve use cases for customers all over the world. Splunkbase has 1000+ apps and add-ons from Splunk, our partners and our community. Find an app or add-on for most any data source and user need, or simply create your own with help from our developer portal.