REST API Modular Input
Overview
Details
Splunk REST API Modular Input v1.5.6
Overview
This is a Splunk modular input add-on for polling REST APIs.
Activation Key
You require an activation key to use this App. Visit http://www.baboonbones.com/#activation to obtain a non-expiring key
Features
- Perform HTTP(s) GET/POST/PUT/HEAD requests to REST endpoints and output the responses to Splunk
- Multiple authentication mechanisms
- Add custom HTTP(s) Header properties
- Add custom URL arguments
- HTTP(s) Streaming Requests
- HTTP(s) Proxy support , supports HTTP CONNECT Verb
- Response regex patterns to filter out responses
- Configurable polling interval
- Configurable timeouts
- Configurable indexing of error codes
- Persist and retrieve cookies
Authentication
The following authentication mechanisms are supported:
- None
- HTTP Basic
- HTTP Digest
- OAuth1
- OAuth2 (with auto refresh of the access token)
- Custom
Custom Authentication Handlers
You can provide your own custom Authentication Handler. This is a Python class that you should add to the rest_ta/bin/authhandlers.py module.
http://docs.python-requests.org/en/latest/user/advanced/#custom-authentication
You can then declare this class name and any parameters in the REST Input setup page.
Custom Response Handlers
You can provide your own custom Response Handler. This is a Python class that you should add to the rest_ta/bin/responsehandlers.py module.
You can then declare this class name and any parameters in the REST Input setup page.
Token substitution in Endpoint URL
There is support for dynamic token substitution in the endpoint URL
ie : /someurl/foo/$sometoken$/goo
$sometoken$ will get substituted with the output of the 'sometoken' function in bin/tokens.py
So you can add you own tokens simply by adding a function to bin/tokens.py
Currenty there is 1 token implemented , $datetoday$ , which will resolve to today's date in format "2014-02-18"
Token replacement functions in the URL can also return a list of values, that will cause
multiple URL's to be formed and the requests for these URL's will be executed in parallel in multiple threads.
Dependencies
- Splunk 5.0+
- Supported on Windows, Linux, MacOS, Solaris, FreeBSD, HP-UX, AIX
Setup
- Untar the release to your $SPLUNK_HOME/etc/apps directory
- Restart Splunk
- If you are using a Splunk UI Browse to
Settings -- Data Inputs -- RESTto add a new Input stanza via the UI - If you are not using a Splunk UI (ie: you are running on a Universal Forwarder) , you need to add a stanza to inputs.conf directly as per the specification in
README/inputs.conf.spec. Theinputs.conffile should be placed in alocaldirectory under an App or User context.
Logging
Any log entries/errors will get written to $SPLUNK_HOME/var/log/splunk/splunkd.log
These are also searchable in Splunk : index=_internal error rest.py
Troubleshooting
- You are using Splunk 5+
- Look for any errors in $SPLUNK_HOME/var/log/splunk/splunkd.log
- Any firewalls blocking outgoing HTTP calls
- Is your REST URL, headers, url arguments correct
- Is you authentication setup correctly
Contact
Release Notes
Version 1.5.6
updated docs
Version 1.5.5
added trial key functionality
Version 1.5.4
added a triggers stanza to app.conf to prevent reloading after saving state back to inputs.conf
Version 1.5.3
Patched a bug to callbacks to Splunk for persisting state that required the activation key in the payload
Version 1.5.2
minor manager xml ui tweak for 7.1
Version 1.5.1
Corrected a build bug with responsehandlers
Version 1.5
Added an activation key requirement , visit http://www.baboonbones.com/#activation to obtain a free,non-expiring key
Added support for HEAD requests
Docs updated
Splunk 7.1 compatible
Version 1.4
Delimiter fix
Version 1.3.9
Can now declare a CRON pattern for your polling interval.
Multiple requests spawned by tokenization can be declared to run in parallel or sequentially.
Multiple sequential requests can optionally have a stagger time enforced between each request.
Version 1.3.8
Minor code fix for a logging statement error
Version 1.3.7
Added support for token replacement functions in the URL to be able to return a list
of values, that will cause multiple URL's to be formed and the requests for these
URL's will be executed in parallel in multiple threads. See tokens.py
Version 1.3.6
Added a custom response handler for rolling out generic JSON arrays
Refactored key=value delimited string handling to only split on the first "=" delimiter
Version 1.3.5
Ensure that token substitution in the endpoint URL is dynamically applied for each
HTTP request
Version 1.3.4
Added support for dynamic token substitution in the endpoint URL
ie : /someurl/foo/$sometoken$/goo
$sometoken$ will get substituted with the output of the 'sometoken' function
in bin/tokens.py
Currently have just shipped with 1 example token $datetoday$ which will dynamically resolve to today's date in format 2014-02-18
Version 1.3.3
Added support for sending and persisting cookies
Version 1.3.2
Changed the logic for persistence of state back to inputs.conf to occur directly after polling/event indexing has completed rather than waiting for the polling loop frequency sleep period to exit. This potentially deals with situations where you might terminate Splunk before the REST Mod Input has persisted state changes back to inputs.conf because it was in a sleep loop during shutdown.
Version 1.3.1
Cosmetic fix for 1.3 release
Version 1.3
Added a new feature that will automatically persist updates to URL Arguments , HTTP Header Propertys or HTTP Request Body content back to your inputs.conf stanza. Such a scenario might occur if you are using a custom response handler to dynamically calculate URL Arguments , such as a timestamp or event paging cursor, and you want this latest state to be persisted back into your configuration so that if you need to restart the REST input , it's configuration is in the latest polled state and can resume polling from where it left off.
Version 1.2
Upgraded underlying python requests library to version 2.0 , primarily to support the HTTP CONNECT verb
Version 1.1
Added support for user defined delimiter for multiple "key=value" fields .
Added hooks in responsehandlers.py for custom handling of responses, use cases such as URL arguments/HTTP header properties that might require a dynamic value per request , HTTP REL Header link following , dynamically changing the endpoint URL.
Version 1.0.6beta
Strip newlines from default output ++ add unbroken attribute to XML output stream so that Splunk props /transforms can be applied
Version 1.0.5beta
Added more robust exception handling
Version 1.0.4beta
Fixed minor script bug when printing http errors
Version 1.0.3beta
Added support for POST and PUT HTTP Methods for getting data. Not RESTful per say but a useful out for API's that are "REST like"
Version 1.0.2beta
Renamed the manager xml file to avoid naming clashes
Version 1.0.1beta
Fixed some spelling typos
Version 1.0beta
First release