REST API Modular Input
Overview
Details
Splunk REST API Modular Input v1.4
by Damien Dallimore , damien@baboonbones.com
Overview
This is a Splunk Modular Input for polling REST APIs and indexing the responses.
Features
- Perform HTTP(s) GET requests to REST endpoints and output the responses to Splunk
- Optional POST and PUT HTTP Methods
- Multiple authentication mechanisms
- Add custom HTTP(s) Header properties
- Add custom URL arguments
- HTTP(s) Streaming Requests
- HTTP(s) Proxy support
- Response regex patterns to filter out responses
- Configurable polling interval
- Configurable timeouts
- Configurable indexing of error codes
- Custom Response Handling plugin architecture
- Persist and retrieve cookies
Authentication
The following authentication mechanisms are supported:
- None
- HTTP Basic
- HTTP Digest
- OAuth1
- OAuth2 (with auto refresh of the access token)
- Custom
Custom Authentication Handlers
You can provide your own custom Authentication Handler. This is a Python class that you should add to the
rest_ta/bin/authhandlers.py module.
http://docs.python-requests.org/en/latest/user/advanced/#custom-authentication
You can then declare this class name and any parameters in the REST Input setup page.
Custom Response Handlers
You can provide your own custom Response Handler. This is a Python class that you should add to the
rest_ta/bin/responsehandlers.py module.
You can then declare this class name and any parameters in the REST Input setup page.
Such use cases for implementing a custom response handler might include
- dynamic setting of URL arguments/HTTP header propertys ie: a stream cursor
- extracting the next link to follow from the response content or HTTTP REL header
- custom pre-processing of the raw HTTP response before indexing in Splunk
- dynamically changing the REST endpoint URL
Token substitution in Endpoint URL
There is support for dynamic token substitution in the endpoint URL
ie : /someurl/foo/$sometoken$/goo
$sometoken$ will get substituted with the output of the 'sometoken' function in bin/tokens.py
So you can add you own tokens simply by adding a function to bin/tokens.py
Currenty there is 1 token implemented , $datetoday$ , which will resolve to today's date in format "2014-02-18"
Token replacement functions in the URL can also return a list of values, that will cause
multiple URL's to be formed and the requests for these URL's will be executed in parallel in multiple threads.
Dependencies
- Splunk 5.0+
- Supported on Windows, Linux, MacOS, Solaris, FreeBSD, HP-UX, AIX
Setup
- Untar the release to your $SPLUNK_HOME/etc/apps directory
- Restart Splunk
- Browse to Manager -> Data Inputs -> REST and setup your inputs
Logging
Any modular input log errors will get written to $SPLUNK_HOME/var/log/splunk/splunkd.log
Troubleshooting
- You are using Splunk 5+
- Look for any errors in $SPLUNK_HOME/var/log/splunk/splunkd.log
- Any firewalls blocking outgoing HTTP calls
- Are your REST URL, headers, url arguments correct
- Is your authentication setup correctly
Release Notes
Version 1.4
Delimiter fix
Version 1.3.9
Can now declare a CRON pattern for your polling interval.
Multiple requests spawned by tokenization can be declared to run in parallel or sequentially.
Multiple sequential requests can optionally have a stagger time enforced between each request.
Version 1.3.8
Minor code fix for a logging statement error
Version 1.3.7
Added support for token replacement functions in the URL to be able to return a list
of values, that will cause multiple URL's to be formed and the requests for these
URL's will be executed in parallel in multiple threads. See tokens.py
Version 1.3.6
Added a custom response handler for rolling out generic JSON arrays
Refactored key=value delimited string handling to only split on the first "=" delimiter
Version 1.3.5
Ensure that token substitution in the endpoint URL is dynamically applied for each
HTTP request
Version 1.3.4
Added support for dynamic token substitution in the endpoint URL
ie : /someurl/foo/$sometoken$/goo
$sometoken$ will get substituted with the output of the 'sometoken' function
in bin/tokens.py
Currently have just shipped with 1 example token $datetoday$ which will dynamically resolve to today's date in format 2014-02-18
Version 1.3.3
Added support for sending and persisting cookies
Version 1.3.2
Changed the logic for persistence of state back to inputs.conf to occur directly after polling/event indexing has completed rather than waiting for the polling loop frequency sleep period to exit. This potentially deals with situations where you might terminate Splunk before the REST Mod Input has persisted state changes back to inputs.conf because it was in a sleep loop during shutdown.
Version 1.3.1
Cosmetic fix for 1.3 release
Version 1.3
Added a new feature that will automatically persist updates to URL Arguments , HTTP Header Propertys or HTTP Request Body content back to your inputs.conf stanza. Such a scenario might occur if you are using a custom response handler to dynamically calculate URL Arguments , such as a timestamp or event paging cursor, and you want this latest state to be persisted back into your configuration so that if you need to restart the REST input , it's configuration is in the latest polled state and can resume polling from where it left off.
Version 1.2
Upgraded underlying python requests library to version 2.0 , primarily to support the HTTP CONNECT verb
Version 1.1
Added support for user defined delimiter for multiple "key=value" fields .
Added hooks in responsehandlers.py for custom handling of responses, use cases such as URL arguments/HTTP header properties that might require a dynamic value per request , HTTP REL Header link following , dynamically changing the endpoint URL.
Version 1.0.6beta
Strip newlines from default output ++ add unbroken attribute to XML output stream so that Splunk props /transforms can be applied
Version 1.0.5beta
Added more robust exception handling
Version 1.0.4beta
Fixed minor script bug when printing http errors
Version 1.0.3beta
Added support for POST and PUT HTTP Methods for getting data. Not RESTful per say but a useful out for API's that are "REST like"
Version 1.0.2beta
Renamed the manager xml file to avoid naming clashes
Version 1.0.1beta
Fixed some spelling typos
Version 1.0beta
First release