Recorded Future Add-on for Splunk ES
Overview
Details
This Add-on integrates with the Splunk ES Threat Intelligence Framework by adding a formatted feed containing information security threat indicators. With this added feature, defenders can automate the process of finding connections between internal incidents and external sources. This can work bidirectionally: searching Recorded Future for more context around internally observed indicators, or testing trending indicators from open source reporting against internal datasets.
The Add-on also simplifies the workflow of analysts working within the ES environment by adding contextual actions to the Incident Review and event searching and reporting views. This includes information on IPs, domains, file hashes and CVEs.
Overview:
The Splunk Add-on for Recorded Future is designed specifically for Splunk Enterprise Security.
This Add-on integrates with the Splunk ES Threat Intelligence Framework by adding a feed containing information security threat indicators from Recorded Future. With this added feature, defenders can automate the process of finding connections between internal incidents and external sources. This can work bidirectionally: searching Recorded Future for more context around internally observed indicators, or testing trending indicators from open source reporting against internal data sets.
The Add-on also simplifies the workflow of analysts working within the ES environment by adding contextual actions to the Incident Review and event searching and reporting views. This includes information on IPs, domains, file hashes and CVEs.
For more information on Recorded Future, visit www.recordedfuture.com.
Documentation:
Requirements
- Splunk ES must be installed on the Splunk system
- A valid Recorded Future API token is required.
- The Splunk server running the app must be able to download a CSV
file containing Recorded Future's IP risk list from
https://api.recordedfuture.com/.
Installation
To install this Add-on, perform the following steps:
- Download the latest TA release from Splunkbase (apps.splunk.com)
- In Splunk, select "Manage Apps" from the drop-down menu next
to the Splunk logo on the upper left of the screen - Select "Install app from file"
- Browse to the location of the TA-recorded_future.spl file,
select it and upload. Restart Splunk when prompted to do so. - Go back to "Manage Apps". Locate "Splunk ES Add-on for
Recorded Future" in the list and run "Set up". - Go to Settings->Data inputs->Scripts. Enable the script
get-rf-threatlists.py that corresponds to your platform
(Windows or *nix). - In the Enterprise Security menu bar,
click Configure -> Incident Management -> Incident Review Settings. -
Click the button 'Add new entry' in the "Incident Review -
Event Attributes" section. Add the following Label and Field
combinations:Label Field --------------------- ---------------------- RF Risk Score rf_a_risk_score RF Risk Score Threshold rf_risk_score_threshold RF Triggered Rules rf_b_risk_string RF Evidence Details rf_evidence_details_0 RF Evidence Details 2 rf_evidence_details_1 RF Evidence Details 3 rf_evidence_details_2 RF Evidence Details 4 rf_evidence_details_3 RF Evidence Details 5 rf_evidence_details_4 RF Evidence Details (remaining) rf_evidence_details_rest -
A restart of the Splunk instance will be required once the
installation has completed. - If you haven't already done so, enable the Enterprise
Security correlation search called "Threat Activity Detected"
a. In the Enterprise Security menu bar,
click Configure -> Content Management
b. In the filter bar, type "Threat Activity Detected"
c. Click the link 'Enable' to enable the correlation search - Optionally, make the logs from the app available in the Splunk
GUI. Ex:
- Add an index _TA-recorded_future
- Setup a Data input that monitors
$SPLUNK_HOME/var/log/TA-recorded_future and insert these events
into the new index.
Alternatively, you can download the Add-on using the Splunk Web interface's "Find more apps online" feature. Steps 5 and onwards above must still be completed.
Upgrade from 2.x versions
The lookup file's name has changed, the old file should be removed. Please delete:
$SPLUNK_HOME/etc/apps/TA-recorded_future/lookups/rf-threatlist.csv
Setup
After installation, you will need to set up the Add-on for Recorded Future to communicate with the Recorded Future API. The setup page will request your API token, and the Add-on will not import threat information without this. This is described in step 4 of the installation procedure above.
In the setup view it's also possible to add proxy information if the Splunk server lacks direct Internet
access. The format for the proxy configuration is "https://proxy.example.com:1234" (please note that Splunk
only allows https based proxies).
It is also possible to increase the verbosity of the logging in the Setup view by checking the enable
debug logging checkbutton.
Advanced setup
The script which retrieves the lists of IP number etc will select a Risk Score Threshold which will
produce a set number of entries in the list. By default this number is 25.000. This number can be
changed by editing the file recorded_future.conf in the local directory in the app directory. Ex:
[ip_risk_list]
max_entries = 25000
Be aware of that increasing this number will significantly increase the load on the server.
External data access and scripts
The app uses a script to retrieve threat information from Recorded Future. This script is run (default every 60 minutes) from a stanza in inputs.conf. This threat information contains IP numbers, Domain names and Hashes which are considered suspicious or malicious by Recorded Future. Each entity has a Risk Score and evidence of why that is assigned to it. The data contains all entities with a Risk Score of 25 or more (100 being the maximum).
During the execution one https call is made to https://api.recordedfuture.com/ to retrieve the information which is then processed and stored on the Splunk server for each type of data.
Cluster considerations
If using a large risk lists (see the Setup section above) it may be advisable to avoid transferring it across cluster nodes. This will however force the lookups to be performed locally on the search head, putting additional strain on that node.
If you want it to perform the search locally, then you can:
- add the parameter local=true to the custom search,
- you can blacklist the lookup file from the indexers:
https://docs.splunk.com/Documentation/Splunk/6.4.3/Admin/Distsearchconf
If the Knowledge bundle is too large it may be necessary to lower the max_entries values in recorded_future.conf.
In the README folder of the app directory there is a more detailed instruction about how to deploy the app in a search head cluster environment.
More information
For more information and to set up your trial or paid subscription, please contact splunk@recordedfuture.com (US east coast business hours).
Release Notes
Version 4.0.0
Support for Recorded Future Fusion.
- With Fusion it's possible to tune risklist according to requirements (ex ignore certain types of threats that aren't applicable).
- Any number of more targeted risklists can be added to the Splunk system to improve detection accuracy.
- Proprietary risk data can be applied to enhance risklists.
Support for monitoring Recorded Future alerts within the Splunk system.
- A new modular input can retrieve selected alerts and create events for the alerts.
New Adaptive response action which can be added to high fidelity correlation searches to enrich with absolutely up-to-date information regardless of risk level. This action can also be called in ad-hoc fashion for any notable event with a compatible IOC type.
Extensive app documentation is available within the app.
A number of reports have been added to facilitate monitoring and troubleshooting of the app.
Version 3.2.2
The main addition with the 3.0 release is the addition of two new threat lists:
- domains: allows for correlation and enrichment on events from proxies, web application firewalls, DNS servers etc
- hashes: events from anti-virus systems, email gateways, file alteration monitors etc
In addition to the new risk lists there are many other improvements:
- significant performance enhancements
- support for clustering (in particular search head clusters)
- support for running on Windows based Splunk servers
- support for Splunk servers with a non standard management port.
Version 3.1.0 adds a Setup GUI where in addition to the Recorded Future API key it's easy to toggle debug logging or adding proxy information.
Version 3.2.0 adds a verification script that can be run to check that configuration and access needed for the app to work is present.
Version 3.1.1
The main addition with the 3.0 release is the addition of two new threat lists:
- domains: allows for correlation and enrichment on events from proxies, web application firewalls, DNS servers etc
- hashes: events from anti-virus systems, email gateways, file alteration monitors etc
In addition to the new risk lists there are many other improvements:
- significant performance enhancements
- support for clustering (in particular search head clusters)
- support for running on Windows based Splunk servers
- support for Splunk servers with a non standard management port.
Version 3.1.0 adds a Setup GUI where in addition to the Recorded Future API key it's easy to toggle debug logging or adding proxy information.