Recorded Future Add-on for Splunk ES
Overview
Details
This Add-on integrates with the Splunk ES Threat Intelligence Framework by adding a formatted feed containing information security threat indicators. With this added feature, defenders can automate the process of finding connections between internal incidents and external sources. This can work bidirectionally: searching Recorded Future for more context around internally observed indicators, or testing trending indicators from open source reporting against internal datasets.
The Add-on also simplifies the workflow of analysts working within the ES environment by adding contextual actions to the Incident Review and event searching and reporting views. This includes information on IPs, domains, file hashes and CVEs.
Overview:
The Splunk Add-on for Recorded Future is designed specifically for Splunk Enterprise Security.
This Add-on integrates with the Splunk ES Threat Intelligence Framework by adding a feed containing information security threat indicators from Recorded Future. With this added feature, defenders can automate the process of finding connections between internal incidents and external sources. This can work bidirectionally: searching Recorded Future for more context around internally observed indicators, or testing trending indicators from open source reporting against internal data sets.
The Add-on also simplifies the workflow of analysts working within the ES environment by adding contextual actions to the Incident Review and event searching and reporting views. This includes information on IPs, domains, file hashes and CVEs.
For more information on Recorded Future, visit www.recordedfuture.com.
Requirements
- Splunk ES must be installed on the Splunk system
- A valid Recorded Future API token is required.
- The Splunk server running the app must be able to download a CSV
file containing Recorded Future's IP risk list from
https://api.recordedfuture.com/.
Installing and configuring
Installation
To install this Add-on, perform the following steps:
- Download the latest TA release from Splunkbase (apps.splunk.com)
- In Splunk, select "Manage Apps" from the drop-down menu next
to the Splunk logo on the upper left of the screen - Select "Install app from file"
- Browse to the location of the TA-recorded_future.spl file,
select it and upload. Restart Splunk when prompted to do so. - In the Enterprise Security menu bar,
click Configure → Incident Management → Incident Review Settings. -
Click the button 'Add new entry' in the "Incident Review -
Event Attributes" section. Add the following Label and Field
combinations:| Label | Field | ------------------------------- | ----------------------- | | RF Risk Score | rf_a_risk | | RF Triggered Rules | rf_b_rules | | RF Very Malicious Evidence | rf_evidence_critical | | RF Malicious Evidence | rf_evidence_malicious | | RF Suspicious Evidence | rf_evidence_suspicious | | RF Unusual Evidence | rf_evidence_unusual | -
A restart of the Splunk instance will be required once the
installation has completed. - If you haven't already done so, enable the Enterprise
Security correlation search called "Threat Activity Detected"- In the Enterprise Security menu bar, click Configure → Content Management
- In the filter bar, type "Threat Activity Detected"
- Click the link 'Enable' to enable the correlation search
- Optionally, create a post install verification report. Run the
"Validate app deployment" report. It will perform a number of
tests, none of which should yield an error.- Go to Dashboards, Reports... → Reports.
- Run Validate app deployment.
Alternatively, you can download the Add-on using the Splunk Web
interface's "Find more apps online" feature. Steps 5 and onwards above
must still be completed.
Configuration
After installation, you will need to set up the Add-on for Recorded
Future to communicate with the Recorded Future API.
- Go to Configuration → Configuration.
- Select the Add-on Settings tab.
- Enter the API Key.
- Review the other tabs if additional configuration is required.
Upgrading from previous versions
The setup needs to be run after the upgrade. The API key (previously
called token in our documentation) will not carry over from the old
configuration. The same goes for proxy and loglevel configurations.
Upgrade from 3.x versions
Due to the extent of the changes between version 2 and 3 of the app
we recommend that you remove the app directory
($SPLUNK_HOME/etc/apps/TA-recorded_future) and make a fresh install
of the app.
If that is not possible proceed with the instructions below.
Files and directories that can be removed
The following files and directories can be removed
since they are not used anymore:
From the bin folder:
future
libfuturize
past
requests
rf_integrations
rf_splunk
rfapi
splunklib
get-rf-threatlists.py
rf_es_setup.py
verify_rf_app.pyFrom the local folder (if present):
commands.confFrom the local/data/ui/nav folder (if present):
default.xmlFiles that must be reviewed
Any file in the local folder is the result of a
local configuration. These have precedence over the
new settings shipped with the app. Review differences
in each file in the local folder compared to the
new default in the default folder and adjust if
required.
In particular correlation searches in
savedsearches.conf are likely to cause issues if
in place.
Further help
Recorded Future has a support web site:
Installing and configuring where
up-to-date ways of contacting support is also available.
Do not contact Splunk support about "Recorded Future for Splunk ES".
Release Notes
Version 4.0.5
## [4.0.5] - 2019-05-10
### Splunk Cloud Support Release
- Creation of risklists on cloud search head from indexed data (scheduled search)
- Creation of Splunk IOC lists from risklists (scheduled search)
- Creation of lookup table definitions (transforms)
- Collection of Splunk IOC lists from the download manager (threat intelligence download manager)
- Remove purging saved searches
- Modify search macro to remove threat_collection=direct.csv
Version 4.0.4
New in version 4.0:
- Adaptive responses have been added. These can be used to enrich any supported IOC while performing correlations or in ad-hoc mode.
- Added URL risk lists.
- Improved display of risk information in the Incidents review dashboard.
- Support for custom risk lists from Recorded Future Fusion was added.
- Support for retrieving alerts from Recorded Future was added. A dashboard to view alerts is available.
- Built-in help pages.
- New reports added to view "Latest updates of all risk lists", "All log events from the add-on" and a validation status.
- New options to customize access to Recorded Future's API (custom URL and optional SSL verification).
- Search head cluster synchronization.
Specific for version 4.0.4:
- Issue in Search Head clusters which prevented risk list updates with some configuration settings.
- Issue with retrieving the API key from the storage passwords.
- Increased timeout before entries are purged from the Threat Intelligence framework.
- Added server.conf
Version 3.2.2
The main addition with the 3.0 release is the addition of two new threat lists:
- domains: allows for correlation and enrichment on events from proxies, web application firewalls, DNS servers etc
- hashes: events from anti-virus systems, email gateways, file alteration monitors etc
In addition to the new risk lists there are many other improvements:
- significant performance enhancements
- support for clustering (in particular search head clusters)
- support for running on Windows based Splunk servers
- support for Splunk servers with a non standard management port.
Version 3.1.0 adds a Setup GUI where in addition to the Recorded Future API key it's easy to toggle debug logging or adding proxy information.
Version 3.2.0 adds a verification script that can be run to check that configuration and access needed for the app to work is present.
Version 3.1.1
The main addition with the 3.0 release is the addition of two new threat lists:
- domains: allows for correlation and enrichment on events from proxies, web application firewalls, DNS servers etc
- hashes: events from anti-virus systems, email gateways, file alteration monitors etc
In addition to the new risk lists there are many other improvements:
- significant performance enhancements
- support for clustering (in particular search head clusters)
- support for running on Windows based Splunk servers
- support for Splunk servers with a non standard management port.
Version 3.1.0 adds a Setup GUI where in addition to the Recorded Future API key it's easy to toggle debug logging or adding proxy information.