This app pulls down lookups from the Hurricane Labs getThreats API. Corresponding threatlist inputs then ingest
the lookup data into the appropriate threatlist.
Go to 'Apps' < 'Manage Apps'
Find the 'Hurricane Labs Threat Intelligence Feed' app, and then click 'Launch App'
Once taken to the app, click on the 'Setup' link in the navigation
Enable the threatlist inputs. By default these are set to disabled = true. You can enable these one of two ways:
inputs.confmisp_IPDST, misp_MD5SUMs, misp_SHA1SUMs, misp_SHA256SUMs, misp_URL, misp_Domains(Non-Cloud Only)
3. Enable the scripted input in inputs.conf.
- By default the scripted input runs every hour. If it is decided to change this interval, and your environment is
clustered, then it is not recommended to use CRON scheduling for the scripted input.
- Confirm the scripted input has run by searching index=_internal "ExecProcessor" "threatlist_handler.py". If it
successfully ran you will get results back. Ensure log_level is INFO.
- If you see no results, you can force the scripted input to run by running debug/refresh.
(Cloud Only or Non-Cloud)
3. Enable the Saved Search HLThreatIntelligence Populator to start populating your Threat Lists as
Scripted Inputs will be removed on Splunk Cloud.
- It is recommended to schedule the saved search to run hourly as the default retention on the Threat Intelligence Downloads
is set to 1 hour by default.
- Once the Saved Search has run, you can confirm it is running successfully by running the following search:
index=_internal sourcetype=scheduler savedsearch_name="HLThreatIntelligence Populator" | table _time status
- You can also manually run | hlthreatintel to make sure everything works as expected. Errors from running this
search should appear in both the messages window as well as the results output. If no results are returned then
everything should be working as expected.
1.0.6 or Earlier1.0.7 the setup process has changed slightly as the setup.xml file has been removed in the default Setup Process section in this README.1.0.3 or EarlierIf you are upgrading from 1.0.3 or earlier version you will need to either do one of
two things as the setup page functionality has changed:
1. Re-enter the API key on the setup page.
- OR -
2. Change the stanza name in the passwords.conf file to be [credential::api_key:]. After a debug/refresh
the new API key should be seen.
The reason for the above is the app up to version 1.0.3 utilized the standard Splunk setup page which made it
impossible to update an API key while also overwriting the old API key in passwords.conf. This update fixes
this issue, but because of this change one of the above changes needs to be made.
By default, the retention for the Threat Intelligence Downloads provided in this app are set to -1h. If you adjust
this, consider adjusting the schedule for the scripted input threatlist_handler.py(non-cloud) or saved search
HLThreatIntelligence Populator (cloud).
You can modify the Threat Intelligence Download settings by going to Configure < Data Enrichment < Intelligence Downloads
in Enterprise Security, and then finding the threat downloads (misp_IPDST, misp_MD5SUMs, misp_SHA1SUMs,
misp_SHA256SUMs, misp_URL, misp_Domains).
Further information regarding Threat Intelligence Download retention is provided via the Splunk Documentation here:
https://docs.splunk.com/Documentation/ES/5.3.1/Admin/Changethreatintel
The above link contains the following information:
Remove threat intelligence from the KV Store collections in Splunk Enterprise Security based on the date that the
threat intelligence was added to Enterprise Security.
The default maximum age is -30d for 30 days of retention in the KV Store. To remove the data more often, use a
smaller number such as -7d for one week of retention. To keep the data indefinitely, use a blank field. However,
if the KV Store collection is stored indefinitely, the .csv files that result from lookup-generating searches can
grow large enough to impact search head cluster replication performance. If you manually delete the data from the
.csv file, the maximum age timer does not reset based on the edit date, and the data is still removed from the
KV Store after the maximum age expires.
If the threat intelligence source is not a TAXII feed, define the maximum age of the threat intelligence. This
field is not used for TAXII feeds.
- From the Enterprise Security menu bar, select Configure > Data Enrichment > Intelligence Downloads.
- Select an intelligence source.
- Change the Maximum age setting using a relative time specifier.
- Enable the retention search for the collection.
- From the Splunk platform menu bar, select Settings and click Searches, reports, and alerts.
- Search for "retention" using the search filter.
- Enable the retention search for the collection that hosts the threat source. All retention searches
are disabled by default.Configure threat intelligence file retention):As of v. 1.0.4 you can set up a proxy specifically for these feeds. Just modify feed_proxy.conf
For detailed logging you can search index=_internal source="*sa_hlthreatintelligencefeed.log" which will
tell you if anything goes wrong with either the Scripted Input or Custom Command.
If you use the above search, but see the following:
[{'type': 'ERROR', 'text': 'Could not find object id=:api_key:', 'code': None}] or a 403 forbidden, and
are upgrading from version 1.0.3 or earlier, then you need to re-enter your API key.
See the Upgrading section in this README for more information.
Once you've configured everything, you should first check to see that your threatlist shows up on the
"Threat Artifacts" dashboard in Enterpise Security (Security Intelligence < Threat Intelligence < Threat Artifacts).
If you do not see your threatlist on the "Threat Artifacts" dashboard then it is possible it being omitted as the top
panel specifically (Threat Overview) appends multiple threat intel lists together (file_intel, ip_intel etc.) and
limits the output of those lookups to 10,000 results each. This does not mean your threatlist has not been added.
You can confirm everything is populating as expected by running the following searches:
| inputlookup http_intel | stats count by threat_key
- http_intel will have misp_URLs
| inputlookup ip_intel | stats count by threat_key
- ip_intel will have misp_Domains, and misp_IPDST
| inputlookup file_intel | stats count by threat_key
- file_intelwill have misp_MD5SUMs, misp_SHA1SUMs, and misp_SHA256SUMs
If you still do not see results confirm that the necessary threatintel inputs are enabled in ES:
- Configure < Data Enrichment < Intelligence Downloads
1.0.8
- Updated API URL
1.0.7
- Empty <label></label> in default/data/ui/views/setup.xml caused Python to throw an error. This has been fixed.
- Removed default/setup.xml as Splunk Cloud no longer supports it.
- Must be installed on Splunk 8 or greater
1.0.6
- Fix of potential bug in JavaScript preventing the Setup page from loading correctly
1.0.5
- Custom command hlthreatintel and saved search HLThreatIntelligence Populator added for Splunk Cloud usage.
- Updated Setup Page
1.0.4
- Added proxy support
1.0.3
- Simplified installation process. No longer uses a KVStore for managing API endpoints
1.0.2
- Breaking change fixed, installation process updated to reflect necessary changes
v1.2.5
- Added Threat Intel Dashboard
Fixed an issue with max_age which causes indicators to prematurely expire off before they are actually removed from our API.
v1.2.3
- Removed Search Head JavaScript dependencies in Setup Page files to meet new Splunk Cloud vetting requirements.
Added a new feed called hl-current-threats. This will contain indicators for current-event threats where indicators are high severity for a short period of time, but are not good indicators of compromise later on. These will be expired off more quickly than normal indicators. An example of this would be a new RCE vulnerability.
1.1.1: Added success message when feed is retrieved.
Version 1.0.7
- Empty <label></label> in default/data/ui/views/setup.xml caused Python to throw an error. This has been fixed.
- Removed default/setup.xml as Splunk Cloud no longer supports it.
- Must be installed on Splunk 8 or greater
1.0.6
- Fix of potential bug in JavaScript preventing the Setup page from loading correctly
If you are upgrading this app from an earlier version (1.0.3 or earlier) you will need to either do one of
two things as the setup page functionality has changed:
1. Re-enter the API key on the setup page.
- OR -
2. Change the stanza name in the passwords.conf file to be [credential::api_key:]. After a debug/refresh
the new API key should be seen.
The reason for the above is the app up to version 1.0.3 utilized the standard Splunk setup page which made it
impossible to update an API key while also overwriting the old API key in passwords.conf. This update fixes
this issue, but because of this change one of the above changes needs to be made.
v. 1.0.5
- Custom command hlthreatintel and saved search HLThreatIntelligence Populator added for Splunk Cloud usage.
- Updated Setup Page
- Better logging. You can view the app's logs by searchingindex=_internal source="*sa_hlthreatintelligencefeed.log"
1.0.3
- Simplified installation process. No longer uses a KVStore for managing API endpoints.
1.0.2
- Breaking change fixed, installation process updated to reflect necessary changes
IMPORTANT: If you happened to download version 1.0.1, please update the app to 1.0.2 to ensure it works properly.
This app pulls down threatlists. It does this by looking in a KVStore called threatlist_urls. This KVStore
defines the URL to download the file from and the name of the file that will be generated. Instructions on how to
populate this KVStore are below.
Purchasing an API Key
- Go here https://www.hurricanelabs.com/threat-intelligence-feed to request an API key.
As a Splunkbase app developer, you will have access to all Splunk development resources and receive a 10GB license to build an app that will help solve use cases for customers all over the world. Splunkbase has 1000+ apps from Splunk, our partners and our community. Find an app for most any data source and user need, or simply create your own with help from our developer portal.