**Datalake2SplunkV3** allows you to get Datalake data into Splunk for Orange Cyberdefense Datalake customers. Datalake2splunk modular input builds a bulk-search using defined parameters and contacts Datalake API. It transforms the received JSON into a csv used as a lookup or can index Data into a custom index. The IOC are available by using the lookups. Threat matches can be found by correlating with events log. Enterprise Security Threat Intelligence automates the threat matches job. # Temporary notes // Migration steps : Because some atom_types will be renamed or deleted in V3 of Datalake, query hashes need to be changed when put in this new app. New inputs need to be created inside Datalake2Splunk-V3. It is recommended to uninstall the previous app once Datalake migration will be over (planned for February 17, 2025). Datalake2Splunk will be archived once Datalake has migrated to V3. # Prerequisites - This app has been developed and tested with Python 3.7. and 3.9 so the minimal Splunk version needed is 9.1. - Internet access. - Bulk-search account on Datalake. # General Usage ## Configuration - Account: you can whoose a name for the account and enter your Long Term TOken generated on Datalake profile. - If needed, configure Proxy and provide your proxy configuration in order to allow internet access to Datalake2Splunk ## Input - *Name*: a name for your input - *Interval*: time in seconds between 2 fetchs. The process can take a while depending on the data volume, so we do not recommend under 1800. This field also support a cron schedule - *Index*: If there is a need to index or if it is Splunk Cloud Classic Experience choose the right index otherwise whatever index you choose, no data will be indexed from this input - *API URL*: automatically filled with the latest Datalake URL but customizable for those who need preprod access - *Query Hash*: enter your datalake query hash obtained via the URL /gui/search?query_hash=<> - *Token*: choose which Token (created in configuration) will be used to get Datalake information On Splunk Cloud Classic experience, the data inputs are managed on the Input Data Manager which is not the Search Head - *Storage Type*: choose if the IOCs should be indexed. Required for Splunk Cloud Classic Experience Lookups : The app creates csv named after atom_types Custom Lookup : a lookup named datalake_ is created containing all indicators. ## Splunk Cloud Classic Experience IOCs are first downloaded by the IDM and then sent to the Indexers to be indexed. When indexed, saved searches have to be enabled on the Search Head to aggregate the IOCs in lookups. Depending on what IOC needs to be used in lookups, the following search can be enabled:

Datalake - Build Datalake  lookup

In addition, the macro *datalake-index* has be to configured with the index name where the IOCs are stored. ## Use Datalake Intelligence Lookups containing your regularly downloaded data are located under $SPLUNK_HOME/etc/apps/TA-datalake2splunk/lookups and named: datalake_.csv All Lookup Definitions have been previously created and included in TA-Datalake2splunk (transforms.conf) with read permission for other apps. ### with Enterprise Security From Enterprise Security main page click Configure -> Threat Intelligence Inside Threat Intelligence Sources page Click New and local to create a new intelligence entry and fill as follow: - *Name* and *Description*: whatever name you choose - *Type*: threatlist by default, (you can put “malware” type for datalake_file) - *URL* is local path to lookup definition so lookup://datalake_ - *Weight* is a risk_score multiplier. 1 to have no impact on assets and identities. - *Interval* (in seconds) no need to have a smaller interval than what’s configured in the input. This control the frequency at which ES will update Intelligence KV Stores - Use *Maximum age* to control when splunk will delete the data from this lookup in Threat_Intelligence KV Stores. If you have very frequent pull, maximum age should remain low. - Keep “Is Threat Intelligence” checked On the Parsing tab you need to customize for fields that are not already present in columns. Atom values are on column 1 and 3 for lookups or column 2 for custom lookups. For instance for datalake_email lookup put src_user:$1 to be able to match src_user For datalake_certificate certificate_serial:$1 so certificates atom_values are perceived as certificates serials For a custom lookup my_type_intel:$2. Threat Intelligence types : https://help.splunk.com/en/splunk-enterprise-security-8/administer/8.1/threat-intelligence/supported-types-of-threat-intelligence-in-splunk-enterprise-security This must be done for each type accepted by ES: ip, domain, url, file, email, certificate. Next step is to check if ES has ingested the data. There is a KV store for each type. ip_intel for domain and ip, email_intel for email, http_intel for url, file_intel for file, certificate_intel for certificates. #### Enrich When indexed and transformed into lookup, a lot of data is stacked in “description” field for this is the only way to add value to each line and will be parsed during Matching searches in order to add value to Threat Activity data later. You can use macro `datalake_parser(description)` to extract the data. If you want to fully exploit collected data, you might want to modify the on-boarded **Threat Activity** search. From Enterprise Security main page Filter Correlationsearch in *DA-ESS-Threat_Intelligence* App and look for “Threat Activity Detected” Edit the Search field At the end add:

| `datalake_parser(threat_description)`

### Without ES Intelligence data is searchable by any app. You can call data thanks to the lookup definitions. Such as

|inputlookup datalake_domain
|inputlookup datalake_url

Now you’ll need to correlate your events with your logs. Either by using lookup command or inputlookup and check for a match. You can easily get detailed data and correlate by using macro `datalake_parser` # Reports and Dashboards *Datalake IOC Stats* offers insight on the content of the lookups such as most frequent threat types or sources. This is the base work to have a macro overview over IOC collection and a Splunk developer can edit those dashboards. # Changes made since former add-on : ## Features New fields available: threat_entities, sighting_sources and last_positive_sighting ## Configuration and Input - The Account page now displays Long Term Token as intended - It is now possible to configure a cron schedule instead of just seconds for interval - The API URL is customisable in the input (used to request preproduction environment) ## Script - Automatically constructed Lookups now contain atom_type and atom_value columns - Better logging to ensure troubleshooting for each input ## Parsing - When ingested, data now have a new sourcetype *datalake:indicators* that includes new parsing for multivalue version of all attributes such as threat_source, threat_entity and sighting_source - Macro datalake_parser updated to do the same parsing as the props does - Updated savedsearches for lookup creation when data is indexed to reflect updated atom_types and new attributes Not yet tested on Splunk 10. But since it runs with Pyton 3.9 it is theoretically compatible.