Cisco eStreamer eNcore Add-on for Splunk
Overview
Details
The following event types are supported with complete schema coverage through the eStreamer API specification for FMC version 6.2.
• Discovery Events
• Correlation and White List Events
• Impact Flag Alerts
• Intrusion Events
• Intrusion Event Packet Data
• User Activity
• Intrusion Event Extra Data
• Malware Events
• File Events
• Connection Events
This app was developed for and tested on Linux platforms only. Windows support is not currently available. Please check with Cisco for any change in status.
eStreamer eNcore for Splunk is a plugin based eStreamer client built from scratch in Python designed to deliver fully qualified event data to Splunk from Firepower 6.x platforms. If you have experienced problems getting the Cisco eStreamer for Splunk app version 2.2.1 and 2.2.2 working with Firepower 6.x you should move to this new application so that you can leverage the many improvements listed below.
eStreamer eNcore for Splunk is a Technical Add-on designed purely to collect data and be installed on a forwarder.
New in this solution:
- Complete API Coverage: Allows Splunk to collect all Firepower event data via the eStreamer API from Firepower Management Center version 6.x. Note: Will not work with Firepower version 5.x
- Plugin Architecture: Combines the Python ‘Client’ with a Splunk plugin to write data into Splunk in the Splunk format
- Dashboard Support: Backwards compatible with older solution’s dashboards A new version of the dashboard is available here: https://splunkbase.splunk.com/app/3663/
- Full Event Qualification: Client writes clear text for all of the eStreamer API’s referential model
- Comprehensively Documented: Detailed Operations Guide is available here: https://www.cisco.com/c/en/us/td/docs/security/firepower/630/api/eStreamer_enCore/eStreamereNcoreSplunkOperationsGuide_354.html
- Multi-Process Design: Will scale with additional compute resources to support event rates
- Cisco TAC Support Option Available: Optional TAC support service available with identical paid version purchased from Cisco. Order details: FP-SPLUNK-SW-K9 Free to use otherwise.
Resolved Issues in V 3.5.0
- An issue where Security Intelligence Category was not populating properly in Connection events. It is now populating correctly.
- An issue where the egress interface was populating with the id instead of the interface name in Intrusion events. It is now populating with the name.
- An issue where the fw_policy field name was appearing in Splunk file event output even though the firewall policy name is not part of the file event. The field name has been removed.
- An issue where the log reported a process poll timeout error even though the process is healthy and operating normally. The issue has been corrected.
- An issue where the name of the source user field was not populating correctly in connection events, intrusion events, file events, and malware events. The field is now populating correctly.
- An issue where the sid field in intrusion events was populating with an "internal sid" instead of the "rendered sid". If a user imports a custom rule file with rules that specify the sid, that sid is the "rendered sid" that appears in the FMC interface. The sid field in intrusion events is now populated by the rendered sid.
- An issue where the DNS Query field in connection event was not populating correctly. It is now populating correctly.
- An issue where in malware events, the virus name/detection name field was not populating correctly. It is now populating correctly.
- An issue where eNcore reported the following error: "UnicodeEncodeError: 'ascii' codec can't encode characters … ". The issue has been resolved.
- An issue where the packet payload output included layer 2-4 headers for TCP and UDP packets. The issue has been corrected and only the payload (without the layer 2-4 headers) appears in the payload field.
- An issue where in correlation events, the cs1 field (for access control policy name) was populating with the uuid instead of the name. The issue has been corrected.
Release Notes
Version 3.6.6
Version 3.6.5
Encore v3.6.5
Updated event_sec definition for connection events, event_sec now a duplicate of the first_packet_sec field since the first packet sec represents the creation time of the connection event. A new field, bookmark_event_sec, has been added that represents time which events are written from the from the FTD's to the eStreamer queue
Version 3.6.1
Encore 3.6.0 updates
- Added in support to configure payload output types, options are utf8, ascii and hex formats. To enable this change add the following parameter in the estreamer.conf file. By default hex is the output format and it does not need to be specified in the config.
- Added in support for FTD/syslog field mapping, props.conf contains all the transformations necessary to extract the available field names in the FTD syslog format. www.cisco.com/c/en/us/td/docs/security/firepower/Syslogs/b_fptd_syslog_guide/security-event-syslog-messages.html
Version 3.6.0
Encore 3.6.0 updates
- Added in support to configure payload output types, options are utf8, ascii and hex formats. To enable this change add the following parameter in the estreamer.conf file. By default hex is the output format and it does not need to be specified in the config.
- Added in support for FTD/syslog field mapping, props.conf contains all the transformations necessary to extract the available field names in the FTD syslog format. www.cisco.com/c/en/us/td/docs/security/firepower/Syslogs/b_fptd_syslog_guide/security-event-syslog-messages.html
Version 3.5.9
Encore 3.5.9 updates
- Added in support to configure payload output types, options are utf8, ascii and hex formats. To enable this change add the following parameter in the estreamer.conf file. By default hex is the output format and it does not need to be specified in the config.
- Added in support for FTD/syslog field mapping, props.conf contains all the transformations necessary to extract the available field names in the FTD syslog format. www.cisco.com/c/en/us/td/docs/security/firepower/Syslogs/b_fptd_syslog_guide/security-event-syslog-messages.html
Version 3.5.8
## Project Summary
This is the rewrite for the SourceFire eStreamer client.
The Cisco Event Streamer (also known as eStreamer) allows you to stream System intrusion,
discovery, and connection data from Firepower Management Center or managed device (also
referred to as the eStreamer server) to external client applications.
eStreamer responds to client requests with terse, compact, binary encoded messages – this
keeps it fast
eNcore is a new all-purpose client which requests all possible events from eStreamer, parses
the binary content and outputs events in various formats to support other SIEMs
*Updates for 3.5.8
*Bug fix - microseconds on pcap data now use the proper field name 'upacket_sec', and seconds use 'packet_sec'
Version 3.5.7
Version 3.5.7 Updates
*Added back aliasing for action/blocked fields
Version 3.5.6
Version 3.5.6 Updates
*Removed default disabling of the cisco:estreamer:data source type
*Removed duplicate aliasing for action/blocked fields
Version 3.5.4
- Fixed encore settings which did now allow for control the writing of metadata
- Added additional notes to readme to address performance improvements and the use of worker processes/batchSizes
Version 3.5.3
eNcore v3.5.3 resolves issues with previous v3.5.x versions where eNcore would crash under certain conditions.
All eNcore v3.5.x versions provide significant performance enhancements over pre-v3.5 versions.
Version 3.5.1
Version 3.5.0
eNcore version 3.5.0 features performance improvements – the ability to process a significantly higher event rate. This improvement requires no additional configuration on the part of the user.
However, if the eNcore platform has four or more cores, then additional performance improvement can be gained by adjusting a parameter in the configuration file, estreamer.conf. This parameter is called “workerProcesses” and is highlighted below:
{
"connectTimeout": 10,
"enabled": true,
"workerProcesses": 4,
"handler": {
--- rest of config file omitted ---
The highlighted line shows the “workerProcesses” to be set at 4. It can be set anywhere from 4-12, but with four or more cores, testing showed the best performance when set to 12.
See Details section for more Release Notes on Version 3.5.0