Add-on for ingesting DMARC XML aggregate reports into Splunk from an IMAP account or local directory, with mitigations against XML, GZ and ZIP-bombs.
|Search head||Yes||Yes||Install this add-on on your search head(s) where CIM compliance of DMARC aggregate reports is required|
|Indexer||Yes||No||This add-on should be installed on a heavy forwarder that does the index time parsing. There is no need to install this add-on on an indexer too.|
|Universal Forwarder||No||No||This add-on is not supported on a Universal Forwarder because it requires Python|
|Heavy Forwarder||Yes||Yes||Install this add-on on a heavy forwarder to ingest DMARC XML aggregate reports into Splunk.|
The following table lists support for distributed deployment roles in a Splunk deployment:
|Search head deployer||Yes||Install this add-on on your search head deployer to enable CIM compliance of DMARC aggregate reports on a Search Head Cluster|
|Cluster Master||No||This add-on should be installed on a heavy forwarder that performs parsing at index time. There is no need to install this add-on on an indexer too.|
|Deployment Server||Depends||This add-on can be (1) deployed unconfigured to a client or (2) deployed preconfigured with a directory input. Due to the encrypted credentials it cannot be deployed preconfigured for IMAP inputs.|
The TA-dmarc supports the following input modes:
TA-dmarc can watch a folder where you drop DMARC aggregate reports manually or otherwise.
It will read files with .xml, .zip or .xml.gz extention, ingest them into Splunk. Any invalid .xml, .zip or .xml.gz files are ignored.
TA-dmarc will leave files untouched in the directory: it uses internal checkpointing to skip files that have been previously ingested.
TA-dmarc can fetch DMARC aggregate report attachments from mails on an IMAP server. It will process attachments in .xml, .zip or xml.gz format and ingest them into Splunk.
TA-dmarc will leave the mails on the server: it uses internal checkpointing to skip mails that have been previously ingested.
This add-on handles the following file formats in which aggregate reports are delivered:
Mitigations are in place against:
From the XML sample below, these fields are created:
|XML field||Splunk field name||Value|
Besides the fields contained in the report, additional fields are mapped from the CIM Authentication datamodel, based on the XML sample below:
|Authentication datamodel field name||Value|
|signature||Use of mail-from domain example.com|
<?xml version="1.0" encoding="UTF-8" ?> <feedback> <report_metadata> <org_name>google.com</org_name> <email>firstname.lastname@example.org</email> <extra_contact_info>https://support.google.com/a/answer/2466580</extra_contact_info> <report_id>13190401177475355109</report_id> <date_range> <begin>1506988800</begin> <end>1507075199</end> </date_range> </report_metadata> <policy_published> <domain>example.com</domain> <adkim>r</adkim> <aspf>r</aspf> <p>none</p> <sp>none</sp> <pct>100</pct> </policy_published> <record> <row> <source_ip>192.0.2.78</source_ip> <count>1</count> <policy_evaluated> <disposition>none</disposition> <dkim>fail</dkim> <spf>fail</spf> </policy_evaluated> </row> <identifiers> <header_from>example.com</header_from> </identifiers> <auth_results> <spf> <domain>example.com</domain> <result>fail</result> </spf> </auth_results> </record> </feedback>
The DMARC-imap input saves checkpointing data in KVstore.
To see its contents:
If you want to reindex an entire mailbox, you can do so by deleting the TA-dmarc KVstore checkpointing data through this Splunk command:
|inputlookup ta_dmarc_checkpointer_lookup |search state!="*input=dmarc_imap, server=imap.gmail.com*" |outputlookup ta_dmarc_checkpointer_lookup`
If you want to reindex a single DMARC report, you can do so by deleting its corresponding record from KVstore:
| inputlookup ta_dmarc_checkpointer_lookup | search state!="*Report-ID: 3596274322387252907*" | outputlookup ta_dmarc_checkpointer_lookup
Reindexing a DMARC report from a directory input is left as an excercise for the reader.
The following software components are used in this add-on:
This is an open source project without warranty of any kind. No support is provided. However, a public repository and issue tracker are available at https://github.com/jorritfolmer/TA-dmarc
Fixed exception in directory input. (Thanks to Georgi Georgiev for providing a patch)
- Added support for Splunk 8.1. (Thanks to Aaron Myers for reporting the issue)
- Fixed exception when using add-on together with listenOnIPv6=yes (Thanks to gryphius for reporting this issue)
- Fixed Gmail POP3 issue
Fixed resolving of IP addresses in src field.
(Thanks to Martin Wright for reporting this issue)
Added support for Splunk 8.x and Python 3.x
- Fixed connection reset errors for large IMAP mailboxes (Thanks to Mike Kolk for the patch)
- Lowercase keys and values in policy_published and records (Thanks to Christopher G Andrews for reporting this issue)
- Added field in configure screen to set batch size for IMAP inputs
## Upgrading from 2.x to 3.x
The event output format changed to JSON. This is the new default because JSON is a better match for the structured aggregate reports than a flat key=value (KV) format. See for example issue #4 on Github. If you want your inputs to to keep using the KV format:
- Disable your inputs before upgrading
- Upgrade the add-on to version 3.x
- Edit your inputs and change the output format to KV
- Enable your inputs
Note that KV format output is likely to be deprecated in future versions, and enhancements like multiple XSD validation aren't available in KV output format.
- Added POP3 support
- Added initial DKIM signature checking
- Added validations against multiple XSDs
- Added XSD validation results in a new field
- Added JSON output to better preserve report structure, issue #4.
- Lots of other small issues, see https://github.com/jorritfolmer/TA-dmarc/CHANGELOG.md
- Added DMARC XML validation against DMARC XSD
- Fixed reading reports with absolute paths from zip files. Thanks to Steve Myers for the bug report.
- Removed forgotten use_ssl from schema following Splunk App Cert Admin review
- Store individual records in KVstore instead of using a single serialized Python set.
- Corrected misconfigured key in KVstore collection
Addressed concerns from Splunk Cert Admin:
- enforced use of TLS for IMAP input, by removing TLS checkbox
- ensure tmp directories are always cleaned up, by using try finally for both directory and imap inputs
- ensure the add-on won't write outside of splunk designated directories by implementing KVstore checkpointing for DMARC directory inputs instead of using file ops to bad/, done/ or tmp/ dirs.
NOTES BEFORE UPGRADING FROM v1.x TO 2.0.0:
Because of the changes to KVstore logic, the add-on will re-index every report on IMAP, leading to duplicate events.
- Corrected issues from Splunk Cert Admin feedback
Added non-multipart support for mail messages on IMAP
Splunk AppInspect evaluates Splunk apps against a set of Splunk-defined criteria to assess the validity and security of an app package and components.
As a Splunkbase app developer, you will have access to all Splunk development resources and receive a 10GB license to build an app that will help solve use cases for customers all over the world. Splunkbase has 1000+ apps and add-ons from Splunk, our partners and our community. Find an app or add-on for most any data source and user need, or simply create your own with help from our developer portal.