icon/x Created with Sketch.

Splunk Cookie Policy

We use our own and third-party cookies to provide you with a great online experience. We also use these cookies to improve our products and services, support our marketing campaigns, and advertise to you on our website and other websites. Some cookies may continue to collect information after you have left our website. Learn more (including how to update your settings) here.
Accept Cookie Policy

We are working on something new...

A Fresh New Splunkbase
We are designing a New Splunkbase to improve search and discoverability of apps. Check out our new and improved features like Categories and Collections. New Splunkbase is currently in preview mode, as it is under active development. We welcome you to navigate New Splunkbase and give us feedback.
Log4Shell Vulnerability: Information and guidance for you. Get resources.

Accept License Agreements

This app is provided by a third party and your right to use the app is in accordance with the license provided by that third-party licensor. Splunk is not responsible for any third-party apps and does not provide any warranty or support. If you have any questions, complaints or claims with respect to this app, please contact the licensor directly.

Thank You

Downloading Microsoft Cloud Services Event Hub True Fashion Add-on for Splunk
SHA256 checksum (microsoft-cloud-services-event-hub-true-fashion-add-on-for-splunk_104.tgz) 9bd22df5f968629e637801bf25d9eb1fd79d434a642febfd3f945422eda02760 SHA256 checksum (microsoft-cloud-services-event-hub-true-fashion-add-on-for-splunk_103.tgz) f3653754be540f6d2114250002fbe5c8de7e3a4c65121f5a7f87ca475e657a71
To install your download
For instructions specific to your download, click the Details tab after closing this window.

Flag As Inappropriate

splunk

Microsoft Cloud Services Event Hub True Fashion Add-on for Splunk

Splunk Cloud
This app is NOT supported by Splunk. Please read about what that means for you here.
Overview
Details
This add-on provides natural access to field names being delivered through Azure Event Hubs in conjunction with Splunk Add-on for Microsoft Cloud Services.

No extra field extractions or CIM compatibility is done.

Instead it uses SEDCMDs to re-format Event Hub message format structure safely. This way Splunk can keep JSON format parsing intact (KV_MODE=json) while overhead in processing and field naming does get reduced. Because of minor changes how data is being indexed upcoming schema changes (new fields) by vendor Microsoft are automatically supported.
Using this add-on will give the following advantages:

- Direct field name access as designed by the original format on the backend, prior data was sent to Event Hub (no more body.properties or body.records.properties prefix prefix in field names for example)
- Cuts off unnecessary fields from Event Hub messages (x-opt-sequence-number, x-opt-offset and x-opt-enqueued-time)
- Ingested data via Event Hub is easier interpreted because of original field names. SPLs and detections are easier to write because field names are kept in original format.

Supports Azure Event Hubs messages sent by:
* Microsoft 365 Defender Streaming-API - Advanced Hunting telemetry like: DeviceAlertEvents DeviceProcessEvents DeviceNetworkInfo DeviceLogonEvents DeviceEvents DeviceTvmSoftwareVulnerabilitiesKB etc.
* Azure Blade monitor logs - Azure platform telemetry like: AuditLogs SignInLogs NonInteractiveUserSignInLogs ServicePrincipalSignInLogs ManagedIdentitySignInLogs RiskyUsers UserRiskEvents etc.

Microsoft Cloud Services Event Hub True Fashion Add-on for Splunk

Introduction

This add-on provides natural access to field names being delivered through Azure Event Hubs in conjunction with Splunk Add-on for Microsoft Cloud Services.

No extra field extractions or CIM compatibility is done.

Instead it uses SEDCMDs to re-format Event Hub message format structure safely. This way Splunk can keep JSON format parsing intact (KV_MODE=json) while overhead in processing and field naming does get reduced. Because of minor changes how data is being indexed upcoming schema changes (new fields) by vendor Microsoft are automatically supported.
Using this add-on will give the following advantages:

  • Direct field name access as designed by the original format on the backend, prior data was sent to Event Hub (no more body.properties or body.records.properties prefix prefix in field names for example)
  • Cuts off unnecessary fields from Event Hub messages (x-opt-sequence-number, x-opt-offset and x-opt-enqueued-time)
  • Ingested data via Event Hub is easier interpreted because of original field names. SPLs and detections are easier to write because field names are kept in original format.

Supports Azure Event Hubs messages sent by: * Microsoft 365 Defender Streaming-API - Advanced Hunting telemetry like: DeviceAlertEvents DeviceProcessEvents DeviceNetworkInfo DeviceLogonEvents DeviceEvents DeviceTvmSoftwareVulnerabilitiesKB etc. * Azure Blade monitor logs - Azure platform telemetry like: AuditLogs SignInLogs NonInteractiveUserSignInLogs ServicePrincipalSignInLogs ManagedIdentitySignInLogs RiskyUsers UserRiskEvents etc.

How is this different from other add-on 'Splunk Add on for Microsoft Azure'?

Splunk Add on for Microsoft Azure

  • The above is used to ingest data from Azure Event Hubs via classic REST-API.
  • Your fields will ALREADY look nice when you use this add-on. Nothing more is required and you will not need Microsoft Cloud Services Event Hub True Fashion Add-on for Splunk.
  • But this approach is deprecated and has no official Splunk CIM support.

How is this different from other add-on 'Splunk Add-on for Microsoft Cloud Services' (MSCS)?

Splunk Add-on for Microsoft Cloud Services

  • The above is used to ingest data from Azure Event Hubs via modern Azure AD app with Event Hub Reader permission.
  • Your fields will NOT LOOK nice using sourcetype: mscs:azure:eventhub
  • You may now use further add-ons to improve field naming, for example "Microsoft Defender Advanced Hunting Add-on for Splunk" using sourcetype mscs:azure:eventhub:defender:advancedhunting or use this add-on here you are looking at which introduces sourcetype mscs:azure:eventhub:truefashion

Difference between Splunk Add-on for Microsoft Cloud Services version 4.3.3 and version 4.4.0?

  • In version 4.4.0 it has been natively improved to remove body and Event Hub messages fields x-opt-sequence-number, x-opt-offset and x-opt-enqueued-time - overall nesting has been reduced (documented in MSCS Release History)
  • Still properties.* fieldnames are used which you can use this add-on for to remove prefix

How is this different from other add-on 'Microsoft Defender Advanced Hunting Add-on for Splunk'?

Microsoft Defender Advanced Hunting Add-on for Splunk

  • The above is used to extract and normalize data from Microsoft 365 Defender Streaming-API into Splunk CIM and it is supported by Splunk in conjunction with 'Splunk Add-on for Microsoft Cloud Services' (MSCS).
  • But it does not map out every field to CIM, does assumptions and does not fix the core field naming prefixes during ingestion at index time.
  • Means this will not help you in case of getting the original field names better readable (even when MSCS v4.3.3+ parses body.records in loop now).

So why and when should I use 'Microsoft Cloud Services Event Hub True Fashion Add-on for Splunk'?

  • If you want to ingest any data from Azure Event Hubs using Splunk Add-on for Microsoft Cloud Services (MSCS) while keeping the original field names intact without doing any further modifications and/or extractions with the data.
  • This simply gets you the original field names. For example to build your own index / summary index / custom CIM.
  • In short: it simply removes body.properties and body.records.properties which makes fields much more readable, without any fancy transformations.

Installation

  1. Configure Microsoft Defender for Endpoint to stream Advanced Hunting events to an Azure Event Hub or configure Azure Blade diagnostic events to be forward to an Event Hub. See: https://docs.microsoft.com/en-us/microsoft-365/security/defender-endpoint/raw-data-export-event-hub?view=o365-worldwide
    You can use multiply sourcetypes of data in the same Event Hub. This add-on re-renders all Event Hub messages safely. It is best-practice to separate Event Hubs per each sourcetype though.

  2. Install this add-on on your Search Heads, Indexers and Heavy Forwarders (if part of your data collection topology)

  3. Install and use this Splunk add-on to ingest data from Azure Event Hub: Splunk Add-on for Microsoft Cloud Services version 4.3.3+

  4. When setting the up the input, enter sourcetype: mscs:azure:eventhub:truefashion
    Note: Starting with version 4.4.0 of Splunk Add-on for Microsoft Cloud Services it is no longer possible to manually set a sourcetype, which is not part of the dropdown list.
    Select sourcetype: mscs:azure:eventhub and edit file
    Splunk_TA_microsoft-cloudservices/local/inputs.conf manually to replace
    sourcetype = mscs:azure:eventhub
    with
    sourcetype = mscs:azure:eventhub:truefashion

  5. Verify that raw data is arriving by running the following search: index=* sourcetype="mscs:azure:eventhub:truefashion"

Changelog

  • Version 1.0.4: Added support for Splunk Add-on for Microsoft Cloud Services version v4.4.0 (includes backwards compatibility for v4.3.3)
  • Version 1.0.3: First public release based on Splunk Add-on for Microsoft Cloud Services version v4.3.3

Data Sample (without and with this add-on)

Original Event Hub message format in Splunk Add-on for Microsoft Cloud Services 4.3.3:

{
    "body": {
        "time": "2022-07-07T12:45:18.3907246Z",
        "tenantId": "a4547165-6daa-4e9e-b2c3-8e7fb7142e4d",
        "operationName": "Publish",
        "category": "AdvancedHunting-DeviceNetworkEvents",
        "properties": {
            "RemotePort": 63967,
            "RemoteIP": "127.0.0.1",
            "Protocol": "Tcp",
            "LocalIP": "127.0.0.1",
            "LocalPort": 63968,
            "RemoteUrl": "",
            "LocalIPType": "Loopback",
            "RemoteIPType": "Loopback",
            "AdditionalFields": null,
            "ActionType": "ConnectionSuccess",
            "InitiatingProcessVersionInfoCompanyName": "Microsoft Corporation",
            "InitiatingProcessVersionInfoProductName": "Microsoft\u00ae Windows\u00ae Operating System",
            "InitiatingProcessVersionInfoProductVersion": "10.8048.22439.1065",
            "InitiatingProcessVersionInfoInternalFileName": "MsSense.exe",
            "InitiatingProcessVersionInfoOriginalFileName": "MsSense.exe",
            "InitiatingProcessVersionInfoFileDescription": "Windows Defender Advanced Threat Protection Service Executable",
            "InitiatingProcessFolderPath": "c:\\program files\\windows defender advanced threat protection\\mssense.exe",
            "InitiatingProcessFileSize": 472368,
            "InitiatingProcessMD5": "f23bada6ff4f6f9bf4c5342093156855",
            "InitiatingProcessSHA256": "cb993f887eff06aec7bcfe5fc7f14e890adb18871ced1e8edbb57e858e126978",
            "InitiatingProcessSHA1": "73285ffb57122d7822a979b909cf41b999b1cc6e",
            "InitiatingProcessAccountSid": "S-1-5-18",
            "InitiatingProcessAccountDomain": "nt authority",
            "InitiatingProcessAccountName": "system",
            "InitiatingProcessAccountUpn": null,
            "InitiatingProcessAccountObjectId": null,
            "InitiatingProcessCreationTime": "2022-07-07T11:02:16.5447066Z",
            "InitiatingProcessId": 2988,
            "InitiatingProcessFileName": "MsSense.exe",
            "InitiatingProcessCommandLine": "\"MsSense.exe\"",
            "InitiatingProcessParentCreationTime": "2022-07-07T11:01:36.8066095Z",
            "InitiatingProcessParentId": 672,
            "InitiatingProcessParentFileName": "services.exe",
            "InitiatingProcessIntegrityLevel": "System",
            "InitiatingProcessTokenElevation": "TokenElevationTypeDefault",
            "DeviceId": "58faa2ef39046f86543ec7fb8849dfd54dd014fa",
            "AppGuardContainerId": null,
            "MachineGroup": null,
            "Timestamp": "2022-07-07T12:44:56.208162Z",
            "DeviceName": "dcwin2016.lab.local",
            "ReportId": 7251
        }
    },
    "x-opt-sequence-number": 6632,
    "x-opt-offset": "21481563656",
    "x-opt-enqueued-time": 1657198004612
}

Original Event Hub message format in Splunk Add-on for Microsoft Cloud Services 4.4.0:
(body and x-opt- fields are now built-in removed but properties. array fields still exists as extra dimension)

{
    "time": "2022-07-21T15:16:03.6719584Z",
    "tenantId": "a4547165-6daa-4e9e-b2c3-8e7fb7142e4d",
    "operationName": "Publish",
    "category": "AdvancedHunting-DeviceEvents",
    "properties": {
        "AccountSid": null,
        "AccountDomain": null,
        "AccountName": null,
        "LogonId": null,
        "FileName": null,
        "FolderPath": null,
        "MD5": null,
        "SHA1": null,
        "FileSize": null,
        "SHA256": null,
        "ProcessCreationTime": null,
        "ProcessTokenElevation": null,
        "RemoteUrl": null,
        "RegistryKey": null,
        "RegistryValueName": null,
        "RegistryValueData": null,
        "RemoteDeviceName": null,
        "FileOriginIP": null,
        "FileOriginUrl": null,
        "LocalIP": null,
        "LocalPort": null,
        "RemoteIP": null,
        "RemotePort": null,
        "ProcessId": null,
        "ProcessCommandLine": null,
        "AdditionalFields": null,
        "ActionType": "NtProtectVirtualMemoryApiCall",
        "InitiatingProcessVersionInfoCompanyName": "Microsoft Corporation",
        "InitiatingProcessVersionInfoProductName": "Microsoft\u00ae .NET Framework",
        "InitiatingProcessVersionInfoProductVersion": "4.8.4320.0",
        "InitiatingProcessVersionInfoInternalFileName": "mscorsvw.exe",
        "InitiatingProcessVersionInfoOriginalFileName": "mscorsvw.exe",
        "InitiatingProcessVersionInfoFileDescription": ".NET Runtime Optimization Service",
        "InitiatingProcessFolderPath": "c:\\windows\\microsoft.net\\framework\\v4.0.30319\\mscorsvw.exe",
        "InitiatingProcessFileName": "mscorsvw.exe",
        "InitiatingProcessFileSize": 125872,
        "InitiatingProcessMD5": "d7365b80e8951ddc95f3a8e3ac01d37d",
        "InitiatingProcessSHA256": "3e5099f573601926e59862fba2495974688e72677c73f10e4c99e26a76cdcf37",
        "InitiatingProcessSHA1": "0636347981cb05b74859ce7c841753da90ce679a",
        "InitiatingProcessLogonId": 999,
        "InitiatingProcessAccountSid": "S-1-5-18",
        "InitiatingProcessAccountDomain": "nt authority",
        "InitiatingProcessAccountName": "system",
        "InitiatingProcessAccountUpn": null,
        "InitiatingProcessAccountObjectId": null,
        "InitiatingProcessCreationTime": "2022-07-21T15:13:22.3631034Z",
        "InitiatingProcessId": 6504,
        "InitiatingProcessCommandLine": "mscorsvw.exe -StartupEvent 218 -InterruptEvent 0 -NGENProcess 20c -Pipe 214 -Comment \"NGen Worker Process\"",
        "InitiatingProcessParentCreationTime": "2022-07-21T15:13:22.3092735Z",
        "InitiatingProcessParentId": 1368,
        "InitiatingProcessParentFileName": "ngen.exe",
        "DeviceId": "c3f1bb7cb3e12d004b0d317c0842d54f9f51c653",
        "AppGuardContainerId": "",
        "MachineGroup": null,
        "Timestamp": "2022-07-21T15:13:22.426665Z",
        "DeviceName": "clientwin10.lab.local",
        "ReportId": 1974
    }
}

Using this add-on data will be re-formatted using SEDCMD so Splunk can parse it easier:

{
        "time": "2022-07-07T12:45:18.3907246Z",
        "tenantId": "a4547165-6daa-4e9e-b2c3-8e7fb7142e4d",
        "operationName": "Publish",
        "category": "DeviceNetworkEvents",
        "RemotePort": 63967,
        "RemoteIP": "127.0.0.1",
        "Protocol": "Tcp",
        "LocalIP": "127.0.0.1",
        "LocalPort": 63968,
        "RemoteUrl": "",
        "LocalIPType": "Loopback",
        "RemoteIPType": "Loopback",
        "AdditionalFields": null,
        "ActionType": "ConnectionSuccess",
        "InitiatingProcessVersionInfoCompanyName": "Microsoft Corporation",
        "InitiatingProcessVersionInfoProductName": "Microsoft\u00ae Windows\u00ae Operating System",
        "InitiatingProcessVersionInfoProductVersion": "10.8048.22439.1065",
        "InitiatingProcessVersionInfoInternalFileName": "MsSense.exe",
        "InitiatingProcessVersionInfoOriginalFileName": "MsSense.exe",
        "InitiatingProcessVersionInfoFileDescription": "Windows Defender Advanced Threat Protection Service Executable",
        "InitiatingProcessFolderPath": "c:\\program files\\windows defender advanced threat protection\\mssense.exe",
        "InitiatingProcessFileSize": 472368,
        "InitiatingProcessMD5": "f23bada6ff4f6f9bf4c5342093156855",
        "InitiatingProcessSHA256": "cb993f887eff06aec7bcfe5fc7f14e890adb18871ced1e8edbb57e858e126978",
        "InitiatingProcessSHA1": "73285ffb57122d7822a979b909cf41b999b1cc6e",
        "InitiatingProcessAccountSid": "S-1-5-18",
        "InitiatingProcessAccountDomain": "nt authority",
        "InitiatingProcessAccountName": "system",
        "InitiatingProcessAccountUpn": null,
        "InitiatingProcessAccountObjectId": null,
        "InitiatingProcessCreationTime": "2022-07-07T11:02:16.5447066Z",
        "InitiatingProcessId": 2988,
        "InitiatingProcessFileName": "MsSense.exe",
        "InitiatingProcessCommandLine": "\"MsSense.exe\"",
        "InitiatingProcessParentCreationTime": "2022-07-07T11:01:36.8066095Z",
        "InitiatingProcessParentId": 672,
        "InitiatingProcessParentFileName": "services.exe",
        "InitiatingProcessIntegrityLevel": "System",
        "InitiatingProcessTokenElevation": "TokenElevationTypeDefault",
        "DeviceId": "58faa2ef39046f86543ec7fb8849dfd54dd014fa",
        "AppGuardContainerId": null,
        "MachineGroup": null,
        "Timestamp": "2022-07-07T12:44:56.208162Z",
        "DeviceName": "dcwin2016.lab.local",
        "ReportId": 7251
}

Support

While this app is not formally supported, the developer can be reached at mail@grobendirk.de
Responses are made on a best effort basis. Feedback is always welcome and appreciated!

Contact

Authored by Dirk Groben (mail@grobendirk.de)

Release Notes

Version 1.0.4
July 21, 2022
  • added support for Splunk Add-on for Microsoft Cloud Services version v4.4.0 & v4.5.0 (includes backwards compatibility for v4.3.3)
Version 1.0.3
July 15, 2022
  • first public version released

Subscribe Share

Are you a developer?

As a Splunkbase app developer, you will have access to all Splunk development resources and receive a 10GB license to build an app that will help solve use cases for customers all over the world. Splunkbase has 1000+ apps from Splunk, our partners and our community. Find an app for most any data source and user need, or simply create your own with help from our developer portal.

Follow Us:
Splunk, Splunk>,Turn Data Into Doing, Data-to-Everything, and D2E are trademarks or registered trademarks of Splunk Inc. in the United States and other countries. All other brand names,product names,or trademarks belong to their respective owners.