Welcome to G Suite for Splunk Apps documentation!

Overview

About G Suite For Splunk

About G Suite For Splunk

G Suite For Splunk allows a Splunk Enterprise administrator to interface with G Suite, consuming the usage and administrative logs provided by Google. The limitations on collection times are specified: https://support.google.com/a/answer/7061566 .

Scripts and binaries

This App provides the following scripts:

Scripts

Release notes

Version 1.3.1

• Bug

  • [GSUITE-13] - Compiled Library caused AppInspect Exception

Version 1.3.0

• Bug

  • [GSUITE-8] - IA Builder Multi MI
  • [GSUITE-10] - Remove Sheets after Successful Creation of Input
  • [GSUITE-11] - Error Messages on Data Not Found - Usage reports

• New Feature

  • [GSUITE-6] - Google Drive Import Export for Sheets and Lookups
  • [EXPERIMENTAL] - BigQuery Integration for GMAIL Logs

• Improvement

  • [GSUITE-5] - Update Auth to be Front End Only
  • [GSUITE-7] - Refactor GA for Scale
  • [GSUITE-9] - Waiting Indicator for Sheet Names

Version 1.2.3

• Improvement

  • Remove a logging line.

Version 1.2.2

• Bug

  • [ASA-245] - Chrome OS API doesnt work
  • [ASA-248] - Timezone Python Error

• New Feature

  • [ASA-233] - Enrich Google Drive reporting with Drive information

• Improvement

  • [ASA-164] - Create CIM Fields for Google Apps for Splunk
  • [ASA-228] - G Suite - new dashboard - Token Usage
  • [ASA-231] - Fix Security Concern (file handlers)

Version 1.2.1

NOTE: THIS IS A BREAKING VERSION. YOU MUST REAUTHORIZE THE CLIENT BEFORE THE APP WILL POPULATE DATA.

• Bug

  • [ASA-218] - Fix the Extra Config not validating on empty object

• New Feature

  • [ASA-216] - Update GSuite to store json credentials in encrypted store.

• Improvement

  • [ASA-211] - Fix Timezone Translation
  • [ASA-217] - Add additional data feed

Version 1.2.0

• Adds the analytics:report, and analytics:metadata reports (advanced)

• Adds the report:mobile and report:groups reports (included in report:all)

• Adds the usage:customer, usage:user, and usage:chrome reports.

• Support

  • [ASA-11] - Setup page malfunction
  • [ASA-57] - Proxy Support for GApps

• Bug

  • [ASA-2] - Docs/Drive/Calendar events not timestamping correctly
  • [ASA-3] - Missed Audit Events
  • [ASA-4] - Failure for Usage Events (customer/user)
  • [ASA-5] - Historical Days Configuration Option
  • [ASA-6] - Transforms are old and deprecated
  • [ASA-12] - Setup.xml
  • [ASA-13] - Authorize_Splunk dashboard reset
  • [ASA-20] - Failure to load checkpoint
  • [ASA-41] - Authorization Failure on HF
  • [ASA-120] - Header breaks with 6.5 compat
  • [ASA-198] - Controller Endpoints not supported.

• New Feature

  • [ASA-19] - Error Dashboard
  • [ASA-103] - Add Access to Google Analytics API
  • [ASA-126] - MC Checklist and Health Path
  • [ASA-206] - Chrome OS Devices

• Improvement

  • [ASA-9] - Google API - 1 hour buffer.
  • [ASA-10] - debug/refresh endpoint causes modular input to fire
  • [ASA-37] - Support Proxy for calls
  • [ASA-115] - Update MI for backfill

About this release

Version 1.3.1 of G Suite For Splunk is compatible with:

Compatability

Known Issues

Version 1.3.1 of G Suite For Splunk has the following known issues:

• According to stackoverflow, there are indications that the Google Apps Admin API has an unspecified delay introduced into the events that are collected. This is most likely due to how Google collects the events and the global nature of the events. To mitigate this issue, the G Suite For Splunk Modular Input has a built-in delay in the consumption of events. If you run the modular input at 30 minutes, there will be a 30 minute delay of events. If you run at 1 hour, there will be a 1 hour delay in events.

• References

  • https://support.google.com/a/answer/7061566
  • http://stackoverflow.com/questions/27389354/minimal-delay-when-listing-activities-using-the-reports-api
  • http://stackoverflow.com/questions/30850838/what-is-the-delay-between-a-event-happens-and-it-is-reflected-in-admin-reports-a

• These are the currently requested scopes:

  • https://www.googleapis.com/auth/admin.reports.audit.readonly
  • https://www.googleapis.com/auth/admin.reports.usage.readonly
  • https://www.googleapis.com/auth/analytics.readonly
  • https://www.googleapis.com/auth/admin.directory.user.readonly
  • https://www.googleapis.com/auth/admin.directory.device.chromeos.readonly
  • https://www.googleapis.com/auth/drive.metadata.readonly

Support and resources

Questions and answers

Access questions and answers specific to G Suite For Splunk at https://answers.splunk.com . Be sure to tag your question with the App.

Support

• Support Email: splunkapps@kyleasmith.info
• Support Offered: Community Engagement

Support is available via email at splunkapps@kyleasmith.info. You can also find the author on IRC (#splunk on efnet.org) or Slack. Feel free to email or ping, most responses will be within 1-2 business days.

Installation and Configuration

Software requirements

Splunk Enterprise system requirements

Because this App runs on Splunk Enterprise, all of the Splunk Enterprise system requirements apply.

Download

Download G Suite For Splunk at https://splunkbase.splunk.com.

Installation steps

NOTE: Where referenced, the IA-GSuiteForSplunk and TA-GSuiteForSplunk versions of this App are located on Splunkbase.

Deploy to single server instance

Follow these steps to install the app in a single server instance of Splunk Enterprise:

1. Deploy as you would any App, and restart Splunk.
2. Install IA-GSuiteForSplunk.
3. Configure.

Deploy to Splunk Cloud

1. Have your Splunk Cloud Support handle this installation. Do NOT install the IA on the same system as the App. 1. You may consider using an on-premise Heavy Forwarder to install IA-GSuiteForSplunk, and send the logs to Splunk Cloud.

Deploy to a Distributed Environment

1. For each Search Head in the environment, deploy a non-configured copy of the App. DO NOT SEND TA or IA to a Search Head Cluster (SHC). 1. For each indexer in the environment, deploy a copy of the TA-GSuiteForSplunk Add-On that is located as mentioned above. 1. For a single Data Collection Node OR Heavy Forwarder (a full instance of Splunk is required), install IA-GSuiteForSplunk and configure through the GUI.

User Guide

Key concepts for G Suite For Splunk

• You must have enabled the G Suite APIs at https://console.developers.google.com
• You must have configured a credential for use with this App at https://console.developers.google.com.
• You must AUTHORIZE this app to make requests into G Suite APIs.

Configure G Suite For Splunk for use with G Suite Admin Reporting.

Requires: Admin SDK API, and Google Drive API Optional: Google Analytics Reporting API Each API endpoint has individual APIs that need to be enabled within https://console.developers.google.com.

1. • report:[all, groups, mobile, admin, calendar, drive, login, token, rules, saml, gplus, chat]

     1. These input service names require the Admin SDK API enabled.
     2. Additionally, the drive report requires the Google Drive API enabled.
     3. These inputs generally do not require Extra Configuration options in the Modular Input. An empty {} is still needed where advanced features are not.
     4. These inputs should be adjusted per Google guidelines for the different activities.
     5. By default, the Modular Input will only pull the previous 24 hours of data to prevent memory overflows.

2. • analytics:[metadata, report]

     1. These input service names require the Analytics Reporting API v4 and Analytics API APIs enabled.
     2. These inputs do require Extra Configuration. These inputs should not be enabled lightly, and require a little bit of prior research and planning.

3. • usage:[customer, user, chrome]

     1. These input service names require the same as the report services.

     2. • These inputs can have extra configuration, namely historical_days to do the initial data ingestion.

          1. When configuring the modular input, use the Extra Configuration option of {historical_days: 180}

     3. IMPORTANT: BE CAREFUL WITH USER REPORTING. If you ingest 365 days of data (back fill the information), you will end up with 365 * # of users events to pull and could cause a Splunk/System failure.

     4. If you see a 404 Error in the logs relating to the usage reports, THESE ARE NORMAL.

Configure G Suite For Splunk for use with Google Spreadsheets

Requires: Google Sheets API

1. When setting up the modular input, make sure you grab the Spreadsheet ID from the URL of the spreadsheet you need. Auto-discovery of available spreadsheets is not available (but an ER is in for it).

Spreadsheet Destinations

1. • Index

     1. Takes the information from the sheet and indexes it to the specified index. This is useful to get lookups from a Heavy Forwarder to a Search head.
     2. Use the provided Dashboard to re-assemble via saved scheduled search.

2. • KVStore

     1. Takes the information from the sheet and places it into a KVStore collection.
     2. It will create the needed collections and transforms if needed.
     3. Order of the COLUMNS is NOT kept, and the KVStore will be sorted via ASCII sort based on the column name.

3. • Ordered KVStore

     1. Takes the information from the sheet and places it into a KVStore collection.
     2. It will create the needed collections and transforms if needed.
     3. Order of the COLUMNS IS kept, the column names are stored in ROW 0

4. • CSV Lookup

     1. Takes the information from the sheet and places it into a CSV based lookup.
     2. It will create the needed transforms if needed.
     3. Order of the COLUMNS is NOT kept, and the CSV lookup will be sorted via ASCII sort based on the column name.

5. • Ordered CSV Lookup

     1. Takes the information from the sheet and places it into a CSV based lookup.
     2. It will create the needed transforms if needed.
     3. Order of the COLUMNS IS kept, the column names are stored in ROW 0.

Configure G Suite For Splunk for use with Google BigQuery

Requires: `BigQuery API

NOTE: This is EXPERIMENTAL. Enjoy breaking the input. This section to be updated when working correctly.

Notes

IMPORTANT: You must Authorize the APIS with the SAME USER that allowed access to the APIs in the developer console (for GSuite customers - GCP see below).

Overview of authorization procedures are found here: https://developers.google.com/identity/protocols/OAuth2ServiceAccount#overview.

GCP Users: It has been tested to use an Credential generated in the GCP console (same credential type as outlined on the OAuth App Config page). You can use an authorized admin to Approve the OAuth Scopes. It is unknown what happens when the approving Admin user account is disabled.

Modular Input

NOTE: You will need to configure a new modular input for each domain

1. Follow the steps on the Application Configuration dashboard to configure the modular input.

NOTE: After testing in a much bigger environment, weve been able to set these recommendations for intervals. You will need 4 modular input definitions.

1. calendar, token, mobile, groups, login, saml, Chrome OS Devices #. These are done at an cron interval of 15 */4 * * *
2. drive #. Drive is done at a seconds interval of 600 - 1200 depending on organization size, and traffic flow of drive operations.
3. Usage - User, Customer #. These are done at a seconds interval of 86400
4. admin, rules, chat, gplus #. These are done at a seconds interval of 600

Indexes

By default all events will be written to the main index. You should change the index in the configuration files to match your specific index.

Configure Proxy Support

This App Supports proxy configuration. Configure the proxy first in the Application Configuration dashboard, and then choose it during the modular input configuration. The proxy name MUST BE gapps_proxy for the authorization to work correctly.

Troubleshoot G Suite For Splunk

1. Check the Monitoring Console (>=v6.5) for errors
2. Visit the Application Health dashboard
3. Search for eventtype=googleapps_error

CIM

As of v1.2.2 of this app, we should support version 4.10 of the CIM.

EXPERIMENTAL

There are portions of this app that are experimental, or you might see odd code. This is for some up coming features, might work, might not.

Lookups

G Suite For Splunk contains no lookup files.

Event Generator

G Suite For Splunk does not make use of an event generator. This allows the product to display data, when there are no inputs configured.

Acceleration

1. Summary Indexing: No
2. Data Model Acceleration: No
3. Report Acceleration: No

Binary File Declaration

1. bin/google/protobuf/internal/_api_implementation.so is apparently a binary file. Required for Google Things.
2. bin/google/protobuf/internal/_message.so is apparently a binary file. Required for Google Things.

Third Party Notices

Version 1.3.1 of G Suite For Splunk incorporates the following Third-party software or third-party services.

Google Apps APIs

Please visit https://developers.google.com/google-apps/ for full terms and conditions.

2019, alacercogitatus. | Powered by Sphinx 1.6.4 & Alabaster 0.7.10