Author | Kyle Smith |
App Version | 1.4.2 |
App Build | 310 |
Vendor Products | G Suite utilizing OAuth2 |
Has index-time operations | true, the included TA add-on must be placed on the indexers |
Creates an index | false |
Implements summarization | Currently, the app does not generate summaries |
About G Suite For Splunk
G Suite For Splunk allows a Splunk Enterprise administrator to interface with G Suite, consuming the usage and administrative logs provided by Google. The limitations on collection times are specified: https://support.google.com/a/answer/7061566 .
This App provides the following scripts:
ga.py | This python file controls the ability to interface with the Google APIs. |
ga_authorize.py | This Python custom endpoint allows the authorization of the App to G Suite For Splunk from the web UI. |
Diag.py | Allows diag-targeted collection of information. |
ModularInput.py | Inheritable Class to create Modular Inputs |
Utilities.py | Allows utility interactions with Splunk Endpoints |
Scripts
Improvement
[GSUITE-25] - Cloud App Vetting
[GSUITE-26] - Fix and Update Proxy settings
Test and QA
Bug
[GSUITE-19] - Interval check doesnt adjust for default.
[GSUITE-20] - BigQuery not caching last row
Improvement
[GSUITE-21] - Fix Proxy Code
[GSUITE-24] - Create New Dashboards
New Feature
[GSUITE-4][EXPERIMENTAL] - GMAIL LOGS and BigQuery
[GSUITE-15] - Splunk 8 Compatibility
[GSUITE-16] - Directory API Ingestion
Improvement
[GSUITE-12] - Auto Discover Available Spreadsheets
[GSUITE-14][Experimental] - Alert Center API
Added chat, gcp, meet, jamboard to allowed Reports input.
Version 1.4.2 of G Suite For Splunk is compatible with:
Splunk Enterprise versions | 8.0 |
Platforms | Splunk Enterprise |
Compatability
Version 1.4.2 of G Suite For Splunk has the following known issues:
According to stackoverflow, there are indications that the Google Apps Admin API has an unspecified delay introduced into the events that are collected. This is most likely due to how Google collects the events and the global nature of the events. To mitigate this issue, the G Suite For Splunk Modular Input has a built-in delay in the consumption of events. If you run the modular input at 30 minutes, there will be a 30 minute delay of events. If you run at 1 hour, there will be a 1 hour delay in events.
References
These are the currently requested scopes:
Access questions and answers specific to G Suite For Splunk at https://answers.splunk.com . Be sure to tag your question with the App.
Support Email: splunkapps@kyleasmith.info
Support Offered: Community Engagement
Support is available via email at splunkapps@kyleasmith.info. You can also find the author on IRC (#splunk on efnet.org) or Slack. Feel free to email or ping, most responses will be within 1-2 business days.
Because this App runs on Splunk Enterprise, all of the Splunk Enterprise system requirements apply.
Download G Suite For Splunk at https://splunkbase.splunk.com.
NOTE: Where referenced, the IA-GSuiteForSplunk and TA-GSuiteForSplunk versions of this App are located on Splunkbase.
1. Deploy as you would any App, and restart Splunk. 1. NOTE: Only the App (or IA for no dashboards) is required. Install only 1 of the G Suite add ons or app. 1. Configure.
1. Have your Splunk Cloud Support handle this installation. Do NOT install the IA on the same system as the App. 1. You may consider using an on-premise Heavy Forwarder to install IA-GSuiteForSplunk, and send the logs to Splunk Cloud.
1. For each Search Head in the environment, deploy a non-configured copy of the App. DO NOT SEND TA or IA to a Search Head Cluster (SHC). 1. For each indexer in the environment, deploy a copy of the TA-GSuiteForSplunk Add-On that is located as mentioned above. 1. For a single Data Collection Node OR Heavy Forwarder (a full instance of Splunk is required), install IA-GSuiteForSplunk and configure through the GUI.
You must have enabled the G Suite APIs at https://console.developers.google.com
You must have configured a credential for use with this App at https://console.developers.google.com.
You must AUTHORIZE this app to make requests into G Suite APIs.
Scopes Defined are here: https://developers.google.com/identity/protocols/googlescopes
Requires: Admin SDK API
, and Google Drive API
Optional: Google Analytics Reporting API
Each API endpoint has individual APIs that need to be enabled within https://console.developers.google.com.
report:[all, gcp, chat, meet, jamboard, access_transparency, groups_enterprise, user_accounts, groups, mobile, admin, calendar, drive, login, token, rules]
These input service names require the Admin SDK API enabled.
Additionally, the drive report requires the Google Drive API enabled.
These inputs generally do not require Extra Configuration options in the Modular Input. An empty {} is still needed where advanced features are not.
These inputs should be adjusted per Google guidelines for the different activities.
By default, the Modular Input will only pull the previous 24 hours of data to prevent memory overflows.
analytics:[metadata, report]
These input service names require the Analytics Reporting API v4 and Analytics API APIs enabled.
These inputs do require Extra Configuration. These inputs should not be enabled lightly, and require a little bit of prior research and planning.
IF YOU DONT KNOW WHAT THIS IS, DO NOT ENABLE IT
THIS IS A DARK FEATURE.
usage:[customer, user, chrome]
These input service names require the same as the report services.
These inputs can have extra configuration, namely historical_days to do the initial data ingestion.
IMPORTANT: BE CAREFUL WITH USER REPORTING. If you ingest 365 days of data (back fill the information), you will end up with 365 * # of users events to pull and could cause a Splunk/System failure.
If you see a 404 Error in the logs relating to the usage reports, THESE ARE NORMAL.
The Customer Usage should include classrooms usage by default.
Requires: Google Sheets API
Index
Takes the information from the sheet and indexes it to the specified index. This is useful to get lookups from a Heavy Forwarder to a Search head.
Use the provided Dashboard to re-assemble via saved scheduled search.
KVStore
Takes the information from the sheet and places it into a KVStore collection.
It will create the needed collections and transforms if needed.
Order of the COLUMNS is NOT
kept, and the KVStore will be sorted via ASCII sort based on the column name.
Ordered KVStore
Takes the information from the sheet and places it into a KVStore collection.
It will create the needed collections and transforms if needed.
Order of the COLUMNS IS
kept, the column names are stored in ROW 0
CSV Lookup
Takes the information from the sheet and places it into a CSV based lookup.
It will create the needed transforms if needed.
Order of the COLUMNS is NOT
kept, and the CSV lookup will be sorted via ASCII sort based on the column name.
Ordered CSV Lookup
Takes the information from the sheet and places it into a CSV based lookup.
It will create the needed transforms if needed.
Order of the COLUMNS IS
kept, the column names are stored in ROW 0
.
Requires: `BigQuery API
NOTE: This is EXPERIMENTAL. Enjoy breaking the input. This section to be updated when working correctly. NOTE: DOES NOT CURRENTLY WORK WITH PROXIES NOTE: To consume all tables in a dataset, use the table name all
Service Account JSON File from GCP. (https://console.developers.google.com/iam-admin/serviceaccounts)
Create a new Splunk credential with Realm: gsuite_bigquery and username is \<your_domain> (your domain as configured in the input)
The password for that credential is the ENTIRE ON ONE LINE JSON file from GCP for the service account.
Requires: G Suite Alert Center API
Note: EXPERIMENTAL
(scope not valid) #. alerts:[all, takeout, gmail, identity, operations, state, mobile]
1. These inputs generally do not require Extra Configuration options in the Modular Input. An empty {} is still needed where advanced features are not. 2. By default, the Modular Input will only pull the previous 24 hours of data to prevent memory overflows. 3. Uses the https://www.googleapis.com/auth/apps.alerts
scope. 4. View more information at https://developers.google.com/admin-sdk/alertcenter/reference/alert-types .
IMPORTANT: You must Authorize the APIS with the SAME USER that allowed access to the APIs in the developer console (for GSuite customers - GCP see below).
Overview of authorization procedures are found here: https://developers.google.com/identity/protocols/OAuth2ServiceAccount#overview.
GCP Users: It has been tested to use an Credential generated in the GCP console (same credential type as outlined on the OAuth App Config page). You can use an authorized admin to Approve the OAuth Scopes. It is unknown what happens when the approving Admin user account is disabled.
NOTE: You will need to configure a new modular input for each domain
NOTE: After testing in a much bigger environment, weve been able to set these recommendations for intervals. You will need 4 modular input definitions.
calendar, token, mobile, groups, login, saml, Chrome OS Devices #. These are done at an cron interval of 15 */4 * * *
drive #. Drive is done at a seconds interval of 600 - 1200 depending on organization size, and traffic flow of drive operations.
Usage - User, Customer #. These are done at a seconds interval of 86400
admin, rules, chat, gplus #. These are done at a seconds interval of 600
By default all events will be written to the main index. You should change the index in the configuration files to match your specific index.
This App Supports proxy configuration. Configure the proxy first in the Application Configuration dashboard, and then choose it during the modular input configuration. The proxy name MUST BE gapps_proxy for the authorization to work correctly.
Check the Monitoring Console (>=v6.5) for errors
Visit the Application Health dashboard
Search for eventtype=googleapps_error
As of v1.4.0 of this app, we should support version 4.15 of the CIM.
There are portions of this app that are experimental, or you might see odd code. This is for some up coming features, might work, might not.
G Suite For Splunk contains the following lookup files:
gsuite_labels.csv
: This allows pretty labels on select dashboards.G Suite For Splunk does not make use of an event generator. This allows the product to display data, when there are no inputs configured.
Summary Indexing: No
Data Model Acceleration: No
Report Acceleration: No
bin/google/protobuf/internal/_api_implementation.so
is apparently a binary file. Required for Google Things.
bin/google/protobuf/internal/_message.so
is apparently a binary file. Required for Google Things.
For these two, please see https://github.com/protocolbuffers/protobuf/tree/3.6.x/python/google/protobuf/internal for source and attribution.
Version 1.4.2 of G Suite For Splunk incorporates the following Third-party software or third-party services.
Please visit https://developers.google.com/google-apps/ for full terms and conditions.
Test and QA
- [GSUITE-23] - App inspect Failures
Bug
- [GSUITE-19] - Interval check doesnt adjust for default.
- [GSUITE-20] - BigQuery not caching last row
Improvement
- [GSUITE-21] - Fix Proxy Code
- [GSUITE-24] - Create New Dashboards
As a Splunkbase app developer, you will have access to all Splunk development resources and receive a 10GB license to build an app that will help solve use cases for customers all over the world. Splunkbase has 1000+ apps from Splunk, our partners and our community. Find an app for most any data source and user need, or simply create your own with help from our developer portal.