Cloud Billing v1.2.0
This cloud billing app is designed to give insights into your spending with cloud services. It allows you to see which services are costing money unnecessarily across AWS, Azure and Google Cloud environments.
It is important to note that this app will work for ANY combination of the previously mentioned Cloud Environments set up. ALL of them are optional. If you do not have all of these Cloud environments, only follow the instructions for the specific cloud environment(s) that you have.
1. Install the Splunk AWS TA from Splunkbase ensuring that the following sourcetypes are populated correctly and being sent to an appropriate index:
aws:description aws:cloudwatch aws:billing
Install and configure the following Azure TA's. Do not change the default sourcetypes utilised by these TA's.
Splunk Add-on for Microsoft Cloud Services: https://splunkbase.splunk.com/app/3110/
Microsoft Azure Billing and Usage Add-on for Splunk: https://splunkbase.splunk.com/app/4109/
Microsoft Azure Billing Add-on for Splunk: https://splunkbase.splunk.com/app/4103/
Install and configure the Splunk Add-on for Google Cloud Platform from Splunkbase.
Configuration in GCP is imperative to get the TA working appropriately. Info on configuring a Service Account can be found here.
In order to populate this app fully, all three of the input types will need to be utilized. Billing Data, CPU data and asset inventory data are all required.
First configure your billing data. On your GCP project, create a service account and create a key. Give it access to read buckets. Create a bucket in Storage. Navigate to Billing->Billing Export. Click File Export. Set Bucket name to the one that you just created. Set a report prefix and format type (either will do). Then, in the GCP TA, configure the credentials for the service account that you created with read access to the bucket and set the report prefix. Next configure your cpu data monitoring. Using a service account that has the correct read access, configure the Cloud monitoring input in the Google Cloud TA with the monitoring metric "compute.googleapis.com/instance/cpu/utilization" Finally, we will set up the asset inventory data onboarding. Unfortunately, there is at present no default way to onboard this data, so we have to create a publish/subscription service to send the data to Splunk. Detailed information on how to do this can be found in the README of this app and below.
INFO FOR SETTING UP ASSET INVENTORY ONBOARDING IN GOOGLE CLOUD:
Due to the way that GCP stores and allows the publication of data with service accounts, this Process must be repeated in each Project that you want to be monitored in the Splunk Multicloud Billing App.
Create a service account eg email@example.com
Create a key
Give it access to : Cloud Asset Viewer, Cloud Asset Feeds Create, Cloud Asset Assets Export Resource, Storage Object Creator, Pub/Sub Publisher
Create a storage bucket
Name the bucket (e.g. gs://asset-inventory-bucket)
Set lifecycle on objects in bucket
Set age of object
Set action to delete
Create a Pub/Sub Topic
Topic Name contains the project id and topic id: e.g. project/operating-ally-123456/topics/asset-inventory-topic
Create a Pub/Sub Subscription
This will be the Link to Splunk
Create a streaming Dataflow -> storage to pubsub
Create a job from template
Name the Job
Choose the Dataflow Template - Text Files on Cloud Storage to Cloud Pub/Sub
Input Cloud Storage File(s) - name the storage bucket that you created earlier and specify a pattern for objects created (eg. gs://asset-inventory-bucket/asset-list-.json)
Choose the Ouput Pub/Sub Topic - name the pub sub topic you created earlier (eg. project/operating-ally-123456/topics/asset-inventory-topic)
Create a micro instance (make sure it is free tier to not ensue costs)
Associate Service Account to VM
take service account private key json and store on vm
Activate service account on vm
gcloud auth activate-service-account firstname.lastname@example.org --key-file=key.json
Write a bash script that creates buckets in the storage with asset inventory snapshot
$ cat get-asset-inventory.sh
for project in $(gcloud projects list --format="value(project_id)" --quiet)
gcloud asset export --content-type resource --project $project --output-path "gs://asset-inventory-bucket/asset-list-$project-$datetime.json"
Set up a cron job to run the script
0 * * /directory/to/script/get-asset-inventory.sh
This example returns the asset inventory every hour
(OPTIONAL) Enable Monitoring Asset Changes - This means that any updates to your environment are picked up immediately. All assets are registered at an interval set by you in the crontab above.
gcloud asset feeds create asset-inventory-feed --project=operating-ally-123456 --content-type=resource --asset-types="compute.googleapis.com/Disk","compute.googleapis.com/Instance","compute.googleapis.com/Address",\
Configure Splunk GCP TA inputs to use the Cloud Pub/Sub Service monitoring the correct Subscription
Download the app from Splunkbase.
Either unpack the app in $SPLUNK_HOME/etc/app or install the app from file using the App Manager Page.
First, go to the Inputs of this app, and input the appropriate information to onboard Azure, Google Cloud or AWS prices.
Configure the macro
aws_index to search the index/indexes where your AWS data is stored.
Configure the macro
azure_index to search the index/indexes where your Azure data is stored.
Configure the macro
gcp_index to search the index/indexes where your Google Cloud data is stored.
This can be found in Settings -> Advanced Search -> Macros
For any enquiries or requests for more customised billing, alerting and prediction please contact: email@example.com
Now compatible with Python 3 and Splunk version 8
Allows aggregation of AWS, Azure AND Google Cloud Billing Data.
Now with the inclusion of data from Microsoft Azure Billing
Minor Fixes - savedsearch ownership
Initial release. First version for AWS only, Azure coming soon.
Splunk AppInspect evaluates Splunk apps against a set of Splunk-defined criteria to assess the validity and security of an app package and components.
As a Splunkbase app developer, you will have access to all Splunk development resources and receive a 10GB license to build an app that will help solve use cases for customers all over the world. Splunkbase has 1000+ apps and add-ons from Splunk, our partners and our community. Find an app or add-on for most any data source and user need, or simply create your own with help from our developer portal.