Deep Learning Toolkit for Splunk
Overview
Details
Deep Learning Toolkit for Splunk (DLTK)
Prerequisites
- Splunk Machine Learning Toolkit installed
- Docker OR Kubernetes OR OpenShift environment
Quick start guide
- Ensure Splunk Machine Learning Toolkit is installed and configured properly for your Splunk deployment.
- Restart your Splunk instance after installing the Machine Learning Toolkit and the Deep Learning Toolkit App for Splunk.
- You need to have an internet connected Docker environment accessible with permissions to pull the prebuilt MLTK container images from Dockerhub and start containers. If you are running Docker in an air-gapped environment read the description in the app.
- Setup the Deep Learning Toolkit App for Splunk by connecting it to your Docker environment using the setup page in the app.
- Start a development container from the container management dashboard and depending on your selected image run one of the examples to verify that the Deep Learning Toolkit app works:
- Neural Network Classifier Example for Tensorflow
- Logistic Regression Classifier Example for PyTorch
Build your own containers
Extend the app with custom MLTK Containers: if you want to rebuild the existing MLTK Container images or want to build your own custom images navigate to the github repo
Further information, blogs and resources
- Deep Learning Toolkit 3.2 - Graphics, Rapids, Spark and More
- DLTK 3.1 release for Kubernetes and Openshift
- DLTK 3.1 new examples for Prophet, Graphs, DASK and GPU
- .conf19 presentation and recording
- Introduction to DLTK 3.0 release on splunk blogs
- Detailed walkthrough to create custom container images for DLTK by Anthony mirror link
- Detailed walkthrough in Japanese by @maroon
- TensorFlow World Poster Session with PDF download in high resolution
- .conf18 presentation including a CPU/GPU benchmark
FAQ
Q: When I launch a container the first time, I cannot access Jupyter Lab.
A: The selected container image will be downloaded from dockerhub automatically in the background when you launch a container for the first time. Depending on your network this can take a while to download the docker image for the first time as the image sizes range from 2-6 GB. Please allow for some time to get the images initially pulled from Dockerhub. You can check which docker images are available locally by running docker images on your CLI.
Q: The example dashboards show no results or throw errors.
A: First, ensure that the right container image is downloaded and up and running for the specific example (e.g. TensorFlow examples require a TensorFlow container). Secondly, ensure that you have verified the associated notebook code exists in Juypter Lab and you have explicitly saved the notebook again (hit save button). By doing this, a python module is saved automatically generated (located in the /app/model folder in Juypter) which is needed to run the examples and populate the dashboards.
Q: Where are my notebooks stored in the docker environment?
A: By default, there are 2 docker volumes automatically mounted for persistance in your docker environment. Those volumes are named "mltk-container-app" and "mltk-container-notebooks". You can verify by running docker volume ls on your CLI. Important note: from DLTK version 3.1 onwards there is a new default volume called "mltk-container-data" - see migration notes below.
Q: What is the password for Jupyter Lab?
A: Please have a look at the Model Development Guide page in the Deep Learning Toolkit app.
Notebooks Migration note for change to 3.1
Due to the addition of Kubernetes there was a change and addition made to the way how volumes behave. From 3.1 on and with the use of the golden image the container directory /srv is the default and notebooks and app code is in there. In earlier versions there were two docker volumes mounted into /srv/app and /srv/notebooks which will be mapped into a backup folder in Jupyter from version 3.1 on. For migration simply copy your notebooks from the backup folder back into the notebooks and app folder in case those are empty.
Release Notes
Version 3.2.0
Graphics:
- Background graphics in content overview page
- Docker + Kubernetes status green highlight in setup dashboard
- Content UI icons refresh
New docker images for Spark and Rapids:
- Spark Image (Experimental)
- Rapids Image (Experimental)
Content updates:
- Correlation Matrix and seaborn plot embedding
- DGA datashader example
- Spark XGBoost (non distributed, local client only)
- Spark Hello World / Barebone / Pi
- Spark FP Growth
- Spark ALS Recommender System
- Rapids Graph example
- Rapids UMAP example
Other:
- Passwords.conf add for missing kubernetes field
- Search head cluster config replication (Thank you Martin!)
- Return dataframe of arbitrary/changed shapes
Version 3.1.1
- Setup options for Kubernetes and Openshift environment
- Refresh of the container image ("golden image") with added Jupyter Lab Extensions for integrated Tensorboard and DASK management
- New example for forecasting with Prophet
- New example for distributed machine learning with DASK
- New example for graph related algorithms with NetworkX
- New examples for device-agnostic PyTorch CPU/GPU
- New example for Japanese language NLP library Ginza
- Fix of splunk_server=local in | rest calls
- Several UI updates on dashboards
- Bugfix with auth_mode in sync handler
- Several other bug fixes
Version 3.0.0
This version is only compatible with Splunk 8.0+ and MLTK 5.0+
If you run on Splunk 7.x and MLTK 4.x please select, download and install Deep Learning Toolkit version 2.3.0
Version 2.3.0
This version is only compatible with Splunk 7.x and MLTK 4.x only.
If you run on Splunk 8.0+ and MLTK 5.0+ please select, download and install Deep Learning Toolkit version 3.0.0