The goal of this application is to provide full Python functionality to Splunk. Currently, the Python distribution built in to Splunk runs on a version 2.7.x depending on the Splunk version. Additionally, there are significant core modules that are excluded in this distribution. This suite of apps will allow developers to create Python virtual environments and pick the Python version and modules installed to the environment. This includes core distributions of the interpreter in multiple 2.7.x and 3.5+ versions, as well as the use of pip for the installation of additional modules within a virtual environment.
The PyDen app is based on the premise that the Splunk built-in CPython distribution is insufficient for advanced development within the Splunk platform. The key example of this is Splunk's own Machine Learning toolkit (MLTK). The Splunk MLTK requires the installation of an add-on which provides access to the Anaconda Python interpreter and several Python libraries that are common in the community for Machine Learning applications, such as
While this is an excellent solution for providing the needed functionality for the MLTK, it's insufficient for generalized use for three reasons. These are the three primary concerns that PyDen attempts to address: version flexibility, proper isolation, and access to PyPI packages.
Splunk's built-in Python utilizes version 2.7 of CPython. While there are a decent number of common libraries that are continuing support of 2.7, the number is dwindling and newer projects do not support it at all. The version will also reach end of life in 2020, causing significant supportability and security issues. The PyDen app allows a developer to build CPython from source in a variety of versions including 3.5 and higher through the use of the
Modules added to a Splunk app are available to the entire installation, which is in contrast to Python development best practices which calls for the isolation of package requirements through the use of virtual environments. This isolation of packages resolves issues around dependency conflicts and keeps the scope of libraries to the context of the application. This is done through making Python virtual environments available via PyDen's
Splunk does not allow a developer to add additional packages not included with the built-in distribution except through inclusion within the app directly. Ease of inclusion of third-party libraries is a key benefit of Python. However, including non-native libraries within a Splunk application poses two key challenges: dependency chasing and library conflicts (which we discuss in the next section). In order to include a non-native library within a Splunk app, a developer must include it directly within the app's
bin directory. If the package contains any dependencies, those must be added as well. If any of the dependencies have their own dependencies, they must be included as well. And so on, and so on. By creating access to the
pip command within Splunk, PyDen does not suffer this problem.
PyDen is broken into two separate Splunk apps called: PyDen and PyDen Manager. The PyDen Manager app contains all the functionality needed for a user to download, compile, and build Python distributions and virtual environments, as well as the ability to use pip to install packages to those environments. The actual CPython builds and environments are placed in the PyDen app. The location of the PyDen app can vary depending on the Splunk architecture.
For a single search head environment the PyDen app can be placed along side the PyDen Manager in the Splunk apps directory.
In a deployment with multiple search heads, PyDen can be placed onto the deployment server repository location.
In a deployment utilizing search head clustering, it can be placed into the deployer's shcluster/apps directory.
In all of these scenarios, PyDen Mnagaer must be installed to the same host as PyDen. The
pyden.conf configuration file inside of the PyDen Manager app contains a stanza called
app which has a
location attribute which should specific the absolute path of the PyDen app.
The PyDen suite of apps has been tested for compatibility with Splunk 7.2. Use with other version of Splunk are considered experimental and should be tested thoroughly.
The app builds CPython from source and therefore the success of the build is highly dependent on the operating system libraries available. The host OS must have a compiler and the packages needed in order to build Python from source. Please refer to the documentation for your OS for needed packages. Two options are included in the ./configure command during the Python build process:
--enable-optimizations (if configured) and
Installation of the PyDen Manager follows the same process as any other Splunk app. Please refer to Splunk's documentation on the subject. The PyDen app installation process will vary depending on the Splunk deployment architecture. Please see the Architecture section for details on where to install PyDen. If PyDen is installed to any location other than the Splunk apps directory, the installation process will consist of extracting the app from the gzipped tarball and placing it in the specified directory on the host.
Both PyDen and PyDen Manager need to be installed on the same host. After both apps are installed, the PyDen Manager needs to be configured through the
pyden.conf file. This file contains two stanzas:
download stanza has a single attribute called
url which is used to specify the location from where Python is downloaded. It is not recommended to change this location except in the need to app proxy information. If a location other than www.python.org there will be significant gaps in the functionality of the dashboards. The
app stanza contains two attributes
location attribute is the absolute path of the location of the PyDen app and the
optimize attribute is a boolean which indicates whether or not Python will be built using the
--enable-optimizations parameter. Enabling optimizations will provide significant speed improvements but takes significantly longer to build.
There are three primary custom commands that make up the core functionality of PyDen:
pip. There are additional suplemental commands that primarily aid in working with the dashboards.
createdist command is how PyDen downloads and builds a CPython distribution. This command contains two keyword arguments
version argument is simply the version number of the Python installation to be built. The
download argument is optional and only used if not doing an automatic download from www.python.org. Instead of being downloaded, the Python source package can be placed inside the PyDen Manager's bin/build directory. When done this way, the
download argument should provide the name of the package to be used. Package must be a file of
Note: the first distribution created by this command will be set as the default version used by other commands like
Download and install version 3.7.2 from www.python.org
| createdist version=3.7.2
Install 3.7.2 version of Python from source package called mypython.tgz
| createdist version=3.7.2 download=mypython.tgz
createvenv command creates a Python virtual environment with a specified version and name. The command has two required keyword arguments: version and name. The version is a version number that references an installed Python distribution from the
createdist command. The name argument is a name to be associated with the environment for future reference.
Note: the first environment created by this command will be set as the default environment used by other commands like
Create a virtual environment with Python version 3.7.2 and named mypy
| createvenv version=3.7.2 name=mypy
pip command installs Python packages available from the Python Package Index. This command works identically to the command-line tool of the same name with one exception. The command takes a single keyword argument called environment which specifies which virtual environment the command applies to. If used, this must be the first argument listed. If not used, the command will apply to the default virtual environment.
Install the package
requests to the default environment
| pip install requests
Upgrade the pip version of the environment
| pip environment=myvenv install --upgrade pip
The following commands are included in the PyDen Manager app but are of limited value. They're typically used to provide some needed functionality for a dashboard.
This command creates a list of events with a field called
version whose values are PyDen compatible versions of Python available from www.python.org.
This command deletes distributions and virtual environments created through the
createvenv commands. The command takes a single positional argument of the name or version number of the environment or distribution to be deleted.
| pydelete 3.7.2
Delete the virtual environment named
| pydelete mypy
This command creates a set of events with a single field called
environment whose values are the names of virtual environments created by the
getpackages command is used to get package information from the Python Package Index. There are two modes to this command. The command itself takes a single positional argument. If
pypi_simple_index is provided as the argument then the command will create a set of events with a single field called
package whose value is the name of a package from PyPI's simple index.
If any other argument is provided, the command will use its other mode which looks up the json data for a PyPI package found at
https://pypi.python.org/pypi/package_name/json and returns an event with a single field called
description whose value is the PyPI description of the package provided.
Get all PyPI packages from the PyPI simple index
| getpackages pypi_simple_index
Get the description for the requests package
| getpackages requests
Leveraging the environments in the PyDen app is simply a matter of importing the activation modules provided with PyDen.
PyDen comes with two scripts in its bin directory:
activate.py script contains a function called
activate_venv_or_die. In order to run a script with a PyDen virtual environment, the script must include the following code at the top, substituting your virtual environment name for
from activate import activate_venv_or_die activate_venv_or_die("environment_name")
This will utilize Python's
os.execve function to restart the script with the provided virtual environment.
While this will work with any valid virtual environment provided, you may not wish to muddy up your import statements with functions. In order to avoid this problem the
activate_default.py script is provided. This script will activate the default virtual environment defined in the PyDen app's
pyden.conf configuration file when imported and without the function call. Instead of the above code, simply add the following code to the top of your script:
It is important to note that these two scripts are included in the PyDen app and in order to import them as Python modules, you will need to do one of the following:
- Place the script you are writing in the PyDen bin directory (this is highly discouraged as scripts can be overwritten during an upgrade of the app)
- Modify the
sys.path to include the PyDen bin directory before importing the scripts (this may have unintended side effects)
activate_default.py files into the app which contains your script (preferred method)
This release primarily provides additional features for more mature deployments as well as a couple small conveniences.
This version officially supports distributed environments. If PyDen is installed to the indexers, they will be able to execute any Python script that leverages the PyDen activation from search bundles. This version also support modular alert actions.
The default version to be used for new virtual environments, specified in the
pyden.conf file, can be set via the UI of the
Python Versions dashboard.
The version of a virtual environment is now shown in the table of the
Virtual Environments dashboard.
The activation script will now use information from
sys.executable instead of adding an argument to
* The activation script will now read the
pyden.conf file from a
btool subprocess instead of attempting to open the file using relative path.
This release primarily provides three new dashboards which help the user manage the use of the custom commands from the initial release. There have also been some significant error handling and bug fixes in this release. This release also contains some much improved and much needed documentation.
initial Release. Please refer to README for comprehensive documentation.
As a Splunkbase app developer, you will have access to all Splunk development resources and receive a 10GB license to build an app that will help solve use cases for customers all over the world. Splunkbase has 1000+ apps from Splunk, our partners and our community. Find an app for most any data source and user need, or simply create your own with help from our developer portal.