Monitoring of Java Virtual Machines with JMX
Overview
Details
The Python code in this App is dual 2.7/3 compatible.
This version of the App enforces Python 3 for execution of the modular input script when running on Splunk 8+ in order to satisfy Splunkbase AppInspect requirements.
If running this App on Splunk versions prior to 8 , then Python 2.7 will get executed.
Release Notes
Version 3.1
added a setup page to encrypt any credentials you require in your configuration
Version 3.0
enforced python3 for execution of the modular input script.If you require Python2.7 , then download a prior version (such as 2.8).
updated all of the dashboards to be compatible with Splunk 8
removed the default 'jmx' index , 'main' is the new default index
Version 2.8
Dual Python 2.7 and 3+ compatibility.
App will run on :
Splunk Enterprise versions back to Splunk 5 where there is only a Python 2.7 runtime shipped
Splunk Enterprise version 8 where there is both a Python 2.7 and Python 3+ runtime shipped
Future versions of Splunk Enterprise where there is only a Python 3+ runtime
Version 2.7
added JAXB dependencies for JRE 9+
fixed Splunk 8 compatibility for manager.xml file
Version 2.6.4
correcting some minor docs errors
Version 2.6.3
cosmetic fixes
Version 2.6.2
updated docs
Version 2.6.1
added trial key functionality
Version 2.6
docs updated
Version 2.5
Added an activation key requirement , visit http://www.baboonbones.com/#activation to obtain a free,non-expiring key
Docs updated
Splunk 7.1 compatible
Version 2.4
Minor HEC data handling tweaks
Version 2.3
Added support to optional output to Splunk via a HEC (HTTP Event Collector) endpoint
Version 2.2.2
Tidied up a dead link in the navigation menu
Version 2.2.1
Fixed some minor typos
Version 2.2
Enabled TLS1.2 support by default.
Made the core Modular Input Framework compatible with latest Splunk Java SDK
Please use a Java Runtime version 7+
If you need to use SSLv3 , you can turn this on in bin/jmx.py
SECURE_TRANSPORT = "tls"
#SECURE_TRANSPORT = "ssl"
Version 2.1
Config file is now dynamically reloaded if it changes
PID File contents are read in on each poller execution
PID Command is executed on each poller execution
PID Command can also return JVM Descriptions in format : 1234,somedescription
Version 2.0.4
Minor change so that when using "dumpAllAttributes" , only READABLE attributes will be polled.
Version 2.0.3
Added a property to optionally specify an alternate config file directory location , relative to SPLUNK_HOME
Version 2.0.2
Put app icon images in the correct directory(this has changed for Splunk 6)
Fixed a classloading clash for direct process attachment jars for win32 vs linux deployments
Version 2.0
NOTE : As per Splunk legal naming standard requirements , this app has now been renamed from "Splunk for JMX" to "Monitoring of Java Virtual Machines with JMX". But it is still functionally the same with everything you know and love , plus more !
Release Features:
Refactored the core data collection to be a Modular Input vs the prior versions' scripted inputs.This is much better for performance as now you can run multiple polling stanzas in 1 single JVM, versus having to fire up a JVM for every time you execute a configuration file
As the Modular Input is a Python wrapper to a JVM instance , all Splunk platforms are now supported
The app can either be run in app mode(with dashboards) or in add-on mode(just the modular input.)
Removed all Advanced XML and replaced with Simple XML
Added support for listening for MBean Notifications
All logging now goes to splunkd.log , searchable via "index=_internal ExecProcessor jmx.py"
All documentation brought up to date.
Version 1.7
Updated to be compatible with Splunk 6
Version 1.6
Tweaked the GC views , Added support to roll out Composite Type arrays
Version 1.5.7
When using direct process attachment and specifying a PID Command script, previously Splunk for JMX would only support a single PID in the command output , now it supports multiple PIDs (delimited by newlines)
Version 1.5.6
Minor cosmetic update , removed paypal links
Version 1.5.5
Added heap/non-heap memory committed charts to the jvm_memory dashboard
Version 1.5.4
Added a setting to SearchSelectLister modules so that their populating searches inherit the parent TimeRangePicker selection
Version 1.5.3
++added a configuration parameter(stripPattern) for Formatter declarations in config.xml that allows you to specify a regex pattern, or list of patterns, that if matched will strip the matched text from the raw MBean attribute/operations values++
Version 1.5.2
++just a tweak to the build, some files were erroneously getting CHMOD'd executable in the tarball++
Version 1.5.1
++Converted all the demo views to Advanced XML and added in dynamic time,host,jvm selectors++The new Advanced XML views use output generated by the TokenizedMBeanNameQuotesStrippedFormatter, this is the formatter now declared in the example config xml files, the DefaultFormatter is still available if you were using and still want to use that, but won't work with the new Advanced XML demo views++
Version 1.5
++Refactored core engine so you can plugin both custom Formatters and custom Transports++Bundled STDout(default), Splunk Rest, TCP,Syslog,File Transport implementations++added ability to add parameters to Formatter and Transport declarations++3 bundled Formatters can be customised with various different parameters in the configuration xml file, such as the ability to prepend a date and specify the date format pattern++updated docs
Version 1.4
Added a setup screen ++ Added a MAX_TIMESTAMP_LOOKAHEAD setting in props.conf to prevent erroneous epoch time rsolutions from non time fields ++ Added links in navigation bar to pdf/html docs
Version 1.3.4
++minor adjustment to logging, no more daily rolling files , now only keeping 1MB with 1 backup as the JMX logs get indexed by Splunk anyway ++ documentation housekeeping
Version 1.3.3
+ More robust handling of primitive type arrays + All Formatter implementations now strip out all newlines and tabs from attribute values and replace with single space characters, thus flattening out all possible attribute values into a single line
Version 1.3.2
+++Added several new demo views, tables, charts...notably, CPU usage per JVM(based on the same algorithm that JConsole uses to derive the JVMs cpu usage), garbage collection, memory pools/memory managers , more thorough OS and Runtime metrics, enhanced connectivity errors page.+++no changes to core engine or config schema, so full backward compatibility maintained
Version 1.3.1
Tweaked the demo forms and navigation bar >>Added a data input to monitor the splunk4jmx error logs and also added view on the navigation menu that reports on jmx connection errors.
Version 1.3
Docs updated
Version 1.2.9
Fixed minor bug in poll_mx4j_jmx.sh script
Version 1.2.8
Added an optional "Formatter" implementation that rather than outputting the mbean name as a single canonical string, breaks up the mbean name into its constituent parts and outputs as individual fields.In your config XML file you can use this formatter by declaring.....<formatter className="com.dtdsoftware.splunk.formatter.TokenizedMBeanNameFormatter" />
Version 1.2.7
Added support for discovering and extracting ALL attributes of an MBean.Provides an alternative to having to explicitly declare each attribute you want to extract.
...a few examples....
dump the attributes from every MBean from every domain :
<mbean domain="*" properties="*" dumpAllAttributes="true" />
....
dump the attributes from all MBeans in the java.lang domain :
<mbean domain="java.lang" properties="*" dumpAllAttributes="true"/>
...
dump the attributes from all MBeans in all of the cassandra domains :
<mbean domain="org.apache.cassandra.*" properties="*" dumpAllAttributes="true" />
Version 1.2.6
MX4J support is now implemented, in addition to the standard JSR160 rmi and iiop connectors , there are now MX4J http/https connectors (hessian, burlap and soap). If your target JVM is using MX4J 3.0.2+ as its JMX implementation, then you can use these additional connectors.
In my testing, these http connectors all work well for simple attributes.
Hessian/Burlap work well with CompositeData attributes, but not so with TabularData attributes
Soap seems to have issues with CompositeData and TabularData attributes.
I would recommend the hessian connector if using an http based connector.
I have also updated the lib layout and scripts to allow easier addition of custom jar files, 3rd party jar files etc...
All documentation updated.
Version 1.2.5
>>Tested against IBM J9 JVMs >>Fixed bug with multiple cluster elements in config xml file >> Tidied up processing of TabularData
Version 1.2.4
>>>Added more thorough Java Collection type support for Attribute value and Operation return types. >>>Can now deal with Arrays, Lists, Maps, Sets and recursively deeply inspect Collections of Collections.>>>Docs updated.
Version 1.2.3
Added support for JMX operations , updated documentation.
Version 1.2.2
Created a PDF User Guide
Version 1.2.1
Added the notion of a "cluster" of JMX servers to the configuration schema.
This will be beneficial when you have many JVMs with the same MBeans ie: JEE App server clusters, large scale deployments of Hadoop/Cassandra nodes on commodity hardware, JVM's across dev/test/qual/prod environments etc..
So the "cluster" element allows you to declare the MBean attributes to extract just once, followed by a list of the JMX servers.Avoids unnecessary repetition and potential bloating of the config file.
For direct attachment to a local JVM using the Process ID , you can now also specify a script to execute to obtain the PID.
This is specified in the "pidCommand" attribute of the jmxserver.
ie: on *Nix a command such as
ps -eafH | grep "com.foo.MyMainClass" | grep -v "grep" | awk '{print $2}'
could be put in a script and used to output the Process ID of the JVM with main class "com.foo.MyMainClass" to STDOUT
Added some more documentation
javadoc API for creating a custom Formatter implementation
schema documentation for the configuration XML
Version 1.2
Added XSD and XSD validation for the config XML
Tidied up bin/config and bin directorys
Changed scripts to take the name of the config file as an argument(set from SPLUNK Manager) , rather than as a variable in the script itself.
Adding logging for error level events
Added Java API to allow users to create a custom "Formatter" for SYSOUT lines that are fed into SPLUNK indexing
Logic to write out host name when connecting to localhost or 127.0.0.1
Version 1.1.2
Tweaked windows script
Version 1.1.1
Tweaked packaging