Monitoring of Java Virtual Machines with JMX
Overview
Details
Monitoring of Java Virtual Machines with JMX
- Version : 2.5
- Date : May 2018
- Author : Damien Dallimore, damien@baboonbones.com
Overview
This app provides the means to :
- connect to any local or remote JVM's JMX server, either via the remote JMX interface,attaching to a local JVM process or using an MX4J HTTP based connector.
- query any MBean running on that server
- extract any MBean attributes (simple, composite or tabular)
- invoke MBean operations
- listen for MBean notifications and declare custom notification filters
- write attributes and operation results out in a default key/value format, or plugin your own custom format, for SPLUNK indexing and searching
- declare clusters of JVM's for large scale JVM deployments
- manage the JMX Modular Input programmatically via the REST API
Tested to date against Hotspot ,JRockit and IBM J9 JVM's.
Currently supports the following JMX Connectors :
- rmi (JSR160 Standard Implementation and MX4J's JSR160 Implementation)
- iiop (JSR160 Standard Implementation and MX4J's JSR160 Implementation)
- local (MX4J only)
- soap (MX4J only)
- hessian (MX4J only)
- burlap (MX4J only)
Dependencies
- Splunk 5.0+ , if you only want to use as a JMX Modular Input
- Splunk 6.0+ , if you want to use the Simple XML dashboards also
- Java Runtime 1.7+
- Supported on all Splunk platforms : Windows, Linux, MacOS, Solaris, FreeBSD, HP-UX, AIX
Setup
- Optionally set your JAVA_HOME environment variable to the root directory of you JRE installation.If you don't set this , the input will look for a default installed java executable on the path.
- Untar the release to your $SPLUNK_HOME/etc/apps directory
- Restart Splunk
Activation Key
You require an activation key to use this App. Visit http://www.baboonbones.com/#activation to obtain a free,non-expiring key
Configuration
The data collection logic is implemented as a modular input.
You can configure your JMX inputs via Manager->DataInputs->JMX
You will need to configure your JMX config file, this is the config file where you specify JMX server(s), MBeans and MBean attributes/operations/notifications.
You can create as many config files,with any name, as you want and place them in the SPLUNK4JMX/bin/config directory
Any changes to the config file at runtime will cause it to be dynamically reloaded
In your setup stanzas, you can optionally specify an alternate config file directory location relative to SPLUNK_HOME ie: etc/apps/someotherapp
This might come in handy if you want to create another Splunk App that depends on this JMX App , and ship your own config files.
The app sets up a default input stanza for you. To get started you should edit the default config.xml file and then enable the stanza.
CONFIG.XML
Note : also refer to the configuration schema documentation in the docs directory
Included in the distribution is a comprehensive sample config.xml
This sample file queries various java.lang MBeans and obtains attributes
You can use this as a basis for creating your own config files for whatever MBeans and attributes/operations you need to poll across your environment
To try out this sample config.xml file , just set the values for your environment in the jmxserver element , and you should be good to go.
"jmxpoller" is the root xml element
A custom formatter can be specified.Only 1 may be specified for the entire configuration.
This is a Java implementation of the "com.dtdsoftware.splunk.formatter.Formatter" interface.
The formatter class must be placed on the java classpath.
If the optional formatter declaration is omitted, then the default formatter will be used.
Refer to the javadoc API in the docs directory for viewing bundled Formatter implementations or the signature of the interfaces for creating your own custom implementations.
1 or more "jmxserver" elements can be included in this XML defintion ie: if you need to accesss multiple different jmx sources within the same scheduled execution
Each "jmxserver" element you declare will be run in parallel.
"jmxuser" and "jmxpass" are optional , if not specified, they will be ignored
"host" can be a hostname, dns alias or ip address.
"jmxport" is the JMX server port to connect to.
"protocol"(default is rmi) is the jmx service protocol to use : rmi, iiop and mx4j specific protocols soap, hessian etc..
For more advanced finer grained control over the format of the jmx service URL you can also specify the "stubSource"(default is jndi), "encodedStub" and url "lookupPath"(default is jmxrmi).
Refer to the schema docs for more details.
Or, to facilitate complete user flexibility, you can enter the full jmx service url as a raw string using the "jmxServiceURL" attribute.
This is raw jmx service url in standard format "service:jmx:protocol:sap"
If this attribute is set, it will override any other set connection related attributes("host", "jmxport", "protocol", "stubSource", "encodedStub" and "lookupPath")
You can also attach directly to a locally running JVM process by specifying the Process ID in the "pid" attribute.
In this case, the "host", "jmxport", "jmxuser","jmxpass" attributes will not be used.
The host value will default to the localhost's name.
For more dynamic flexibility,rather than specifying a static Process ID value in the config xml , you can also specfy the name of a PID file that contains the JVM's Process ID.
You set this in the "pidFile" attribute.
Many long running Java processes output the PID value to a file, particularly if using a JVM Service Wrapper such as Tanuki.
Each time the mod input poller runs, it will dynamically inspect the value of the PID from this file.
This way, you dont have to alter the config.xml each time you do a JVM restart.
You can also specify the path to a command to execute to output the JVM's PID.
You set this in the "pidCommand" attribute.
This command might be a custom script that looks for the JVM process and extracts the PID to STD OUT.
On Linux you could do this with a simple script that uses ps, grep and awk.
Example : ps -eafH | grep "java" | grep -v "grep" | awk '{print $2}'
If the pidCommand outputs multiple PIDs on multiple lines , then Splunk for JMX will handle this by spawning additional JMXServer elements internally in memory.
Optionally you can also have your pid script output a seperate JVM Description for each pid in format "pid,jvmDescription"
Example :
1234,Some JVM Description
3456,Another JVM Description
6789,MOAR JVM Description
Look at the example xml config file in the bin/config directory for usage examples.
Groups of jmxserver's that share that same mbeans can be grouped together in a cluster element, so that you only have to declare the MBeans/MBean attributes/operations once ie: a cluster of JEE appservers, a cluster of Hadoop or Cassandra nodes etc...This will be useful for enterprises with very large scale JVM deployments.
MBean inheritance is supported, so you can have mbeans defined that are common to the cluster, and mbeans that are
specific to a cluster member.
At index time the "host" field is extracted and transformed into the SPLUNK internal host value.
"jvmDescription" is just meta data, useful for searching on in SPLUNK where you might have multiple JVM's on the same host
By default, no timestamp is added , instead relying on the SPLUNK index time as the event time.
However you may customise the 3 bundled formatters and configure them to prepend a timestamp if you wish.
See the examples in the default config.xml file
For MBean definitions , standard JMX object name wildcard patterns * and ? are supported for the "domain" and "properties" attributes
http://download.oracle.com/javase/1,5.0/docs/api/javax/management/ObjectName.html
If no values are specified for the "domain" and "properties" attributes , the value will default to the * wildcard
The MBean's canonical objectname will be written out to SPLUNK.
If you would like the components of the MBean canonical objectname broken out into individual fields, then there is a custom formatter available which you can declare in your config XML file to achieve this :
''<formatter classname="com.dtdsoftware.splunk.formatter.TokenizedMBeanNameFormatter"/>''
Single, composite and tabular MBean attributes are supported.
You can set the "dumpAllAttributes" value to true and all of an MBean's attributes will be extracted.
ie: dump all the attributes in the java.lang domain
''<mbean domain="java.lang" properties="*" dumpallattributes="true"/>''
ie: dump all the attributes in all of the cassandra domains
''<mbean domain="org.apache.cassandra." properties="" dumpallattributes="true"/>''
Or you can specify each individual attribute you want to extract using the "attribute" element.
For attributes that are multi level ie: composite and tabular attributes , then you can use a ":" delimited notation for specifying the attribute name.
See examples of this in config.xml
MBean operation invocation is supported.
You can declare operations that return a result or not , take arguments or not.
Operation overloading is supported for methods of the same name with different signatures.
The following parameter types are supported :
- int
- float
- double
- long
- short
- byte
- boolean
- char
- string
Internally these get autoboxed into their respective Object counterparts.
See config_operations_example.xml for usage scenarios.
MBean notification listening is supported simply by adding a "<notificationlistener/>" element as a child of any MBean.
And you can declare custom notification filters "<notificationlistener filterimplementationclass="com.foo.MyFilter/">" .
The filter is the canonical name of a class implementing the javax.management.NotificationFilter interface.
See config_notifications.xml for an example configuration.
Example output line (default formatter)
host=damien-laptop,jvmDescription="My test JVM",mbean="java.lang:type=Threading",count="47"
SUPPORTED RETURN DATA TYPES FOR OPERATIONS AND ATTRIBUTES
Simple, Composite and Tabular MBean data structures are supported.
The value obtained from the Attribute or as the return type of an Operation can be :
- Any primitive type(Object wrapped)
- String
- Array
- List
- Map
- Set
Collection types will be recursively deeply inspected, so you can have Collections of Collections etc..
JVM "ATTACH" NATIVE LIBRARYS
When connecting directly to local JVM using a process ID, native librarys are used. You shouldn't need to do anything, they are loaded automatically.
Windows : JRE_HOME/bin/attach.dll
Linux : JDK_HOME/jre/lib/i386/libattach.so
NOTE : For some reason the Linux "attach" library is only packaged in the JRE that is part of a JDK install. Weird. So if you don't have libattach.so , you can get it from the JDK and copy it into the jre lib directory.
JMX SERVERS
To set up a JMX server for remote access via jvm system properties at startup, see this :
http://download.oracle.com/javase/1.5.0/docs/guide/management/agent.html#remote
MX4J SUPPORT
If you wish to use MX4J as your JMX implementation for remote connectors(rmi and iiop) or use any of the MX4J specific JMX connectors(soap, burlap, hessian, local), then it is simply a matter of setting the USE_MX4J variable in the SPLUNK4JMX/bin/jmx.py script to True.
For more details about MX4J browse here : http://mx4j.sourceforge.net
Note : If using any of the HTTPS connectors(soap+ssl, hessian+ssl, burlap+ssl), the root certification authority must be present in the trusted certificates, normally stored in the "$JAVA_HOME/jre/lib/security/cacerts" file.
JVM HEAP Settings
The JMX modular input executes all of the stanzas in a single JVM instance.
If you need to boost HEAP size , then you can adjust the variables MIN_HEAP and MAX_HEAP in in the SPLUNK4JMX/bin/jmx.py script
ADDING JAVA CLASSES TO THE APPLICATION CLASSPATH
You can dump a jar file in the "SPLUNK4JMX/bin/lib/" directory and it will be automatically loaded.
Why would you want to do this ? Well perhaps you have created a custom formatter implementation as described above.
ADDING JAVA CLASSES TO THE BOOT CLASSPATH
Any classes that you need prepended to the java bootclasspath should be put in "SPLUNK4JMX/bin/lib/boot".
Why would you want to do this ? Well perhaps you are targeting a JVM with some proprietary JMX logic that requires additional libraries on the JMX client side ie: using the IIOP connector with IBM Websphere products is one such scenario I've encountered.
Logging
Any runtime errors will get written to $SPLUNK_HOME/var/log/splunk/splunkd.log
Troubleshooting
- check your firewall setup
- check the correct spelling/case of your MBeans and MBean attribute/operation definitions
- check your hostname and port
- check your user and pass
- check that your JVM remote JMX access is correctly setup , can you connect using JConsole ?
- check your process ID if trying to attach to a local JVM
- ensure your XML config adheres to the XSD
- look for errors in the log files or via Splunk search in the _internal index
- check for JMX connection timeouts
- check that the MBean server is not adding/removing domains and mbeans whilst you are concurrently polling them
Release Notes
Version 2.5
Added an activation key requirement , visit http://www.baboonbones.com/#activation to obtain a free,non-expiring key
Docs updated
Splunk 7.1 compatible
Version 2.4
Minor HEC data handling tweaks
Version 2.3
Added support to optional output to Splunk via a HEC (HTTP Event Collector) endpoint
Version 2.2.2
Tidied up a dead link in the navigation menu
Version 2.2.1
Fixed some minor typos
Version 2.2
Enabled TLS1.2 support by default.
Made the core Modular Input Framework compatible with latest Splunk Java SDK
Please use a Java Runtime version 7+
If you need to use SSLv3 , you can turn this on in bin/jmx.py
SECURE_TRANSPORT = "tls"
#SECURE_TRANSPORT = "ssl"
Version 2.1
Config file is now dynamically reloaded if it changes
PID File contents are read in on each poller execution
PID Command is executed on each poller execution
PID Command can also return JVM Descriptions in format : 1234,somedescription
Version 2.0.4
Minor change so that when using "dumpAllAttributes" , only READABLE attributes will be polled.
Version 2.0.3
Added a property to optionally specify an alternate config file directory location , relative to SPLUNK_HOME
Version 2.0.2
Put app icon images in the correct directory(this has changed for Splunk 6)
Fixed a classloading clash for direct process attachment jars for win32 vs linux deployments
Version 2.0
NOTE : As per Splunk legal naming standard requirements , this app has now been renamed from "Splunk for JMX" to "Monitoring of Java Virtual Machines with JMX". But it is still functionally the same with everything you know and love , plus more !
Release Features:
Refactored the core data collection to be a Modular Input vs the prior versions' scripted inputs.This is much better for performance as now you can run multiple polling stanzas in 1 single JVM, versus having to fire up a JVM for every time you execute a configuration file
As the Modular Input is a Python wrapper to a JVM instance , all Splunk platforms are now supported
The app can either be run in app mode(with dashboards) or in add-on mode(just the modular input.)
Removed all Advanced XML and replaced with Simple XML
Added support for listening for MBean Notifications
All logging now goes to splunkd.log , searchable via "index=_internal ExecProcessor jmx.py"
All documentation brought up to date.
Version 1.7
Updated to be compatible with Splunk 6
Version 1.6
Tweaked the GC views , Added support to roll out Composite Type arrays
Version 1.5.7
When using direct process attachment and specifying a PID Command script, previously Splunk for JMX would only support a single PID in the command output , now it supports multiple PIDs (delimited by newlines)
Version 1.5.6
Minor cosmetic update , removed paypal links
Version 1.5.5
Added heap/non-heap memory committed charts to the jvm_memory dashboard
Version 1.5.4
Added a setting to SearchSelectLister modules so that their populating searches inherit the parent TimeRangePicker selection
Version 1.5.3
++added a configuration parameter(stripPattern) for Formatter declarations in config.xml that allows you to specify a regex pattern, or list of patterns, that if matched will strip the matched text from the raw MBean attribute/operations values++
Version 1.5.2
++just a tweak to the build, some files were erroneously getting CHMOD'd executable in the tarball++
Version 1.5.1
++Converted all the demo views to Advanced XML and added in dynamic time,host,jvm selectors++The new Advanced XML views use output generated by the TokenizedMBeanNameQuotesStrippedFormatter, this is the formatter now declared in the example config xml files, the DefaultFormatter is still available if you were using and still want to use that, but won't work with the new Advanced XML demo views++
Version 1.5
++Refactored core engine so you can plugin both custom Formatters and custom Transports++Bundled STDout(default), Splunk Rest, TCP,Syslog,File Transport implementations++added ability to add parameters to Formatter and Transport declarations++3 bundled Formatters can be customised with various different parameters in the configuration xml file, such as the ability to prepend a date and specify the date format pattern++updated docs
Version 1.4
Added a setup screen ++ Added a MAX_TIMESTAMP_LOOKAHEAD setting in props.conf to prevent erroneous epoch time rsolutions from non time fields ++ Added links in navigation bar to pdf/html docs
Version 1.3.4
++minor adjustment to logging, no more daily rolling files , now only keeping 1MB with 1 backup as the JMX logs get indexed by Splunk anyway ++ documentation housekeeping
Version 1.3.3
+ More robust handling of primitive type arrays + All Formatter implementations now strip out all newlines and tabs from attribute values and replace with single space characters, thus flattening out all possible attribute values into a single line
Version 1.3.2
+++Added several new demo views, tables, charts...notably, CPU usage per JVM(based on the same algorithm that JConsole uses to derive the JVMs cpu usage), garbage collection, memory pools/memory managers , more thorough OS and Runtime metrics, enhanced connectivity errors page.+++no changes to core engine or config schema, so full backward compatibility maintained
Version 1.3.1
Tweaked the demo forms and navigation bar >>Added a data input to monitor the splunk4jmx error logs and also added view on the navigation menu that reports on jmx connection errors.
Version 1.3
Docs updated
Version 1.2.9
Fixed minor bug in poll_mx4j_jmx.sh script
Version 1.2.8
Added an optional "Formatter" implementation that rather than outputting the mbean name as a single canonical string, breaks up the mbean name into its constituent parts and outputs as individual fields.In your config XML file you can use this formatter by declaring.....<formatter className="com.dtdsoftware.splunk.formatter.TokenizedMBeanNameFormatter" />
Version 1.2.7
Added support for discovering and extracting ALL attributes of an MBean.Provides an alternative to having to explicitly declare each attribute you want to extract.
...a few examples....
dump the attributes from every MBean from every domain :
<mbean domain="*" properties="*" dumpAllAttributes="true" />
....
dump the attributes from all MBeans in the java.lang domain :
<mbean domain="java.lang" properties="*" dumpAllAttributes="true"/>
...
dump the attributes from all MBeans in all of the cassandra domains :
<mbean domain="org.apache.cassandra.*" properties="*" dumpAllAttributes="true" />
Version 1.2.6
MX4J support is now implemented, in addition to the standard JSR160 rmi and iiop connectors , there are now MX4J http/https connectors (hessian, burlap and soap). If your target JVM is using MX4J 3.0.2+ as its JMX implementation, then you can use these additional connectors.
In my testing, these http connectors all work well for simple attributes.
Hessian/Burlap work well with CompositeData attributes, but not so with TabularData attributes
Soap seems to have issues with CompositeData and TabularData attributes.
I would recommend the hessian connector if using an http based connector.
I have also updated the lib layout and scripts to allow easier addition of custom jar files, 3rd party jar files etc...
All documentation updated.
Version 1.2.5
>>Tested against IBM J9 JVMs >>Fixed bug with multiple cluster elements in config xml file >> Tidied up processing of TabularData
Version 1.2.4
>>>Added more thorough Java Collection type support for Attribute value and Operation return types. >>>Can now deal with Arrays, Lists, Maps, Sets and recursively deeply inspect Collections of Collections.>>>Docs updated.
Version 1.2.3
Added support for JMX operations , updated documentation.
Version 1.2.2
Created a PDF User Guide
Version 1.2.1
Added the notion of a "cluster" of JMX servers to the configuration schema.
This will be beneficial when you have many JVMs with the same MBeans ie: JEE App server clusters, large scale deployments of Hadoop/Cassandra nodes on commodity hardware, JVM's across dev/test/qual/prod environments etc..
So the "cluster" element allows you to declare the MBean attributes to extract just once, followed by a list of the JMX servers.Avoids unnecessary repetition and potential bloating of the config file.
For direct attachment to a local JVM using the Process ID , you can now also specify a script to execute to obtain the PID.
This is specified in the "pidCommand" attribute of the jmxserver.
ie: on *Nix a command such as
ps -eafH | grep "com.foo.MyMainClass" | grep -v "grep" | awk '{print $2}'
could be put in a script and used to output the Process ID of the JVM with main class "com.foo.MyMainClass" to STDOUT
Added some more documentation
javadoc API for creating a custom Formatter implementation
schema documentation for the configuration XML
Version 1.2
Added XSD and XSD validation for the config XML
Tidied up bin/config and bin directorys
Changed scripts to take the name of the config file as an argument(set from SPLUNK Manager) , rather than as a variable in the script itself.
Adding logging for error level events
Added Java API to allow users to create a custom "Formatter" for SYSOUT lines that are fed into SPLUNK indexing
Logic to write out host name when connecting to localhost or 127.0.0.1
Version 1.1.2
Tweaked windows script
Version 1.1.1
Tweaked packaging
Version 1.1
1. Added a script to provision running on Windows Operating Systems
2. Added functionality to connect directly to a locally running JVM using the JVM's Process ID or by looking up the Process ID from a PID file. This alleviates the need to remotely enable your JMX server for simple local access.
3. All features tested and working on Windows and Linux , HOTSPOT JVMs.
Version 1.0.2
Updating screenshot for splunkbase page
Version 1.0.1
Added a screenshot to the release