Information

-Title: Dell EMC VMAX Add-on for Splunk Enterprise
-Add-on Version: 2.0.7
-Document Version: 2.0.0
-Date: 18th June 2019
-Vendor Products: Dell EMC VMAX3, All-Flash, and PowerMax
-Visible in Splunk Web: Yes. This add-on contains set-up configuration options.

Contact

For any and all issues or queries, please contact vmax.splunk.support@emc.com.
Include as much information as possible about the issue, the Operating System
and Splunk versions, and associated log files listed in the troubleshooting
section.

• Response Time: Within 48hrs
• How we track issues: Internal Bug tracking system

Hours of Operation & Holidays Observed 2018

All times are specific to Ireland (GMT+00:00)

Working Hours:
Monday : 09:00 - 17:00
Tuesday : 09:00 - 17:00
Wednesday : 09:00 - 17:00
Thursday : 09:00 - 17:00
Friday : 09:00 - 16:00
Saturday : Closed
Sunday : Closed

Holidays Observed 2019/20:
Date Weekday Holiday Name Holiday Type
Jan 1 Monday New Year's Day National holiday
Mar 18 Monday St. Patrick's Day Observance
Apr 22 Monday Easter Monday National holiday
May 6 Monday May Day National holiday
Jun 3 Monday June Bank Holiday National holiday
Aug 5 Monday August Bank Holiday National holiday
Oct 28 Monday October Bank Holiday National holiday
Dec 24 Monday Christmas Eve Observance
Dec 25 Tuesday Christmas Day National holiday
Dec 26 Wednesday St. Stephen's Day National holiday
Dec 31 Monday New Year's Eve Observance
Jan 1 Monday New Year's Day National holiday

About the VMAX Add-on for Splunk Enterprise

The Splunk Technology Add-on (TA) for Dell EMC VMAX allows a Splunk® Enterprise
administrator to collect inventory, performance information, and summary
information from VMAX storage arrays. You can then directly analyse the data or
use it as a contextual data feed to correlate with other operational or security
data in Splunk Enterprise.

The Splunk VMAX TA is configured to report events in 5 minute intervals which is
the lowest possible granularity for performance metrics reporting. Event metric
values are representative of the value recorded at that point in time on the
VMAX. Values shown for an event in Splunk at 10:00am represent their respective
values at 10:00am on the VMAX.

Download the Splunk Add-on for Dell EMC VMAX from Splunkbase.

Release Notes

Version 2.0.7 of the VMAX TA is compatible with:
-Splunk Platform: 6.4 and above
-CIM: 4.6 and above
-OS Platforms: Splunk compatible platforms
-Vendor Hardware: Dell EMC VMAX3 and VMAX All-Flash
-Vendor Software: Solutions Enabler 8.4.0.7 and newer
Unisphere for VMAX 8.4.0.10 and newer

Third-Party Software Attributions

Version 2.0.7 of the Splunk VMAX TA incorporates the following bundled
non-standard (packaged by default) third-party libraries:
-Requests - http://docs.python-requests.org
-IP Address - https://github.com/phihag/ipaddress

VMAX TA Requirements and Considerations

-Splunk Administrator Requirements
To install and configure the Splunk VMAX TA, you must have admin privileges.

-Hardware & Software
As this add-on runs on Splunk Enterprise, all Splunk Enterprise system
requirements apply. Dell EMC VMAX-3 Series and VMAX All-Flash are supported by
the VMAX TA. Solutions Enabler 8.4.0.7 or newer and Unisphere for VMAX 8.4.0.10
and newer are required.

-Additional Recommendations
The add-on does not require the ability to modify VMAX configuration. It is
highly recommended that you create a read-only user account for Splunk in
Unisphere for VMAX.

Before any metrics can be collected from a VMAX you must also ensure that the
VMAX is registered to collect performance metrics. This is enabled from within
the Unisphere for VMAX Web UI. For more information see the section ‘Enabling
Performance Metrics Collection’ below.

Performance of data collection is dependent on many factors, such as VMAX system
load, Splunk Enterprise system load, and environmental factors such as network
latency. Performance considerations are discussed in the section ‘Running Sizer’
and how to mitigate these in your environment for best possible performance from
metric collections.

Enabling Performance Metrics Collection

To enable Splunk to gather performance data on the VMAX(s) in your environment
it is necessary to first enable performance metrics for the given VMAX(s) by
registering the system to collect performance data. To register your VMAX(s)
follow these steps:

1. Log in to Unisphere and from the main home screen identify the VMAX you want
   to add to Splunk

2. In the VMAX’s summary panel, under ‘System Utilization’ click ‘Register this
   system to collect performance metrics’

3. A new page for ‘System Registrations’ will open where you will see your VMAX
   listed. Click the VMAX to highlight it and click ‘Register’

4. When the ‘Registration’ dialogue window opens, select the check-box for ‘Root
   Cause Analysis’ and click OK.

5. If the registration process is successful, you will see a green dot to
   signify root-cause analysis is enabled.

6. With the registration process complete, leave Unisphere for 8-24hrs to start
   gathering performance metrics before adding the VMAX to Splunk. Performance
   metrics collection is not immediate, for more information please refer to the
   ‘Performance Management – Metrics’ section of the ‘Unisphere 8.4 Online Help’
   guide available on support.emc.com

SSL Configuration

SSL is enabled by default in the VMAX TA when adding inputs. In order to
retrieve the required certificate from Unisphere follow the following steps:

1. Get the CA certificate of the Unisphere server. This pulls the CA cert file
   and saves it as .pem file:

openssl s_client -showcerts -connect {unisphere_host}:8443 </dev> \

/dev/null|openssl x509 -outform PEM >{unisphere_host}.pem

Where {unisphere_host} is the IP address or hostname of the Unisphere instance.

1. OPTIONAL: If you want to add the cert to the system certificate bundle so no
   certificate path is specified in the VMAX data input, copy the .pem file to the
   system certificate directory as a .crt file:

{unisphere_host}.pem /usr/share/ca-certificates/{unisphere_host}.crt

1. OPTIONAL: Update CA certificate database with the following commands:

dpkg-reconfigure ca-certificates

update-ca-certificates

Check that the new {unisphere_host}.crt will activate by selecting ask on the
dialog. If it is not enabled for activation, use the down and up keys to select,
and the space key to enable or disable.

1. If steps 2 & 3 are skipped and instead the cert from step 1 will just remain
   in a local directory, you can specify the location of the .pem cert in the VMAX
   data input setting 'SSL Cert Location'. Otherwise, leave ‘SSL Cert Location’
   blank and ‘Enable SSL’ enabled to use the cert from the system certificate
   bundle.

VMAX for Splunk Sizer

An additional script has been included with the VMAX TA to help determine the
optimum reporting interval required for your VMAX data inputs.

This sizer is meant to be used with one instance of Unisphere at a time, it is
not concerned with performance across multiple instances of Unisphere as this
would fall under the remit of Splunk performance.

This sizer will help set the VMAX TA input intervals so that each input has
enough time to complete before the reporting interval is exceeded and metric
collection intervals are missed.

Metrics collection run times depend entirely on the environment, the VMAX
itself, how heavily utilised and loaded with resources it is, so there is no one
size fits all option. This script will simulate Splunk and gather summary and
performance metrics from an instance of Unisphere and VMAX(s) of your choosing.
These collection runs will also run concurrently as Splunk does. When complete,
information will be output as to how long metric collection lasted for a given
VMAX, and the recommended reporting interval time.

To run VMAX for Splunk sizer script, you will require Python 2.7 and the Python
Requests library:

• Python 2.7 - https://www.python.org/download/releases/2.7
• Requests - https://pypi.python.org/pypi/requests

To run the sizer script, follow the steps below:

1. Navigate to the VMAX TA folder containing the sizer script and configuration
   file:

cd {splunk_dir}/etc/apps/TA-DellEMC-VMAX/bin/sizer

1. Open the vmax_splunk_sizer_config.ini configuration file for editing.

2. Under [ENVIRONMENT_SETTINGS] set:
   • The Unisphere IP address or hostname
   • The Unisphere port (default is 8443)
   • The Unisphere username & password
   • The required SSL setup:
   o If you require no SSL, set this to False
   o If you have an SSL cert loaded into the system bundle, set this to
   True
   o If you have an SSL cert but want to specify the path, set this to
   the path to the cert
   • Your VMAX numerical IDs, for more than one VMAX separate with a comma

3. Under [REPORTING_LEVELS], if you want to turn on or off any specific
   reporting level change the value to False

4. Debug mode is not necessary unless diagnosing an issue with VMAX for Splunk
   support, but if you would like to see all calls output to screen, change this
   to True

5. With all the environment settings configured, run the VMAX for Splunk
   environment sizer script using the python file 'rest_vmax_splunk_sizer.py'

$ python rest_vmax_splunk_sizer.py

1. Once the script has run to completion, details of the metrics collection run
   will be output to the screen along with recommendations on the optimum reporting
   interval for each VMAX.

Source Type for the VMAX Add-on for Splunk

The VMAX TA provides the index-time and search-time knowledge for inventory,
performance metrics, and summary information. By default, all VMAX data is
indexed into the default Splunk index, this is the ‘main’ index unless changed
by the admin.

The source type used for the Splunk Add-on for VMAX is dellemc:vmax:rest. All
events are in key/value pair formats. All events have an assigned
'reporting_level' which indicates the level at which the event details, along with
the associated VMAX array ID & if reporting at lower levels, the object ID e.g.
Storage Group, Director, Host.

The add-on collects many different kinds of events for VMAX, including
performance, inventory, and summary metrics. Depending on the activity of the
Port Groups & Initiators in your environment, there may be events where there
are no performance metrics collected. This can be confirmed if there is a metric
present in the event named {reporting_level}_perf_details with a value of False
(where reporting_level is the reporting level of the event itself). For more
information see the section ‘Active vs. Inactive Object Performance Reporting’.

Active vs. Inactive Object Performance Reporting

To limit the amount of data collected and stored on a VMAX, only active Port
Groups, Hosts, and Initiators are reported against for performance metrics.
Inactivity is determined by no activity being recorded by performance monitors
for a specified amount of time. The VMAX TA ingests a wide range of metrics
across each of the reporting levels. To get detailed definitions of each of the
performance metrics see the ‘Performance Management’ section of the
‘Unisphere 8.4 Online Help’ guide available on support.emc.com

This is not enforced by Splunk but is the behaviour of the VMAX, recording zero
values for every Port Group, Host, and Initiator in an environment would very
quickly fill databases with useless data.

When the VMAX TA is collecting information on the Port Groups, Hosts, or
Initiators in your environment, it will first obtain a list of all objects for
each reporting level. Using this list, calls will be to Unisphere for
performance metrics for each, if an object is inactive, no performance metrics
will be returned. This inactivity is reflected in the VMAX events through the
key/value pairs below.

{reporting_level}_perf_details: false
{reporting_level}_perf_message: No active {reporting_level} performance data
available

Troubleshooting the VMAX Add-on for Splunk

The Splunk VMAX TA has been developed to give the end-user as much detail as
possible about the activity of the add-on in their environment. All add-on
logged events will either be marked as info, error, or critical depending on the
nature of the event. If you are having any issues with the add-on, the logs will
be able to give you precise information as to the cause of the problem. These
issues could be related, but not limited to:

• Incorrect Unisphere configuration or username/password combination
• Incorrect SSL setup
• Incorrect Array ID
• VMAX is not performance registered
• Performance metrics timestamp is not up-to-date

The two log files that you can use to diagnose problems with this add-on are:
• /{splunk_install_dir}/splunk/var/log/splunk/ta_dellemc_vmax_inputs.log
• /{splunk_install_dir}/splunk/var/log/splunk/splunkd.log

Before the add-on successfully runs for the first time, error logs go to
splunkd.log. After the add-on successfully runs, error logs go to
ta_dellemc_vmax_inputs.log.

Performance Metrics Definitions

The VMAX TA ingests a wide range of metrics across each of the reporting levels.
To get detailed definitions of each of the performance metrics see the
‘Performance Management – Metrics’ section of the ‘Unisphere 8.4 Online Help’
guide available on support.emc.com

Contact

For any and all issues or queries, please contact vmax.splunk.support@emc.com.
Include as much information as possible about the issue, the Operating System
and Splunk versions, and associated log files listed in the troubleshooting
section.