Information

-Title: Dell EMC VMAX Add-on for Splunk Enterprise
-Add-on Version: 2.0.4
-Document Version: 2.0.0
-Date: 3rd Sept 2018
-Vendor Products: Dell EMC VMAX3, VMAX All-Flash, PowerMax
-Visible in Splunk Web: Yes. This add-on contains set-up configuration options.

Download the Splunk Add-on for Dell EMC VMAX from Splunkbase.
Get the user guide with demonstrative screenshots here: https://community.emc.com/docs/DOC-63597

SECURITY NOTICE - It has been brought to my attention that there is a known bug in the Splunk platform with adding data inputs through Settings>Data Inputs and passwords being saved in clear text. -DO NOT ADD VMAX INPUTS VIA THE SPLUNK DATA INPUTS PAGE-ADD THROUGH THE VMAX ADD-ON DASHBOARD MENU ITEM-

For a full step-by-step guide to setup both the VMAX for Splunk add-on and app check out the blog article on DECN: https://community.emc.com/people/michaelmcaleer1/blog/2018/04/20/vmax-for-splunk-20

Contact

For any and all issues or queries, please contact vmax.splunk.support@emc.com.
Include as much information as possible about the issue, the Operating System
and Splunk versions, and associated log files listed in the troubleshooting
section.

• Response Time: Within 48hrs
• How we track issues: Internal Bug tracking system

Hours of Operation & Holidays Observed 2018

All times are specific to Ireland (GMT+00:00)

Working Hours:
Monday : 09:00 - 17:00
Tuesday : 09:00 - 17:00
Wednesday : 09:00 - 17:00
Thursday : 09:00 - 17:00
Friday : 09:00 - 16:00
Saturday : Closed
Sunday : Closed

Holidays Observed 2018:
Date Weekday Holiday Name Holiday Type
Jan 1 Monday New Year's Day National holiday
Mar 19 Monday St. Patrick's Day Observance
Mar 30 Friday Good Friday Observance
Apr 1 Sunday Easter Observance
Apr 2 Monday Easter Monday National holiday
May 7 Monday May Day National holiday
Jun 4 Monday June Bank Holiday National holiday
Aug 6 Monday August Bank Holiday National holiday
Oct 29 Monday October Bank Holiday National holiday
Dec 24 Monday Christmas Eve Observance
Dec 25 Tuesday Christmas Day National holiday
Dec 26 Wednesday St. Stephen's Day National holiday
Dec 31 Monday New Year's Eve Observance

About the VMAX Add-on for Splunk Enterprise

The Splunk Technology Add-on (TA) for Dell EMC VMAX allows a Splunk® Enterprise
administrator to collect inventory, performance information, and summary
information from VMAX storage arrays. You can then directly analyse the data or
use it as a contextual data feed to correlate with other operational or security
data in Splunk Enterprise.

The Splunk VMAX TA is configured to report events in 5 minute intervals which is
the lowest possible granularity for performance metrics reporting. Event metric
values are representative of the value recorded at that point in time on the
VMAX. Values shown for an event in Splunk at 10:00am represent their respective
values at 10:00am on the VMAX.

Download the Splunk Add-on for Dell EMC VMAX from Splunkbase.

Release Notes

Version 2.0.4 of the VMAX TA is compatible with:
-Splunk Platform: 6.4 and above
-CIM: 4.6 and above
-OS Platforms: Splunk compatible platforms
-Vendor Hardware: Dell EMC VMAX3 and VMAX All-Flash
-Vendor Software: Solutions Enabler 8.4.0.7 and newer
Unisphere for VMAX 8.4.0.10 and newer

VMAX TA Requirements and Considerations

-Splunk Administrator Requirements
To install and configure the Splunk VMAX TA, you must have admin privileges.

-Hardware & Software
As this add-on runs on Splunk Enterprise, all Splunk Enterprise system
requirements apply. Dell EMC VMAX-3 Series and VMAX All-Flash are supported by
the VMAX TA. Solutions Enabler 8.4.0.7 or newer and Unisphere for VMAX 8.4.0.10
and newer are required.

-Additional Recommendations
The add-on does not require the ability to modify VMAX configuration. It is
highly recommended that you create a read-only user account for Splunk in
Unisphere for VMAX.

Before any metrics can be collected from a VMAX you must also ensure that the
VMAX is registered to collect performance metrics. This is enabled from within
the Unisphere for VMAX Web UI. For more information see the section ‘Enabling
Performance Metrics Collection’ below.

Performance of data collection is dependent on many factors, such as VMAX system
load, Splunk Enterprise system load, and environmental factors such as network
latency. Performance considerations are discussed in the section ‘Running Sizer’
and how to mitigate these in your environment for best possible performance from
metric collections.

Enabling Performance Metrics Collection

To enable Splunk to gather performance data on the VMAX(s) in your environment
it is necessary to first enable performance metrics for the given VMAX(s) by
registering the system to collect performance data. To register your VMAX(s)
follow these steps:

1. Log in to Unisphere and from the main home screen identify the VMAX you want
   to add to Splunk

2. In the VMAX’s summary panel, under ‘System Utilization’ click ‘Register this
   system to collect performance metrics’

3. A new page for ‘System Registrations’ will open where you will see your VMAX
   listed. Click the VMAX to highlight it and click ‘Register’

4. When the ‘Registration’ dialogue window opens, select the check-box for ‘Root
   Cause Analysis’ and click OK.

5. If the registration process is successful, you will see a green dot to
   signify root-cause analysis is enabled.

6. With the registration process complete, leave Unisphere for 8-24hrs to start
   gathering performance metrics before adding the VMAX to Splunk. Performance
   metrics collection is not immediate, for more information please refer to the
   ‘Performance Management – Metrics’ section of the ‘Unisphere 8.4 Online Help’
   guide available on support.emc.com

Installation and configuration overview for the Splunk Add-on for VMAX

As there are no dependencies required for the installation of the Splunk Add-on
for VMAX, the set-up is completed entirely from within the Splunk web UI.

1. From your Splunk home screen, click the cog icon beside ‘Apps’ to navigate
   to the ‘Manage Apps’ section.

2. Within the ‘Manage Apps’ section, click the button ‘Install App from file’.

3. Click ‘Choose File’, select the VMAX Add-on for Splunk, and click ‘Upload’.

4. Once the upload is complete you will be prompted to restart Splunk to
   complete the installation, click ‘Restart now’. When Splunk restarts, navigate
   back to the home screen and you will now see a dashboard panel for the VMAX TA.
   Click on the panel to start adding your VMAX(s) to Splunk.

5. Once opened, you can add VMAX(s) to Splunk by clicking on the
   ‘Create New Input’ button in the top right of the UI.

6. To add a VMAX to Splunk, you must enter a number of details into Splunk about
   the instance of Unisphere used, VMAX details, SSL details, and reporting metrics
   configuration. The table below lists each option, its default value if there is
   one, and a description of the option. Once all options are set, click ‘Add’ to
   add the VMAX as a data input to Splunk. A breakdown of all options can be found
   in the TA README.txt file.

7. To add another VMAX to the TA, repeat steps 5-6 as many times as necessary.

8. When all VMAX(s) have been added to the TA, you will see them listed within
   the TA. From here you can enable, disable, or edit the options for a given VMAX
   after it has been configured.

9. Once a VMAX has been added to the VMAX TA, it starts gathering information
   immediately. To access that data, use Splunk Search to start looking at VMAX
   related events using the SPL query: sourcetype="dellemc:vmax:rest”

SSL Configuration

SSL is enabled by default in the VMAX TA when adding inputs. In order to
retrieve the required certificate from Unisphere follow the following steps:

Get the CA certificate of the Unisphere server. This pulls the CA cert file
and saves it as .pem file:

$ openssl s_client -showcerts -connect {unisphere_host}:8443 </dev> \
/dev/null|openssl x509 -outform PEM >{unisphere_host}.pem

Where {unisphere_host} is the IP address or hostname of the Unisphere instance.

OPTIONAL: If you want to add the cert to the system certificate bundle so no
certificate path is specified in the VMAX data input, copy the .pem file to the
system certificate directory as a .crt file:

$ cp {unisphere_host}.pem /usr/share/ca-certificates/{unisphere_host}.crt

OPTIONAL: Update CA certificate database with the following commands:

$ sudo dpkg-reconfigure ca-certificates
$ sudo update-ca-certificates

Check that the new {unisphere_host}.crt will activate by selecting ask on the
dialog. If it is not enabled for activation, use the down and up keys to select,
and the space key to enable or disable.

If the optional steps are skipped and instead the .pem cert from will just remain
in a local directory, you can specify the location of the .pem cert in the VMAX
data input setting 'SSL Cert Location'. Otherwise, leave ‘SSL Cert Location’
blank and ‘Enable SSL’ enabled to use the cert from the system certificate
bundle.

VMAX for Splunk Sizer

An additional script has been included with the VMAX TA to help determine the
optimum reporting interval required for your VMAX data inputs.

This sizer is meant to be used with one instance of Unisphere at a time, it is
not concerned with performance across multiple instances of Unisphere as this
would fall under the remit of Splunk performance.

This sizer will help set the VMAX TA input intervals so that each input has
enough time to complete before the reporting interval is exceeded and metric
collection intervals are missed.

Metrics collection run times depend entirely on the environment, the VMAX
itself, how heavily utilised and loaded with resources it is, so there is no one
size fits all option. This script will simulate Splunk and gather summary and
performance metrics from an instance of Unisphere and VMAX(s) of your choosing.
These collection runs will also run concurrently as Splunk does. When complete,
information will be output as to how long metric collection lasted for a given
VMAX, and the recommended reporting interval time.

To run VMAX for Splunk sizer script, you will require Python 2.7 and the Python
Requests library:

• Python 2.7 - https://www.python.org/download/releases/2.7
• Requests - https://pypi.python.org/pypi/requests

To run the sizer script, follow the steps below:

1. Navigate to the VMAX TA folder containing the sizer script and configuration
   file:

cd {splunk_dir}/etc/apps/TA-DellEMC-VMAX/bin/sizer

1. Open the vmax_splunk_sizer_config.ini configuration file for editing.

2. Under [ENVIRONMENT_SETTINGS] set:
   • The Unisphere IP address or hostname
   • The Unisphere port (default is 8443)
   • The Unisphere username & password
   • The required SSL setup:
   o If you require no SSL, set this to False
   o If you have an SSL cert loaded into the system bundle, set this to
   True
   o If you have an SSL cert but want to specify the path, set this to
   the path to the cert
   • Your VMAX numerical IDs, for more than one VMAX separate with a comma

3. Under [REPORTING_LEVELS], if you want to turn on or off any specific
   reporting level change the value to False

4. Debug mode is not necessary unless diagnosing an issue with VMAX for Splunk
   support, but if you would like to see all calls output to screen, change this
   to True

5. With all the environment settings configured, run the VMAX for Splunk
   environment sizer script using the python file 'rest_vmax_splunk_sizer.py'

$ python rest_vmax_splunk_sizer.py

1. Once the script has run to completion, details of the metrics collection run
   will be output to the screen along with recommendations on the optimum reporting
   interval for each VMAX.

Troubleshooting the VMAX Add-on for Splunk

The Splunk VMAX TA has been developed to give the end-user as much detail as
possible about the activity of the add-on in their environment. All add-on
logged events will either be marked as info, error, or critical depending on the
nature of the event. If you are having any issues with the add-on, the logs will
be able to give you precise information as to the cause of the problem. These
issues could be related, but not limited to:

• Incorrect Unisphere configuration or username/password combination
• Incorrect SSL setup
• Incorrect Array ID
• VMAX is not performance registered
• Performance metrics timestamp is not up-to-date

The two log files that you can use to diagnose problems with this add-on are:
• /{splunk_install_dir}/splunk/var/log/splunk/ta_dellemc_vmax_inputs.log
• /{splunk_install_dir}/splunk/var/log/splunk/splunkd.log

Before the add-on successfully runs for the first time, error logs go to
splunkd.log. After the add-on successfully runs, error logs go to
ta_dellemc_vmax_inputs.log.

Contact

For any and all issues or queries, please contact vmax.splunk.support@emc.com.
Include as much information as possible about the issue, the Operating System
and Splunk versions, and associated log files listed in the troubleshooting
section.