IRI Voracity Data Munging and Masking App for Splunk
Overview
Details
The benefit of all this is seamless, simultaneous operational data-to-information flow ... from Voracity’s fast preparation and protection of big and small data sources to Splunk’s powerful visualizations and adaptive response framework. In a single pass through multiple inputs, Voracity jobs can transform, filter, cleanse, reformat, and stage (wrangle) data for analytics, and de-identify the PII within for compliance and data breaches.
The app takes data produced by IRI jobs defined in the 4GL (*.cl) job scripts of "SortCL"-compatible products in Voracity -- including CoSort, NextForm, FieldShield and RowGen -- and automatically indexes their results into Splunk at specified intervals. This IRI App for Splunk is also capable of running additional command line arguments available to these jobs, such as /WARNINGSON, /STATISTICS, and additional /OUTFILE targets.
Note that this app is a modernized, and extended version of the 2016 (and still available) IRI Add on for Splunk. The add-on provides a similar run-and-index service, but without the newer features or packaging of this new app.
While the app is free, it will not produce any data without an attendant IRI product license. Visit www.iri.com, email voracity@iri.com, or call +1.321.777.8889 for more information..
In Q3’19, IRI created a new, free app for Splunk Enterprise and Enterprise Security users that seamlessly indexes data from any Voracity platform data wrangling or protection job. The app -- available on Splunkbase -- is the modern successor to the 2016 Voracity add-on to Splunk, and an alternative to using Splunk Universal Forwarder to index the data.
The app functions as an invocation and integration point for the execution and use of different SortCL-compatible jobs and their results; i.e.,
IRI CoSort data cleansing and transformation
IRI NextForm data migration and replication
IRI FieldShield PII discovery and de-identification
IRI RowGen test data synthesis and population
The app launches these existing jobs -- usually created graphically in IRI Workbench, an IDE built-on Eclipse, -and automatically ingests and indexes the output data from those jobs into Splunk. From there, the data is ready for Splunk’s usual analytics, visuals and actions.
The app works by running any specified IRI job script as a modular input given its specified location. It also accepts and executes additional SortCL command line arguments to the script, such as /WARNINGSON, /STATISTICS, or another named /OUTFILE target. The data flowing to Splunk through the app should be specified in a structured record format.
Installing the IRI App for Splunk
There are two options to install the IRI app:
download the app from Splunkbase and install it via the “install from file” button within your Splunk instance; or,
search for “IRI Voracity Data Munging and Masking App for Splunk” and have it installed directly.
Creating a Data Input / Modular Input
The input to Splunk is the output of a Voracity job. The first /OUTFILE target in the job must be set to stdout for the data to be indexed into Splunk. Additional command line arguments can be specified in a modular input, including additional /OUTFILE= sections to add other output locations in addition to the pipe into Splunk.
Select the Voracity app from the dropdown menu in the upper-left corner of Splunk. “Inputs” should be the default page of the app. To create an input, select the “Create New Input” button and select the IRI Voracity (SortCL) Job to run, which will produce the input data for Splunk.
Enter a unique name for the data input (without spaces). Set the interval and index location for the data input.
The interval indicates how often the modular input will be run. The modular input involves invoking SortCL from the command line with a specified IRI Job location and any other specified additional command line arguments. The index selector specifies what index the data will be stored in. The IRI Job Location field must be filled with the full path to the IRI job (.*cl file).
Additionally, seven different command line arguments can be added to the Modular Input to be run. These commands include /STATISTICS=, /WARNINGSON, /WARNINGSOFF, /DEBUG, /RC, /MONITOR=, and /OUTFILE=. Background on the SortCL program and its syntax can be found in the CoSort overview booklet.
An extra file path must be specified if a command ends in '='. This field should be left blank if no command is selected or a command that does not end in '=' is selected.
Finally, click the “add” button at the bottom of the dialog. The name of your new input (*.scl job script file) should now be listed in the data inputs table.
Searching the Results
To search the results now available in Splunk, select “Search” from the navbar of the IRI App. Type “Source= “name_of_modular_input_type://modular_input_name”” to search the data indexed by the modular input.
This can also be accessed by clicking “Data Summary” and selecting the source from the “Sources” menu. Use Splunk commands such as |stats, |chart, and |timechart to help visualize your data. See this article for a prior example.
Visualization
Here is an example of U.S. president data from 1900 onward sorted in a simple CoSort SortCL job, and then indexed and visualized by Splunk:
Action
In addition to visual analytics, you can also use the Splunk Adaptive Response Framework or a Phantom Playbook to take action on production or log data prepared in Voracity across a wide range of industries and applications. Examples would be wrangling customer transactions for Splunk to flag for promotion and pricing decisions, and MQTT or Kafka fed sensor data aggregated (and anonymized) by Voracity for Splunk to use in diagnostic or preventive alerts.
Contact voracity@iri.com if you would like help building out data preparation, presentation, and prescriptive scenarios using Voracity and Splunk.
Release Notes
Version 1.0.2
Updated app icon
Version 1.0.1
Fixed permissions issues within the app.
Version 1.0.0
V 1.0.0