icon/x Created with Sketch.

Splunk Cookie Policy

We use our own and third-party cookies to provide you with a great online experience. We also use these cookies to improve our products and services, support our marketing campaigns, and advertise to you on our website and other websites. Some cookies may continue to collect information after you have left our website. Learn more (including how to update your settings) here.
Accept Cookie Policy

Accept License Agreements

This app is provided by a third party and your right to use the app is in accordance with the license provided by that third-party licensor. Splunk is not responsible for any third-party apps and does not provide any warranty or support. If you have any questions, complaints or claims with respect to this app, please contact the licensor directly.

Thank You

Downloading NLP Text Analytics
SHA256 checksum (nlp-text-analytics_102.tgz) 635bf9f9ca41deb4c5576529ebe0865e550696fedd7306d95ed2801596291975 SHA256 checksum (nlp-text-analytics_101.tgz) 54d0baa0e5b53927054d445a3226ddf17e2dced670953c079a3501eb4375905c SHA256 checksum (nlp-text-analytics_100.tgz) 10c1d1110efb720a82881e15be54cbbfec4ce1519174890ae0700c82cdc21b3e SHA256 checksum (nlp-text-analytics_095.tgz) 66d8364da8a94e6d5fda4c0ee99b74c50a8c3b8d7d1c75625679951d5419c05f SHA256 checksum (nlp-text-analytics_094.tgz) 5c5c807a3a8e7c42e47ea5f4c1475430e032f378c52a8313559285cb51a93c2b
To install your download
For instructions specific to your download, click the Details tab after closing this window.

Flag As Inappropriate

NLP Text Analytics

Splunk AppInspect Passed
Admins: Please read about Splunk Enterprise 8.0 and the Python 2.7 end-of-life changes and impact on apps and upgradeshere.
Overview
Details
Have you ever wanted to perform advanced text analytics inside Splunk? Splunk has some ways to handle text but also lacks some more advanced features that NLP libraries can offer. This can also benefit use-cases that involve using Splunk’s Machine Learning Toolkit (https://splunkbase.splunk.com/app/2890/). The intent of this app is to provide a simple interface for analyzing text in Splunk using python natural language processing libraries (currently just NLTK 3.3). The app provides custom commands and dashboards to show how to use.

See related Splunk blog https://www.splunk.com/blog/2019/04/11/let-s-talk-about-text-baby.html

The intent of this app is to provide a simple interface for analyzing text in Splunk using python natural language processing libraries (currently just NLTK 3.3) and Splunk's Machine Learning Toolkit. The app provides custom commands and dashboards to show how to use.

Version: 1.0.0

Author: Nathan Worsham
Created for MSDS692 Data Science Practicum I at Regis University, 2018
See associated blog for detailed information on the project creation.

Update
Additional content (combined features algorithms) created for MSDS696 Data Science Practicum II at Regis University, 2018
See associated blog for detailed information on the project creation and associated Splunk blog.
This app was part of the basis for a breakout session at Splunk Conf18 I was lucky enough to present at--Extending Splunk MLTK using GitHub Community.
Session Slides
Session Recording

Description and Use-cases

Have you ever wanted to perform advanced text analytics inside Splunk? Splunk has some ways to handle text but also lacks some more advanced features that NLP libraries can offer. This can also benefit use-cases that involve using Splunk’s ML Toolkit.

Requirements

Splunk ML Toolkit 3.2 or greater https://splunkbase.splunk.com/app/2890/
Wordcloud Custom Visualization https://splunkbase.splunk.com/app/3212/
Parallel Coordinates Custom Visualization https://splunkbase.splunk.com/app/3137/
Force Directed App For Splunk https://splunkbase.splunk.com/app/3767/
Halo - Custom Visualization https://splunkbase.splunk.com/app/3514/
Sankey Diagram - Custom Visualization https://splunkbase.splunk.com/app/3112/

How to use

Install

Normal app installation can be followed from https://docs.splunk.com/Documentation/AddOns/released/Overview/AboutSplunkadd-ons. Essentially download app and install from Web UI or extract file in $SPLUNK_HOME/etc/apps folder.

Example Texts

The app comes with example Gutenberg texts formatted as CSV lookups along with the popular "20 newsgroups" dataset. Load them with the syntax | inputlookup <filename.csv>

Text Names
20newsgroups.csv
moby_dick.csv
peter_pan.csv
pride_prejudice.csv

Custom Commands

bs4

Description

A wrapper for BeautifulSoup4 to extract html/xml tags and text from them to use in Splunk. A wrapper script to bring some functionality from BeautifulSoup to Splunk. Default is to get the text and send it to a new field 'get_text', otherwise the selection is returned in a field named 'soup'. Default is to use the 'lxml' parser, though you can specify others, 'html5lib' is not currently included. The find methods can be used in conjuction, their order of operation is find > find_all > find_child > find children. Each option has a similar named option appended '_attrs' that will accept inner and outer quoted key:value pairs for more precise selections.

Syntax

*| bs4 textfield=<field> [get_text=<bool>] [get_text_label=<string>] [parser=<string>] [find=<tag>] [find_attrs=<quoted_key:value_pairs>] [find_all=<tag>] [find_all_attrs=<quoted_key:value_pairs>] [find_child=<tag>] [find_child_attrs=<quoted_key:value_pairs>] [find_children=<tag>] [find_children_attrs=<quoted_key:value_pairs>]

Required Arguments

textfield
Syntax: textfield=<field>
Description: The search field that contains the text that is the target.
Usage: Option only takes a single field

Optional Arguments

get_text
Syntax: get_text=<bool>
Description: If true, returns text minus html/xml formatting for given selection and places in field get_text otherwise returns the selection in a field called soup.
Usage: Boolean value. True or False; true or false, t or f, 0 or 1
Default: True

get_text_label
Syntax: get_text_label=<string>
Description: If get_text is true, sets the label for the return field.
Usage: Boolean value. True or False; true or false, t or f, 0 or 1
Default: get_text

parser
Syntax: parser=<string>
Description: Corresponds to parsers listed here (currently html5lib not packaged with so not an option).
Usage: Possible values are html.parser, lxml, lxml-xml, or xml
Default: lxml

find
Syntax: find=<tag>
Description: Corresponds to the name attribute of BeautifulSoup's find method.
Usage: HTML or XML element name

find_attrs
Syntax: find_attrs=<quoted_key:value_pairs>
Description: Corresponds to the attrs attribute of BeautifulSoup's find method. Expects inner and outer quoted key:value pairs comma-separated but contained in outer quotes.
Usage: "'key1':'value1','key2':'value2'"

find_all
Syntax: find_all=<tag>
Description: Corresponds to the name attribute of BeautifulSoup's find_all method. Order of operation is find > find_all > find_child > find_children so can be used in conjunction.
Usage: HTML or XML element name

find_all_attrs
Syntax: find_all_attrs=<quoted_key:value_pairs>
Description: Corresponds to the attrs attribute of BeautifulSoup's find_all method. Expects inner and outer quoted key:value pairs comma-separated but contained in outer quotes.
Usage: "'key1':'value1','key2':'value2'"

find_child
Syntax: find_child=<tag>
Description: Corresponds to the name attribute of BeautifulSoup's find_child method. Order of operation is find > find_all > find_child > find_children so can be used in conjunction.
Usage: HTML or XML element name

find_child_attrs
Syntax: find_child_attrs=<quoted_key:value_pairs>
Description: Corresponds to the attrs attribute of BeautifulSoup's find_child method. Expects inner and outer quoted key:value pairs comma-separated but contained in outer quotes.
Usage: "'key1':'value1','key2':'value2'"

find_children
Syntax: find_children=<tag>
Description: Corresponds to the name attribute of BeautifulSoup's find_children method. Order of operation is find > find_all > find_child > find_children so can be used in conjunction.
Usage: HTML or XML element name

find_children_attrs
Syntax: find_children_attrs=<quoted_key:value_pairs>
Description: Corresponds to the attrs attribute of BeautifulSoup's find_children method. Expects inner and outer quoted key:value pairs comma-separated but contained in outer quotes.
Usage: "'key1':'value1','key2':'value2'"

cleantext

Description

Tokenize and normalize text (remove punctuation, digits, change to base_word). Different options result in better and slower cleaning. base_type="lemma_pos" being the slowest option, base_type="lemma" assumes every word is a noun, which is faster but still results in decent lemmatization. Many fields have a default already set, textfield is only required field. By default results in a multi-valued field which is ready for used with stats count by. Optionally return special fields for analysis--pos_tags and ngrams.

Syntax

*| cleantext textfield=<field> [keep_orig=<bool>] [default_clean=<bool>] [remove_urls=<bool>] [remove_stopwords=<bool>] [base_word=<bool>] [base_type=<string>] [mv=<bool>] [force_nltk_tokenize=<bool>] [pos_tagset=<string>] [custom_stopwords=<comma_separated_string_list>] [term_min_len=<int>] [ngram_range=<int>-<int>] [ngram_mix=<bool>]

Required Arguments

textfield
Syntax: textfield=<field>
Description: The search field that contains the text that is the target of the analysis.
Usage: Option only takes a single field

Optional Arguments

keep_orig
Syntax: keep_orig=<bool>
Description: Maintain a copy of the original text for comparison or searching into field called orig_text.
Usage: Boolean value. True or False; true or false, t or f, 0 or 1
Default: False

default_clean
Syntax: default_clean=<bool>
Description: Perform basic text cleaning--lowercase, remove punctuation and digits, and tokenization.
Usage: Boolean value. True or False; true or false, t or f, 0 or 1
Default: True

remove_urls
Syntax: remove_urls=<bool>
Description: Before cleaning remove html links.
Usage: Boolean value. True or False; true or false, t or f, 0 or 1
Default: True

remove_stopwords
Syntax: remove_stopwords=<bool>
Description: Remove stopwords (i.e. common words like "the" and "I"), currently only supports english.
Usage: Boolean value. True or False; true or false, t or f, 0 or 1
Default: True

base_word
Syntax: base_word=<bool>
Description: Turns on lemmatization or stemming, dependant on the value of base_type.
Usage: Boolean value. True or False; true or false, t or f, 0 or 1
Default: True

base_type
Syntax: base_type=<string>
Description: Sets the value for the type of word base to use, dependant on base_word being set to True. Lemmatization without POS tagging (option lemma) assumes every word is a noun but results in a comprable but faster output. Lemmatization with POS tagging (lemma_pos) is slower but more precice, also adds a new field of pos_tag. Porter Stemmer is used when the option is set to stem.
Usage: Possible values are lemma, lemma_pos, stem
Default: True

mv
Syntax: mv=<bool>
Description: Returns the output as a multi-value field (ready for use with stats count), otherwise returns as a space seperated string.
Usage: Boolean value. True or False; true or false, t or f, 0 or 1
Default: True

pos_tagset
Syntax: pos_tagset=<string>
Description: Sets the option for the tagset used--Advanced Perceptron tagger (None) or universal.
Usage: None or universal
Default: None

term_min_len
Syntax: term_min_len=<int>
Description: Only terms greater than or equal to this number will be returned.
Usage: Interger value of minimum length of terms to return
Default: 0

ngram_range
Syntax: ngram_range=<int>-<int>
Description: Returns new ngram column with range of ngrams specified if max is greater than 1.
Usage: Generally values like 1-2 (same as 2-2), 2-3, 2-4 are used, ngrams above 4 may not provide much value
Default: 1-1

ngram_mix
Syntax: mv=<bool>
Description: Determines if ngram output is combined or separate columns. Defaults to false which results in separate columns
Usage: Boolean value. True or False; true or false, t or f, 0 or 1
Default: False

vader

Description

Sentiment analysis using Valence Aware Dictionary and sEntiment Reasoner. Using option full_output will return scores for neutral, positive, and negative which are the scores that make up the compound score (that is just returned as the field "sentiment". Best to feed in uncleaned data as it takes into account capitalization and punctuation.

Syntax

  • | vader textfield=sentence [full_output=<bool>]
Required Arguments

textfield
Syntax: textfield=<field>
Description: The search field that contains the text that is the target of the analysis.
Usage: Option only takes a single field

Optional Arguments

full_output
Syntax: full_output=<bool>
Description: Return scores for neutral, positive, and negative which are the scores that make up the compound score.
Usage: Boolean value. True or False; true or false, t or f, 0 or 1
Default: False

ML Algorithms

TruncantedSVD

Description

From sklearn. Used for dimension reduction (especially on a TFIDF). This is also known in text analytics as Latent Semantic Analysis or LSA. Returns fields prepended with "SVD_". See http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html

Syntax

fit TruncatedSVD <fields> [into <model name="">] k=<int>
The k option sets the number of components to change the data into. It is important that the value is less than the number of features or documents. The documentation on the algorithm recommends to be set to at least 100 for LSA.

LatentDirichletAllocation

Description

From sklearn. Used for dimension reduction. This is also known as LDA. Returns fields prepended with "LDA_". See http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.LatentDirichletAllocation.html

Syntax

fit LatentDirichletAllocation <fields> [into <model name="">] k=<int>
The k option sets the number of components (topics) to change the data into. It is important that the value is less than the number of features or documents.

NMF

Description

From sklearn. Used for dimension reduction. This is also known as Non-Negative Matrix Factorization. Returns fields prepended with "NMF_". See http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.NMF.html

Syntax

fit NMF <fields> [into <model name="">] [k=<int>]
The k option sets the number of components (topics) to change the data into. It is important that the value is less than the number of features or documents.

TFBinary

Description

A modified implemenation of TfidfVectorizer from sklearn. The current MLTK version has TfidfVectorizer but it does not allow the option of turning off IDF or setting binary to True. This is to create a document-term matrix of whether the document has the given term or not. See http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html

Syntax

fit TFBinary <fields> [into <model name="">] [max_features=<int>] [max_df=<int>] [min_df=<int>] [ngram_range=<int>-<int>] [analyzer=<str>] [norm=<str>] [token_pattern=<str>] [stop_words=english] [use_idf=<true|false>] [binary=<true|false>]
In this implementation, the following settings are already set in order to create a binary output: use_idf is set to False, binary has been set to True, and norm has been set to None. The rest of the settings and options are exactly like the MLTK implementation.

MinMaxScaler

Description

From sklearn. Transforms each feature to a given range. Returns fields prepended with "MMS_". See http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html

Syntax

fit MinMaxScaler <fields> [into <model name="">] [copy=<true|false>] [feature_range=<int>-<int>]
Default feature_range=0-1 copy=true.

LinearSVC

Description

From sklearn. Similar to SVC with parameter kernel=’linear’, but implemented in terms of liblinear rather than libsvm, so it has more flexibility in the choice of penalties and loss functions and should scale better to large numbers of samples. See http://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html

Syntax

fit LinearSVC <fields> [into <model name="">] [gamma=<int>] [C=<int>] [tol=<int>] [intercept_scaling=<int>] [random_state=<int>] [max_iter=<int>] [penalty=<l1|l2>] [loss=<hinge|squared_hinge>] [multi_class=<ovr|crammer_singer>] [dual=<true|false>] [fit_intercept=<true|false>]
The C option sets the penalty parameter of the error term.

ExtraTreesClassifier

Description

From sklearn. This class implements a meta estimator that fits a number of randomized decision trees (a.k.a. extra-trees) on various sub-samples of the dataset and use averaging to improve the predictive accuracy and control over-fitting. See http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.ExtraTreesClassifier.html

Syntax

fit ExtraTreesClassifier <fields> [into <model name="">] [random_state=<int>] [n_estimators=<int>] [max_depth=<int>] [max_leaf_nodes=<int>] [max_features=<int|auto|sqrt|log|none>] [criterion=<gini|entropy>]
The n_estimators option sets the number of trees in the forest, defaults to 10.

Support

Support will be provided through Splunkbase (click on Contact Developer) or Splunk Answers or submit an issue in Github. Expected responses will depend on issue and as time permits, but every attempt will be made to fix within 2 weeks.

Documentation

Documenation for the app and will be kept upto date on Github as well as on the Splunkbase page.

Known Issues

Release Notes

Version 1.0.2
March 19, 2019

Version bump for appinspect

Version 1.0.1
March 12, 2019

Minor fix for file permissions found from appinspect. Update 20newsgroups dataset to not contain and index column.

Version 1.0.0
March 8, 2019

Fix to Counts dashboard when searching for usage of term (thank you dalward!). Fix to cleantext command for consistent output on POS tagging when only one result in the text block. Added named entities to Counts dashboard. Added Themes category, renamed Themes dashboard to Clustering. Added Named Entities dashboard under Themes. Updated visualization app requirements. Added 20newsgroups.csv dataset. Added Classification dashboard. Updated documentation. Updated text cleaning option to require minimum term length of 2.

Version 0.9.5
Nov. 9, 2018

Minor redundant fix to algos.conf. Fix ngram output on text that has cleaned itself empty. Add option to maintain a copy of the original text this makes it faster for the Counts dashboard to search the original text. Updated counts dashboard to use this capability.

Version 0.9.4
Aug. 16, 2018

Added related combined features algorithms--TFBinary, MinMaxScaler, LinearSVC, ExtraTreesClassifier

256
Installs
1,347
Downloads
Share Subscribe LOGIN TO DOWNLOAD

Subscribe Share

AppInspect Tooling

Splunk AppInspect evaluates Splunk apps against a set of Splunk-defined criteria to assess the validity and security of an app package and components.

Are you a developer?

As a Splunkbase app developer, you will have access to all Splunk development resources and receive a 10GB license to build an app that will help solve use cases for customers all over the world. Splunkbase has 1000+ apps and add-ons from Splunk, our partners and our community. Find an app or add-on for most any data source and user need, or simply create your own with help from our developer portal.

Follow Us:
© 2005-2019 Splunk Inc. All rights reserved.
Splunk®, Splunk>®, Listen to Your Data®, The Engine for Machine Data®, Hunk®, Splunk Cloud™, Splunk Light™, SPL™ and Splunk MINT™ are trademarks and registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners.