HL7 Add-On for Splunk
Overview
Details
To Be Used with HL7 Dashboard Examples App for Splunk
The value from this Add-on is derived by it's ability to:
1. Parse every HL7 message from log entry.*
2. Extract every segment within any HL7 v2.x message into it's own Splunk Field.**
3. Extract every field within every segment in the message.**
4. Provide examples on how to extract values from HL7 subfields.
i.e. PID-5 contains family_name,given_name,middle_name,suffix,prefix,degree.
Individual fields are created for each of these.
5. Provide examples on how to enrich HL7 coded values with actual contextual descriptions.
i.e. MSH-9 may have ADT^A08.
A new field Message_Type will also be present with a value of "Update Patient Information"
If the logs come from Cloverleaf's Interface engine there are additional extractions for thread_id, client ip, and client port.
NOTE: When migrating to Splunk 7.2 Fields from HL7 Segments will only show up in Verbose search mode because they are extracted using DELIMS and not REGEX. Until a new version is released, if you need fields to show in SmartMode please add | field * to the end of your base search OR change your transforms like this:
[MSH_Fields]
#DELIMS = "|"
#FIELDS = MSH_2,MSH_3,MSH_4,MSH_5,MSH_6,MSH_7,MSH_8,MSH_9,MSH_10,MSH_11,MSH_12,MSH_13,MSH_14,MSH_15,MSH_16,MSH_17,MSH_18,MSH_19,MSH_20,MSH_21
REGEX = (?P[^\|]*?)\|(?P[^\|]*?)\|(?P[^\|]*?)\|(?P[^\|]*?)\|(?P[^\|]*?)\|(?P[^\|]*?)\|(?P[^\|]*?)\|(?P[^\|]*?)\|(?P[^\|]*?)\|(?P[^\|]*?)\|(?P[^\|]*?)\|(?P[^\|]*?)\|(?P[^\|]*?)\|((?P[^\|]*?)\|)*((?P[^\|]*?)\|)*((?P[^\|]*?)\|)*((?P[^\|]*?)\|)*((?P[^\|]*?)\|)*((?P[^\|]*?)\|)*((?P[^\|]*?))*
SOURCE_KEY = MSH
Splunk Add-On for HL7
Author: Joe Welsh - Splunk
Version/Date: 1.13 July 14, 2020
Supported product(s):
This add-on supports logs containing HL7 v2.X data.
Source type(s): This add-on will process data that is source-typed
as "hl7_v2".
Input requirements: N/A
To be used with HL7 Dashboard Examples App for Splunk ( )
Using this Add-On
Configuration
To use this add-on, manually configure the data input with the following input.conf
properties and the add-on will optimize the data automatically.
sourcetype = hl7_v2
These properties will also be utilized by the underlying searches and eventtypes .
The value from this Add-on is derived by it's ability to:
- Parse every HL7 message from log entry.*
- Extract every segment within any HL7 v2.x message into it's own Splunk Field.**
- Extract every field within every segment in the message.**
- Provide examples on how to extract values from HL7 subfields.
i.e. PID-5 contains family_name,given_name,middle_name,suffix,prefix,degree.
Individual fields are created for each of these. - Provide examples on how to enrich HL7 coded values with actual contextual descriptions.
i.e. MSH-9 may have ADT^A08.
A new field Message_Type will also be present with a value of "Update Patient Information"
If the logs come from Cloverleaf's Interface engine there are additional extractions for thread_id, client ip, and client port.
Notes:
*The datetime format can vary for each HL7 message. In lieu of using TIME_FORMAT inside props.conf
a custom datetime.xml was created inside this app. The extracted groups correspond to those specified
in the extracted attribute for define element. The groups (year, month, day, etc.) are specified in
Splunk's default datetime.xml in /etc/system/datetime.xml.
**Repeating Segments will be placed inside multi-value fields.
i.e. If there are four NTE segments in the message there will be one multi-value NTE field with four values.
The subsequent HL7 fields will only be extracted for the first segment in repeating segments.
Sample Files
The reference files used for this Add-on can be found in the cloverleaf_logs directory.
The samples directory contains files used to generate sample messages. If you install Splunk's Eventgen App [https://splunkbase.splunk.com/app/1924/] these samples will be replayed and put in your main index.
You'll also need to modify splunkUser and splunkPass in eventgen.conf to match credentials on your system.
Handling Extraction of Repeating Segments
HL7_Segment_Template transformation uses MV_ADD = true to aggregate repeating segments. However, you ultimately need to construct a mv field using the makemv SPL command to split them out for use. However, I was unable to split it using the newline character as a delimiter, needed to add an anchor to the end of each line first.
So, take this sample message with repeating DG1 segments:
MSH|^~\&|EPIC|MGH||MGH|201503241909|OHEDSCRIBE|ADT^A08|725467|T|2.3|||||||||
EVN|A08|20150324190937||REG_UPDATE|^EMERGENCY^SCRIBE^OH^^^^^OHSA^^^^^MGH||
PID|1||3100068701^^^EPI^MR||MGHTEST^EDSCRIBE||19600324|M|||^^^^^US^P|||||||6100215419|999-99-9999|||||||||||N||
PD1|||MARION GENERAL HOSPITAL^^1016100|||||||||||||||
PV1|1|E|ED^ED1^01^MGH^R^^^^^^DEPID||||15802^BOEHMER^DAVID^C^^^^^STARPROV^^^^STARPROV|||Emergency||||||||EMERGENCY|6100215419|SELF||||||||||||||||||||||^^^MGH^^^^^^^||20150324190300||||||27150830003
PV2||GENERAL||||||||||||||||||||N|||||||||||||||||||||||||||
ZPV|||||||||||||20150324190832||||||||
AL1|1|Drug Class|45202^NO KNOWN ALLERGIES^||||||
DG1|1||52^Headache^HRV|Headache||
DG1|2||53^Neck^NCK|Neck||
DG1|3||54^Ear^EAR|Outer Ear||
GT1|1|100046662|MGHTEST^EDSCRIBE^^||^^^^^US|||19600324|M|P/F|SLF|999-99-9999|||||||||||||||||||||||||||||||||||||
ZMP|1|||||N||||^^^^|||^^^^|||||^^^^||||^^^^||Not eligible for Medicare||||^^^^|||||^^^^|0|||^^^^|||||||||||||||||||||||||||||
T_50005_1T_50005_1{CONNID 0} {IPVERSION 4} {CLIENTIP 172.17.154.197} {CLIENTPORT 63143}
HL7_Segment_Template transform, with MV_ADD = true, collected ALL the DG1 segments and put the aggregated values as the single value for DG1:
1||52^Headache^HRV|Headache||
2||53^Neck^NCK|Neck||
3||54^Ear^EAR|Outer Ear||
Then I ran this SPL: sourcetype="hl7_v2" DG1=*| rex mode=sed field=DG1 "s/$/:/g"
which tacks on a ":" delimiter to end of the line and produces:
1||52^Headache^HRV|Headache||:
2||53^Neck^NCK|Neck||:
3||54^Ear^EAR|Outer Ear||:
This SPL then combines the three segments into single line, using end of line as anchor:
sourcetype="hl7_v2" DG1=* | rex mode=sed field=DG1 "s/$/:/g" | makemv delim="$" DG1
1||52^Headache^HRV|Headache||:2||53^Neck^NCK|Neck||:3||54^Ear^EAR|Outer Ear||:
Then I employ the split function multiple times within an eval to split DG1, and use mvindex to extract the particular iterated DG1 segment:
sourcetype="hl7_v2" DG1=* | rex mode=sed field=DG1 "s/$/:/g" | makemv delim="$" DG1| eval s1=mvindex(split('DG1', ":"),0), s2=mvindex(split('DG1', ":"),1), s3=mvindex(split('DG1', ":"),2)
Writing Searches
If you're not seeing extracted field in your searches try adding "| fields *" to the end of your search.
Release Notes
Version 1.13
-Now supports Splunk 8.x
-Will be Splunk Cloud Certified
-Removed pre-built panels and dashboards. Will create a new, HL7 Analytics App to be used with Add-on
Version 1.12
Version 1.11
-Changed permissions required for Splunk Cloud Vetting
find . -type d -exec chmod 0755 {} \;
find . -type f -exec chmod 0644 {} \;
Version 1.10
-Updated macros to work properly in Splunk 7.x
-Updated eventgen.conf to ensure it works with latest Eventgen in Splunkbase: https://splunkbase.splunk.com/app/1924/
-Ensured all three pre-built panels work as a result of all changes
Version 1.09
-Updated Message Type Lookup ( hl7_v2_message_types.csv). Please use Message_Type_Description in lieu of previous Message_Type in your searches.
Version 1.08
- Updated files in samples to replay messages with missing patient names and identifiers.
- Put SEDCMD back to replace carriage returns with line feeds.
Version 1.07
- Removed indexes.conf
- Added instructions to README.md for extracting individual segments when they are repeating.
- Modified files in samples for eventgen to populate the default/main index instead of previously used hl7 index.
Version 1.06
- Modified LINE_BREAKER in props.conf to properly parse HL7 messages that contain "MSH" in the actual message, not just the three first characters.
- Commented out SEDCMD in props.conf for replacing carriage return with line feed.
- Added new sample file in cloverleaf_logs
Version 1.05
=== Splunk TA HL7 (Add-On) ===
Author: Joe Welsh - Splunk
Version/Date: 1.05 August 25, 2016
*Note: This app was previously Add-on for HL7
Supported product(s):
This add-on supports logs containing HL7 v2.X data.
Source type(s): This add-on will process data that is source-typed
as "hl7_v2".
Input requirements: N/A