Ingesting Creston AV device data into Splunk

Assumptions:
• You already have an instance of syslog-ng running on your syslog server.
• You already have a forwarder set up and have tested data ingestion to your Splunk instance.

The below steps describe the process needed to bring Crestron data into Splunk from DMPS/Pro3, Touchpanel & Airmedia device types:

1.  Create SSL security keys & certificates.
2.  Upload certificate to Crestron devices.
3.  Enable Syslog on the Crestron devices.
4.  Configure syslog-ng.conf to handle incoming Syslog.
5.  Configure inputs.conf to ingest the Syslog data.
6.  Set up Syslog log retention.

-------------- Generate SSL security keys & certificates in order to decrypt Crestron’s Syslog data. ------------

Generate a private key for your root certificate:

$SPLUNK_HOME/bin/splunk cmd openssl genrsa -aes256 -out myCAPrivateKey.key 2048

Generate and sign the certificate:

$SPLUNK_HOME/bin/splunk cmd openssl req -new -key myCAPrivateKey.key -out myCACertificate.csr

Generate a key for your server certificate:

$SPLUNK_HOME/bin/splunk cmd openssl genrsa -aes256 -out myServerPrivateKey.key 204

Generate and sign a new server certificate:

$SPLUNK_HOME/bin/splunk cmd openssl req -new -key myServerPrivateKey.key -out myServerCertificate.csr
$SPLUNK_HOME/bin/splunk cmd openssl x509 -req -in myServerCertificate.csr -SHA256 -CA myCACertificate.pem                 -CAkey myCAPrivateKey.key -CAcreateserial -out myServerCertificate.pem -days 1095

This creates:

myServerCertificate.pem
myServerPrivateKey.key
myCACertificate.pem

Turn off password protection on the private key (if using syslog-ng v3.7 or below):

Run this command using OpenSSL:

openssl rsa -in myServerPrivateKey.key -out CrestronPrivateKeyNoPass.key

The output file CrestronPrivateKeyNoPass.key is now unencrypted. To verify this, open the file using a vi.

Encrypted headers look like this:

-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED DEK-Info: DES-EDE3-CBC,

6AC307785DD187EF...
-----END RSA PRIVATE KEY-----

Unencrypted headers look like this:

-----BEGIN RSA PRIVATE KEY-----
6AC307785DD187EF...
-----END RSA PRIVATE KEY-----

------------------------------------ Upload your newly created cert to Creston. ------------------------------

Connect to your Crestron device.

Use an FTP program, such as FileZilla or WinSCP to upload the myCACertificate.pem cert to the a /SYS/trusted directory on the Crestron device.

------------------------------------ Enable remote Syslog on the Crestron device. ------------------------------

Either SSH to the device using Putty or open the text console via the Crestron Toolbox.

Enter the below string of commands to enable Syslog forwarding;

syslog extrasyslogon
remotesyslog –S:ON –E: OK –A –I:[syslog server IP or hostname] –P:[syslog port] –T:SSL
reboot

Note the reboot command will cause the device to go offline.

------------------------------ Configure syslog-NG to write the incoming Syslog to disk. ------------------------

Make a backup of the syslog-ng.conf before you make any edits.

vi /etc/syslog-ng/syslog-ng.conf

Configure the source to listen on port tcp port 514. (It is also possible to configure Syslog to be sent on a custom port from Crestron).

source s_crestron {
    network{
        ip(0.0.0.0) port(514)
        transport("tls")
        tls(
            key-file("/opt/splunk/etc/keys/CrestronPrivateKeyNoPass.key")
            cert-file((“/opt/splunk/etc/keys/myCACertificate.pem")
            peer-verify(optional-untrusted)
        )
    );
};

Configure filters for the different types of Crestron devices:

filter f_crestron_dmps { match(".*DMPS3" value("MESSAGE")); or match("PRO3" value("MESSAGE")); };
filter f_crestron_tp { match(".*TP.localdomain" value ("MESSAGE")); };
filter f_crestron_am { match(".*-SHARE.localdomain" value ("MESSAGE ")); };

Configure a catch-all filter for anything that does not match the filters configured above:

filter f_all { not (
filter(f_crestron_dmps) or
filter(f_crestron_tp)   or
filter(f_crestron_am)
);
};

Configure destination location where syslog-ng will write the log data (include the catch-all destination):

destination d_crestron_dmps { file("/data/syslog/crestron/dmps/$HOST/$YEAR-$MONTH-$DAY-crestron.log"     create_dirs(yes)); };
log { source(s_crestron); filter(f_crestron_dmps); destination(d_crestron_dmps); };

destination d_crestron_tp { file("/opt/splunk/syslog/crestron/touchpanel/$HOST/$YEAR-$MONTH-$DAY-crestron.log"     create_dirs(yes)); };
log { source(s_crestron); filter(f_crestron_tp); destination(d_crestron_tp); };

destination d_crestron_am { file("/opt/splunk/syslog/crestron/airmedia/$HOST/$YEAR-$MONTH-$DAY-crestron.log" create_dirs(yes)); };
log { source(s_crestron); filter(f_crestron_am); destination(d_crestron_am); };

destination d_catch-all_tcp { file("/opt/splunk/syslog/crestron/catch-all /$HOST/$YEAR-$MONTH-$DAY-catch_all.log"  create_dirs(yes)); };
log { source(s_crestron); filter(f_all); destination(d_catch-all_tcp); };

Restart the syslog-ng service:

sudo systemctl restart syslog-ng.service

------------------------------ Set up the Splunk inputs to index the new Syslog log files. ------------------------

Next step is to set up the forwarder inputs. (Don’t forget to create a crestron index on your indexers before you ingest).

vi /opt/splunkforwarder/etc/apps/example_crestron_inputs/local/inputs.conf

[monitor:///opt/splunk/syslog/crestron/dmps/*/*.log]
sourcetype = crestron:dmps:syslog
index = crestron
disabled = false
host_segment = 6

[monitor:///opt/splunk/syslog/crestron/touchpanel/*/*.log]
sourcetype = crestron:tp:syslog
index = crestron
disabled = false
host_segment = 6

[monitor:///opt/splunk/syslog/crestron/airmedia/*/*.log]
sourcetype = crestron:am:syslog
index = crestron
disabled = false
host_segment = 6

Once you restart the forwarder you should see Crestron data in Splunk.
Please also reference this previous blog which covers using syslog-ng with Splunk:

------------------------------------ Set up Syslog log retention. ------------------------------------

To prevent syslog-ng from filling up the filesystem, implement a cronjob to purge the logs. The below example runs every morning at 2am and removes logs that are older than 3 days.

# crontab –e
0 2 * * * /bin/find /opt/splunk/syslog/ -type f -name \*.log -mtime +3 -exec rm {} \;

Technology Add-on for Crestron

0