Rsyslog Insight allows rapid troubleshooting and operational visibility of Rsyslog. It visualizes Rsyslog statistics (generated by impstats module) and Rsyslog debug logs.

Overview

Inputs

Currently there are 27 different charts and tables grouped in 2 views:

Rsyslog Statistics
    Imjournal: Anomalies
    Imjournal: Operation
    Imjournal: Disk Usage
    Main Queue: Full
    Main Queue: Size vs Max Size
    Main Queue: Operation
    Resource Usage: Operation
    Resource Usage: MaxRss (Maximum Resident Set Size)
    Resource Usage: Inblock
    Resource Usage: Oublock
    Resource Usage: User Time vs System Time
    Action Queues: Processed
    Action Queues: Failed
    Action Queues: Suspended vs Resumed
    Action Queues: Suspended Duration
    Forwarding Output Module
    Unix Socket Input Module: Submitted
    Unix Socket Input Module: Rate Limiting

Rsyslog Debug
    Forwarding Output Module: Resume Operations
    Forwarding Output Module: Resume Operations (Summary Table)
    Actions: Suspend Operations
    Actions: Ready Again Operations
    Forwarding Output Module - TCPSendInit: CREATE
    Forwarding Output Module - TCPSendInit: FAILED
    Forwarding Output Module - TCPSendInit: FAILED (Summary Table)
    Discarded (lost) Messages
    Cannot Connect Events

Debug Log Input

Sourcetype rsyslog:debug - Rsyslog debug log.

Continiously debugging

To run rsyslog continiously in debug mode:

RSYSLOG_DEBUG="NoStdOut" RSYSLOG_DEBUGLOG="/var/log/rsyslog_debug.log" rsyslogd -d 2>/dev/null

Alternatively add following lines in /etc/sysconfig/rsyslog:

RSYSLOG_DEBUG="NoStdOut"
RSYSLOG_DEBUGLOG="/var/log/rsyslog_ondemand.log"

On-demand debugging

To configure on-demand debugging add following lines in /etc/sysconfig/rsyslog:

RSYSLOG_DEBUG="DebugOnDemand NoStdOut"
RSYSLOG_DEBUGLOG=/var/log/ondemand.log

To enable/disable on-demand debugging run this command:

kill -USR1 $(cat /var/run/rsyslogd.pid)

Additional information about Rsyslog Debugging Configuration

• https://www.rsyslog.com/doc/master/troubleshooting/debug.html
• https://www.rsyslog.com/doc/master/troubleshooting/howtodebug.html

Rsyslog debug log is very verbose and can grow very fast, a cron job must be created to delete/truncate a log. The Rsyslog Insight App filters a debug log so that only a small portion of debug log reaches Splunk.

An example of debug log

3003.397218336:main Q:Reg/w0  : ../action.c: actionDoRetry: action-7-builtin:omfwd action->tryResume returned -2007
3003.397227410:main Q:Reg/w0  : ../action.c: action[action-7-builtin:omfwd] transitioned to state: susp
3003.397231586:main Q:Reg/w0  : ../action.c: action 'action-7-builtin:omfwd' suspended, earliest retry=1650983063 (now 1650983003), iNbrResRtry 18, duration 60
3018.029076077:main Q:Reg/w0  : ruleset.c: processBATCH: next msg 193: cannot connect to 192.168.233.34:514: Connection timed out [v8.2102.0-8.el8 try https://www.rsyslog.com/e/2027 ]
3018.071678903:main Q:Reg/w0  : omfile.c: omfile: write to stream, pData->pStrm 0x7fb364005e50, lenBuf 153, strt data Apr 26 16:23:23 sender rsyslogd[37152]: cannot connect to 192.168.233.34:514: Connection timed out [v8.2102.0-8.el8 try https://
3475.230450786:main Q:Reg/w0  : ../action.c: actionDoRetry: action-7-builtin:omfwd enter loop, iRetries=0, ResumeInRow 2
3475.230460720:main Q:Reg/w0  : omfwd.c: TCPSendInit CREATE
3605.506446144:main Q:Reg/w0  : operatingstate.c: osf: MSG cannot connect to 192.168.233.34:514: signaling new internal message via SIGTTOU: 'cannot connect to 192.168.233.34:514: Connection timed out
3605.506480797:main Q:Reg/w0  : omfwd.c: TCPSendInit FAILED with -2027.
3605.506531985:main Q:Reg/w0  : omfwd.c: omfwd: doTryResume 192.168.233.34 iRet -2027
3605.506536756:main Q:Reg/w0  : ../action.c: actionDoRetry: action-7-builtin:omfwd action->tryResume returned -2007
3605.506542611:main Q:Reg/w0  : ../action.c: action[action-7-builtin:omfwd] transitioned to state: susp
3605.506545162:main Q:Reg/w0  : ../action.c: action 'action-7-builtin:omfwd' suspended, earliest retry=1650983665 (now 1650983605), iNbrResRtry 19, duration 60
4076.078541962:main Q:Reg/w0  : ../action.c: actionDoRetry: action-7-builtin:omfwd enter loop, iRetries=0, ResumeInRow 2

Workaround for 4-digits timestamp

The debug log has own four digits timestamp format with nanoseconds (calculated as "%s % 10000", i.e. four last digits of uniq epoch timestamp) that not easy to fix at index time - therefore it can ingested as is, Splunk will fallback to current time and complain with `WARN DateParserVerbose [23819 merging] - The TIME_FORMAT specified is matching timestamps (Thu Jan 1 01:44:44 1970) outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE.". Sadly the highest legal value for MAX_DAYS_AGO is 10951 (30 years) that is not high enough to fix the timestamp (Send me an email if you know a fix!). As a workaround the _time is fixed at search time by taking first 6 digits of the timestamp from _indextime.

Working with ad-hoc debug logs

If you have a debug log created earlier or from other machine, you can still visualize it with splunk, but the timestamp must be fixed first - in this case a file creation timestamp can be used. Therefore it is important that a debug file retain an original metadata (creation time). A perl script for the timestamp conversion can be found in the "support" folder of the application package.

Statistics Log Input

Sourcetype rsyslog:impstats - Rsyslog Statistics

To enable Rsyslog Statistic put this config right on the top of /etc/rsyslog.conf. Modify interval and log.file parameters as needed:

module(
  load="impstats"
  interval="10"
  resetCounters="on"
  log.file="/var/log/impstats"
  log.syslog="off"
 )

More ressources about Impstats module:
* https://www.rsyslog.com/doc/v8-stable/configuration/modules/impstats.html
* https://www.rsyslog.com/how-to-use-impstats/
* https://www.youtube.com/watch?v=k7YxhxoPcBc

An example of impstats log:

Tue Apr 26 10:18:04 2022: global: origin=dynstats
Tue Apr 26 10:18:04 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0 ratelimit.numratelimiters=0
Tue Apr 26 10:18:04 2022: action-0-builtin:omfile: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Apr 26 10:18:04 2022: action-1-builtin:omfile: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Apr 26 10:18:04 2022: action-2-builtin:omfile: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Apr 26 10:18:04 2022: action-3-builtin:omfile: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Apr 26 10:18:04 2022: action-4-builtin:omusrmsg: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Apr 26 10:18:04 2022: action-5-builtin:omfile: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Apr 26 10:18:04 2022: action-6-builtin:omfile: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Apr 26 10:18:04 2022: resource-usage: origin=impstats utime=386167 stime=689640 maxrss=8300 minflt=790 majflt=11 inblock=1984 oublock=14256 nvcsw=10152 nivcsw=2479 openfiles=10
Tue Apr 26 10:18:04 2022: imjournal: origin=imjournal submitted=0 read=0 discarded=0 failed=0 poll_failed=0 rotations=0 recovery_attempts=0 ratelimit_discarded_in_interval=0 disk_usage_bytes=8388608
Tue Apr 26 10:18:04 2022: main Q: origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=310

To see meanful names for action instead of action-5-xxxx use a "name" parameter when defining queues, for example:

action(type="omfwd" name="ForwardingToSIEM" Target="192.168.2.11" Port="10514" Protocol="tcp" Device="eth0")

Resource Usage Parameters

Reference: https://man7.org/linux/man-pages/man2/getrusage.2.html

Input Configuration

An example of local/inputs.conf:

[monitor:///var/log/impstats]
index = test
sourcetype = rsyslog:impstats

[monitor:///var/log/rsyslog_debug.log]
index = test
sourcetype = rsyslog:debug

To allow Splunk to read logs created by Rsyslog:

setfacl -m u:splunk:r /var/log/impstats
setfacl -m u:splunk:r /var/log/rsyslog_debug.log

Use a cron job or logrotate to delete/truncate old logs:

*/5 * * * * truncate -s0 /var/log/impstats
*/5 * * * * truncate -s0 /var/log/rsyslog_debug.log