icon/x Created with Sketch.

Splunk Cookie Policy

We use our own and third-party cookies to provide you with a great online experience. We also use these cookies to improve our products and services, support our marketing campaigns, and advertise to you on our website and other websites. Some cookies may continue to collect information after you have left our website. Learn more (including how to update your settings) here.
Accept Cookie Policy

We are working on something new...

A Fresh New Splunkbase
We are designing a New Splunkbase to improve search and discoverability of apps. Check out our new and improved features like Categories and Collections. New Splunkbase is currently in preview mode, as it is under active development. We welcome you to navigate New Splunkbase and give us feedback.
Log4Shell Vulnerability: Information and guidance for you. Get resources.

Accept License Agreements

This app is provided by a third party and your right to use the app is in accordance with the license provided by that third-party licensor. Splunk is not responsible for any third-party apps and does not provide any warranty or support. If you have any questions, complaints or claims with respect to this app, please contact the licensor directly.

Thank You

Downloading Rsyslog Insight (Statistics and Debugging)
SHA256 checksum (rsyslog-insight-statistics-and-debugging_104.tgz) d3f4e38f3a89cc5c793ae5ebedcce081887d99a0732e2acea4acd386b02f8510 SHA256 checksum (rsyslog-insight-statistics-and-debugging_103.tgz) 653afa404a770c8a2588aec46828077fdc43f47432953f62263e008ca6ac57f2 SHA256 checksum (rsyslog-insight-statistics-and-debugging_101.tgz) 009b625b4fa1add09f2531bc4643cd0709023df3648963516bee760824a46f5e
To install your download
For instructions specific to your download, click the Details tab after closing this window.

Flag As Inappropriate

splunk

Rsyslog Insight (Statistics and Debugging)

Splunk Cloud
Overview
Details
Rsyslog Insight allows rapid troubleshooting and operational visibility of Rsyslog. It visualizes Rsyslog statistics (generated by impstats module) and Rsyslog debug logs.

Currently there are 27 different charts and tables grouped in 2 views:

Rsyslog Statistics:

Imjournal: Anomalies
Imjournal: Operation
Imjournal: Disk Usage
Main Queue: Full
Main Queue: Size vs Max Size
Main Queue: Operation
Resource Usage: Operation
Resource Usage: MaxRss (Maximum Resident Set Size)
Resource Usage: Inblock
Resource Usage: Oublock
Resource Usage: user time vs system time
Action Queues: Processed
Action Queues: Failed
Action Queues: Suspended vs Resumed
Action Queues: Suspended Duration
Forwarding Output Module
Unix Socket Input Module: Submitted
Unix Socket Input Module: Rate Limiting

Rsyslog Debug:

Forwarding Output Module: Resume Operations
Forwarding Output Module: Resume Operations (Summary Table)
Actions: Suspend Operations
Actions: Ready Again Operations
Forwarding Output Module - TCPSendInit: CREATE
Forwarding Output Module - TCPSendInit: FAILED
Forwarding Output Module - TCPSendInit: FAILED (Summary Table)
Discarded (lost) Messages
Cannot Connect Events

For a similar App for Syslog-NG visit https://splunkbase.splunk.com/app/6422/?Syslog-NG%20Insight

Rsyslog Insight allows rapid troubleshooting and operational visibility of Rsyslog. It visualizes Rsyslog statistics (generated by impstats module) and Rsyslog debug logs.

Overview

Inputs

InputDescriptionSourcetype
Rsyslog StatisticsStatistics generated by impstats modulersyslog:impstats
Rsyslog Debug OutputDebug output (running continiously or on demand)rsyslog:debug

Currently there are 27 different charts and tables grouped in 2 views:

Rsyslog Statistics
    Imjournal: Anomalies
    Imjournal: Operation
    Imjournal: Disk Usage
    Main Queue: Full
    Main Queue: Size vs Max Size
    Main Queue: Operation
    Resource Usage: Operation
    Resource Usage: MaxRss (Maximum Resident Set Size)
    Resource Usage: Inblock
    Resource Usage: Oublock
    Resource Usage: User Time vs System Time
    Action Queues: Processed
    Action Queues: Failed
    Action Queues: Suspended vs Resumed
    Action Queues: Suspended Duration
    Forwarding Output Module
    Unix Socket Input Module: Submitted
    Unix Socket Input Module: Rate Limiting

Rsyslog Debug
    Forwarding Output Module: Resume Operations
    Forwarding Output Module: Resume Operations (Summary Table)
    Actions: Suspend Operations
    Actions: Ready Again Operations
    Forwarding Output Module - TCPSendInit: CREATE
    Forwarding Output Module - TCPSendInit: FAILED
    Forwarding Output Module - TCPSendInit: FAILED (Summary Table)
    Discarded (lost) Messages
    Cannot Connect Events

Debug Log Input

Sourcetype rsyslog:debug - Rsyslog debug log.

Continiously debugging

To run rsyslog continiously in debug mode:

RSYSLOG_DEBUG="NoStdOut" RSYSLOG_DEBUGLOG="/var/log/rsyslog_debug.log" rsyslogd -d 2>/dev/null

Alternatively add following lines in /etc/sysconfig/rsyslog:

RSYSLOG_DEBUG="NoStdOut"
RSYSLOG_DEBUGLOG="/var/log/rsyslog_ondemand.log"

On-demand debugging

To configure on-demand debugging add following lines in /etc/sysconfig/rsyslog:

RSYSLOG_DEBUG="DebugOnDemand NoStdOut"
RSYSLOG_DEBUGLOG=/var/log/ondemand.log

To enable/disable on-demand debugging run this command:

kill -USR1 $(cat /var/run/rsyslogd.pid)

Additional information about Rsyslog Debugging Configuration

Rsyslog debug log is very verbose and can grow very fast, a cron job must be created to delete/truncate a log. The Rsyslog Insight App filters a debug log so that only a small portion of debug log reaches Splunk.

An example of debug log

3003.397218336:main Q:Reg/w0  : ../action.c: actionDoRetry: action-7-builtin:omfwd action->tryResume returned -2007
3003.397227410:main Q:Reg/w0  : ../action.c: action[action-7-builtin:omfwd] transitioned to state: susp
3003.397231586:main Q:Reg/w0  : ../action.c: action 'action-7-builtin:omfwd' suspended, earliest retry=1650983063 (now 1650983003), iNbrResRtry 18, duration 60
3018.029076077:main Q:Reg/w0  : ruleset.c: processBATCH: next msg 193: cannot connect to 192.168.233.34:514: Connection timed out [v8.2102.0-8.el8 try https://www.rsyslog.com/e/2027 ]
3018.071678903:main Q:Reg/w0  : omfile.c: omfile: write to stream, pData->pStrm 0x7fb364005e50, lenBuf 153, strt data Apr 26 16:23:23 sender rsyslogd[37152]: cannot connect to 192.168.233.34:514: Connection timed out [v8.2102.0-8.el8 try https://
3475.230450786:main Q:Reg/w0  : ../action.c: actionDoRetry: action-7-builtin:omfwd enter loop, iRetries=0, ResumeInRow 2
3475.230460720:main Q:Reg/w0  : omfwd.c: TCPSendInit CREATE
3605.506446144:main Q:Reg/w0  : operatingstate.c: osf: MSG cannot connect to 192.168.233.34:514: signaling new internal message via SIGTTOU: 'cannot connect to 192.168.233.34:514: Connection timed out
3605.506480797:main Q:Reg/w0  : omfwd.c: TCPSendInit FAILED with -2027.
3605.506531985:main Q:Reg/w0  : omfwd.c: omfwd: doTryResume 192.168.233.34 iRet -2027
3605.506536756:main Q:Reg/w0  : ../action.c: actionDoRetry: action-7-builtin:omfwd action->tryResume returned -2007
3605.506542611:main Q:Reg/w0  : ../action.c: action[action-7-builtin:omfwd] transitioned to state: susp
3605.506545162:main Q:Reg/w0  : ../action.c: action 'action-7-builtin:omfwd' suspended, earliest retry=1650983665 (now 1650983605), iNbrResRtry 19, duration 60
4076.078541962:main Q:Reg/w0  : ../action.c: actionDoRetry: action-7-builtin:omfwd enter loop, iRetries=0, ResumeInRow 2

Workaround for 4-digits timestamp

The debug log has own four digits timestamp format with nanoseconds (calculated as "%s % 10000", i.e. four last digits of uniq epoch timestamp) that not easy to fix at index time - therefore it can ingested as is, Splunk will fallback to current time and complain with `WARN DateParserVerbose [23819 merging] - The TIME_FORMAT specified is matching timestamps (Thu Jan 1 01:44:44 1970) outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE.". Sadly the highest legal value for MAX_DAYS_AGO is 10951 (30 years) that is not high enough to fix the timestamp (Send me an email if you know a fix!). As a workaround the _time is fixed at search time by taking first 6 digits of the timestamp from _indextime.

Working with ad-hoc debug logs

If you have a debug log created earlier or from other machine, you can still visualize it with splunk, but the timestamp must be fixed first - in this case a file creation timestamp can be used. Therefore it is important that a debug file retain an original metadata (creation time). A perl script for the timestamp conversion can be found in the "support" folder of the application package.

Statistics Log Input

Sourcetype rsyslog:impstats - Rsyslog Statistics

To enable Rsyslog Statistic put this config right on the top of /etc/rsyslog.conf. Modify interval and log.file parameters as needed:

module(
  load="impstats"
  interval="10"
  resetCounters="on"
  log.file="/var/log/impstats"
  log.syslog="off"
 )

More ressources about Impstats module:
* https://www.rsyslog.com/doc/v8-stable/configuration/modules/impstats.html
* https://www.rsyslog.com/how-to-use-impstats/
* https://www.youtube.com/watch?v=k7YxhxoPcBc

An example of impstats log:

Tue Apr 26 10:18:04 2022: global: origin=dynstats
Tue Apr 26 10:18:04 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0 ratelimit.numratelimiters=0
Tue Apr 26 10:18:04 2022: action-0-builtin:omfile: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Apr 26 10:18:04 2022: action-1-builtin:omfile: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Apr 26 10:18:04 2022: action-2-builtin:omfile: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Apr 26 10:18:04 2022: action-3-builtin:omfile: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Apr 26 10:18:04 2022: action-4-builtin:omusrmsg: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Apr 26 10:18:04 2022: action-5-builtin:omfile: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Apr 26 10:18:04 2022: action-6-builtin:omfile: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Tue Apr 26 10:18:04 2022: resource-usage: origin=impstats utime=386167 stime=689640 maxrss=8300 minflt=790 majflt=11 inblock=1984 oublock=14256 nvcsw=10152 nivcsw=2479 openfiles=10
Tue Apr 26 10:18:04 2022: imjournal: origin=imjournal submitted=0 read=0 discarded=0 failed=0 poll_failed=0 rotations=0 recovery_attempts=0 ratelimit_discarded_in_interval=0 disk_usage_bytes=8388608
Tue Apr 26 10:18:04 2022: main Q: origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=310

To see meanful names for action instead of action-5-xxxx use a "name" parameter when defining queues, for example:

action(type="omfwd" name="ForwardingToSIEM" Target="192.168.2.11" Port="10514" Protocol="tcp" Device="eth0")

Resource Usage Parameters

ParameterDescription
utimeuser CPU time used
stimesystem CPU time used
maxrssmaximum resident set size
ixrssintegral shared memory size
idrssintegral unshared data size
isrssintegral unshared stack size
minfltpage reclaims (soft page faults)
majfltpage faults (hard page faults)
nswapswaps
inblockblock input operations
oublockblock output operations
msgsndIPC messages sent
msgrcvIPC messages received
nsignalssignals received
nvcswvoluntary context switches
nivcswinvoluntary context switches

Reference: https://man7.org/linux/man-pages/man2/getrusage.2.html

Input Configuration

An example of local/inputs.conf:

[monitor:///var/log/impstats]
index = test
sourcetype = rsyslog:impstats

[monitor:///var/log/rsyslog_debug.log]
index = test
sourcetype = rsyslog:debug

To allow Splunk to read logs created by Rsyslog:

setfacl -m u:splunk:r /var/log/impstats
setfacl -m u:splunk:r /var/log/rsyslog_debug.log

Use a cron job or logrotate to delete/truncate old logs:

*/5 * * * * truncate -s0 /var/log/impstats
*/5 * * * * truncate -s0 /var/log/rsyslog_debug.log

Release Notes

Version 1.0.4
April 27, 2022

added filtering by host, added a script for timestamp conversion

Version 1.0.3
April 26, 2022

added a lookup files for 311 known error codes (rsyslog-8.2204.0)

Version 1.0.1
April 26, 2022

initial public release


Subscribe Share

Are you a developer?

As a Splunkbase app developer, you will have access to all Splunk development resources and receive a 10GB license to build an app that will help solve use cases for customers all over the world. Splunkbase has 1000+ apps from Splunk, our partners and our community. Find an app for most any data source and user need, or simply create your own with help from our developer portal.

Follow Us:
Splunk, Splunk>,Turn Data Into Doing, Data-to-Everything, and D2E are trademarks or registered trademarks of Splunk Inc. in the United States and other countries. All other brand names,product names,or trademarks belong to their respective owners.