Monitoring Log Files the easy and efficient way - Avoid unnecessary, time consuming, frustrating log monitoring configurations

Advanced Tailing - Scan the Content of Log Files Via Time Frames

Full Syntax:

[root@nagios-primary ~]# logrobot autofig /var/log/syslog 24h '.' 'nagios-primary abrtd:' 1 5 -show

Lazy Syntax:

[root@nagios-primary ~]# logrobot /var/log/syslog 1m

Tailing of Log Files Using Time Frames (simply specify the log file and the time frame) (Log Monitoring Scenario 1B):

Instead of tailing random lines of a log file, why not tail the log based on time frames?

For instance, show me all entries that were logged within the last 1 minute (this can be changed to hours, days, weeks):

[root@nagios-primary ~]# ./logxray /var/log/mail.log 1m

Jan 4 13:09:01 nagios-primary CRON[14456]: (root) CMD ( [ -x /usr/lib/php5/maxlifetime ] && [ -x /usr/lib/php5/sessionclean ] && [ -d /var/lib/php5 ] && /usr/lib/php5/sessionclean /var/lib/php5 $(/usr/lib/php5/maxlifetime))
2 nagios-primary postfix/local[25918]: C93C7302554: to=<root@nagios-primary.localdomain>, orig_to=<root>, relay=local, delay=0.02, delays=0.01/0/0/0, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Jan 4 13:09:02 nagios-primary postfix/cleanup[26902]: CB748302555: message-id=<20150104210902.CB748302555@nagios-primary.localdomain>
Jan 4 13:09:02 nagios-primary postfix/bounce[25922]: C93C7302554: sender non-delivery notification: CB748302555
Jan 4 13:09:02 nagios-primary postfix/qmgr[1995]: C93C7302554: removed
Jan 4 13:09:02 nagios-primary postfix/qmgr[1995]: CB748302555: from=<>, size=3383, nrcpt=1 (queue active)
Jan 4 13:09:02 nagios-primary postfix/local[23072]: CB748302555: to=<root@nagios-primary.localdomain>, relay=local, delay=0.04, delays=0.04/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Jan 4 13:09:02 nagios-primary postfix/qmgr[1995]: CB748302555: removed
Jan 4 13:09:02 nagios-primary postfix/pickup[5920]: EC2B5302554: uid=0 from=<root>
Jan 4 13:09:02 nagios-primary postfix/cleanup[863]: EC2B5302554: message-id=<20150104210902.EC2B5302554@nagios-primary.localdomain>
Jan 4 13:09:02 nagios-primary postfix/qmgr[1995]: EC2B5302554: from=<root@nagios-primary.localdomain>, size=1322, nrcpt=1 (queue active)
Jan 4 13:09:02 nagios-primary postfix/local[25918]: EC2B5302554: to=<root@nagios-primary.localdomain>, orig_to=<root>, relay=local, delay=0.03, delays=0.02/0/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Jan 4 13:09:02 nagios-primary postfix/cleanup[26902]: EEBFB302555: message-id=<20150104210902.EEBFB302555@nagios-primary.localdomain>
Jan 4 13:09:02 nagios-primary postfix/bounce[25922]: EC2B5302554: sender non-delivery notification: EEBFB302555
Jan 4 13:09:02 nagios-primary postfix/qmgr[1995]: EEBFB302555: from=<>, size=3381, nrcpt=1 (queue active)
Jan 4 13:09:02 nagios-primary postfix/qmgr[1995]: EC2B5302554: removed
Jan 4 13:09:02 nagios-primary postfix/local[10749]: EEBFB302555: to=<root@nagios-primary.localdomain>, relay=local, delay=0.01, delays=0/0/0/0, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Jan 4 13:09:02 nagios-primary postfix/qmgr[1995]: EEBFB302555: removed
Jan 4 13:10:01 nagios-primary CRON[16039]: (root) CMD (/home/nagios/DEEP/logxray-surgery localhost /var/tmp/logXray,graphite,127.0.0.1:8125,c autonda /var/log/apache2/graphite-web_access.log 60m '.' '.' 1 2 http_status_codes_c -ndfoundapachen)
Jan 4 13:10:01 nagios-primary CRON[16040]: (root) CMD (/home/nagios/DEEP/logxray-surgery localhost /var/tmp/logXray,graphite,127.0.0.1:8125,c autonda /var/log/apache2/graphite-web_access.log 60m '.' '.' 1 2 http_status_codes_d -ndfoundapachen)
Jan 4 13:10:01 nagios-primary CRON[16041]: (root) CMD (/home/nagios/DEEP/logxray-surgery localhost /var/tmp/logXray,graphite,127.0.0.1:8125 autonda

2---0---20---ATWFILF---(Jan/4)-(13:09)---(Jan/4)-(13:10:03) ZEAGMitU

Show All entries logged in the [ kern.log ] log file within the last 2 HOURS (Log Monitoring Scenario 2A):

root@nagios-primary:~#
root@nagios-primary:~#
root@nagios-primary:~#
root@nagios-primary:~# logrobot autofig /var/log/kern.log 2h '.' '.' 1 2 -show

Sep 20 17:55:06 nagios-primary kernel: [87310.160050] usb 5-1: new full-speed USB device number 26 using uhci_hcd
Sep 20 17:55:06 nagios-primary kernel: [87310.388215] hub 5-1:1.0: USB hub found
Sep 20 17:55:06 nagios-primary kernel: [87310.390118] hub 5-1:1.0: 4 ports detected
Sep 20 17:55:06 nagios-primary kernel: [87310.673128] usb 5-1.2: new low-speed USB device number 27 using uhci_hcd
Sep 20 17:55:06 nagios-primary kernel: [87310.831895] input: Logitech USB Receiver as /devices/pci0000:00/0000:00:1d.0/usb5/5-1/5-1.2/5-1.2:1.0/input/input34
Sep 20 17:55:06 nagios-primary kernel: [87310.832071] logitech 0003:046D:C517.001B: input,hidraw0: USB HID v1.10 Keyboard [Logitech USB Receiver] on usb-0000:00:1d.0-1.2/input0
Sep 20 17:55:06 nagios-primary kernel: [87310.863133] logitech 0003:046D:C517.001C: fixing up Logitech keyboard report descriptor
Sep 20 17:55:06 nagios-primary kernel: [87310.865367] input: Logitech USB Receiver as /devices/pci0000:00/0000:00:1d.0/usb5/5-1/5-1.2/5-1.2:1.1/input/input35
Sep 20 17:55:06 nagios-primary kernel: [87310.865633] logitech 0003:046D:C517.001C: input,hiddev0,hidraw3: USB HID v1.10 Mouse [Logitech USB Receiver] on usb-0000:00:1d.0-1.2/input1
Sep 20 17:55:08 nagios-primary kernel: [87312.249129] usb 5-1.3: new low-speed USB device number 28 using uhci_hcd
Sep 20 17:55:08 nagios-primary kernel: [87312.436287] input: No brand 4 Port KVMSwicther as /devices/pci0000:00/0000:00:1d.0/usb5/5-1/5-1.3/5-1.3:1.0/input/input36
Sep 20 17:55:08 nagios-primary kernel: [87312.436429] generic-usb 0003:10D5:55A4.001D: input,hidraw4: USB HID v1.10 Keyboard [No brand 4 Port KVMSwicther] on usb-0000:00:1d.0-1.3/input0
Sep 20 17:55:08 nagios-primary kernel: [87312.442165] usbhid 5-1.3:1.1: couldn't find an input interrupt endpoint

2---3240---13---(Sep/20)-(16:49)---(Sep/20)-(17:55:08)---ETWNFILF---(Sep/20)-(17:55)---(Sep/20)-(17:55:08)

Scan through the above output and show ONLY lines that contain/include the strings 'USB HID' (Log Monitoring Scenario 2B):

root@nagios-primary:~#
root@nagios-primary:~#
root@nagios-primary:~#
root@nagios-primary:~# logrobot autofig /var/log/kern.log 2h '.' 'USB HID' 1 2 -show

Sep 20 17:55:06 nagios-primary kernel: [87310.832071] logitech 0003:046D:C517.001B: input,hidraw0: USB HID v1.10 Keyboard [Logitech USB Receiver] on usb-0000:00:1d.0-1.2/input0
Sep 20 17:55:06 nagios-primary kernel: [87310.865633] logitech 0003:046D:C517.001C: input,hiddev0,hidraw3: USB HID v1.10 Mouse [Logitech USB Receiver] on usb-0000:00:1d.0-1.2/input1
Sep 20 17:55:08 nagios-primary kernel: [87312.436429] generic-usb 0003:10D5:55A4.001D: input,hidraw4: USB HID v1.10 Keyboard [No brand 4 Port KVMSwicther] on usb-0000:00:1d.0-1.3/input0

2---3420---3---(Sep/20)-(16:52)---(Sep/20)-(17:55:08)---ETWNFILF---(Sep/20)-(17:55)---(Sep/20)-(17:55:08)

Show once again All entries recorded in the [ kern.log ] log file within the last 2 HOURS (Log Monitoring Scenario 3A):

root@nagios-primary:~#
root@nagios-primary:~#
root@nagios-primary:~#
root@nagios-primary:~# logrobot autofig /var/log/kern.log 2h '.' '.' 1 2 -show

Sep 20 17:55:06 nagios-primary kernel: [87310.160050] usb 5-1: new full-speed USB device number 26 using uhci_hcd
Sep 20 17:55:06 nagios-primary kernel: [87310.388215] hub 5-1:1.0: USB hub found
Sep 20 17:55:06 nagios-primary kernel: [87310.390118] hub 5-1:1.0: 4 ports detected
Sep 20 17:55:06 nagios-primary kernel: [87310.673128] usb 5-1.2: new low-speed USB device number 27 using uhci_hcd
Sep 20 17:55:06 nagios-primary kernel: [87310.831895] input: Logitech USB Receiver as /devices/pci0000:00/0000:00:1d.0/usb5/5-1/5-1.2/5-1.2:1.0/input/input34
Sep 20 17:55:06 nagios-primary kernel: [87310.832071] logitech 0003:046D:C517.001B: input,hidraw0: USB HID v1.10 Keyboard [Logitech USB Receiver] on usb-0000:00:1d.0-1.2/input0
Sep 20 17:55:06 nagios-primary kernel: [87310.863133] logitech 0003:046D:C517.001C: fixing up Logitech keyboard report descriptor
Sep 20 17:55:06 nagios-primary kernel: [87310.865367] input: Logitech USB Receiver as /devices/pci0000:00/0000:00:1d.0/usb5/5-1/5-1.2/5-1.2:1.1/input/input35
Sep 20 17:55:06 nagios-primary kernel: [87310.865633] logitech 0003:046D:C517.001C: input,hiddev0,hidraw3: USB HID v1.10 Mouse [Logitech USB Receiver] on usb-0000:00:1d.0-1.2/input1
Sep 20 17:55:08 nagios-primary kernel: [87312.249129] usb 5-1.3: new low-speed USB device number 28 using uhci_hcd
Sep 20 17:55:08 nagios-primary kernel: [87312.436287] input: No brand 4 Port KVMSwicther as /devices/pci0000:00/0000:00:1d.0/usb5/5-1/5-1.3/5-1.3:1.0/input/input36
Sep 20 17:55:08 nagios-primary kernel: [87312.436429] generic-usb 0003:10D5:55A4.001D: input,hidraw4: USB HID v1.10 Keyboard [No brand 4 Port KVMSwicther] on usb-0000:00:1d.0-1.3/input0
Sep 20 17:55:08 nagios-primary kernel: [87312.442165] usbhid 5-1.3:1.1: couldn't find an input interrupt endpoint

2---3960---13---(Sep/20)-(17:01)---(Sep/20)-(17:55:08)---ETWNFILF---(Sep/20)-(17:55)---(Sep/20)-(17:55:08)

From the above output, exclude all lines that contain 'logitech' and show me what is left (Log Monitoring Scenario 3B):

root@nagios-primary:~#
root@nagios-primary:~#
root@nagios-primary:~#
root@nagios-primary:~#
root@nagios-primary:~# logrobot autofig /var/log/kern.log 2h '.' 'Logitech' 1 2 -showexcl

Sep 20 17:55:06 nagios-primary kernel: [87310.160050] usb 5-1: new full-speed USB device number 26 using uhci_hcd
Sep 20 17:55:06 nagios-primary kernel: [87310.388215] hub 5-1:1.0: USB hub found
Sep 20 17:55:06 nagios-primary kernel: [87310.390118] hub 5-1:1.0: 4 ports detected
Sep 20 17:55:06 nagios-primary kernel: [87310.673128] usb 5-1.2: new low-speed USB device number 27 using uhci_hcd
Sep 20 17:55:08 nagios-primary kernel: [87312.249129] usb 5-1.3: new low-speed USB device number 28 using uhci_hcd
Sep 20 17:55:08 nagios-primary kernel: [87312.436287] input: No brand 4 Port KVMSwicther as /devices/pci0000:00/0000:00:1d.0/usb5/5-1/5-1.3/5-1.3:1.0/input/input36
Sep 20 17:55:08 nagios-primary kernel: [87312.436429] generic-usb 0003:10D5:55A4.001D: input,hidraw4: USB HID v1.10 Keyboard [No brand 4 Port KVMSwicther] on usb-0000:00:1d.0-1.3/input0
Sep 20 17:55:08 nagios-primary kernel: [87312.442165] usbhid 5-1.3:1.1: couldn't find an input interrupt endpoint

2---4320---(8)-(13)-(61.5385%)-(8)-(0)-(frq=8,zsc=0,asc=[Sep-20-(17:55)])---(Sep/20)-(17:07)---(Sep/20)-(17:55:08)---ETWNFILF---(Sep/20)-(17:55)---(Sep/20)-(17:55:08)

Search through the [ syslog ] file. Find out which HOUR within the last 8 hours had the most entries logged (Analysis):

nagios@logrobot-04:/var/log$
nagios@logrobot-04:/var/log$
nagios@logrobot-04:/var/log$
nagios@logrobot-04:/var/log$ logrobot autofig /var/log/syslog 8h '.' '.' 1 2 -exceldh

frq=553,zsc=1.52781,asc=[Oct-17-(10)]
frq=531,zsc=1.29027,asc=[Oct-17-(11)]
frq=456,zsc=0.480479,asc=[Oct-17-(12)]
frq=384,zsc=-0.296925,asc=[Oct-17-(09)]
frq=383,zsc=-0.307722,asc=[Oct-17-(07)]
frq=376,zsc=-0.383303,asc=[Oct-17-(06)]
frq=362,zsc=-0.534465,asc=[Oct-17-(08)]
frq=247,zsc=-1.77615,asc=[Oct-17-(05)]

Search the [ syslog ] file once again. This time, find which MINUTE(S) within the last 1 Hour had the most entries logged (Analysis):

nagios@logrobot-04:/var/log$
nagios@logrobot-04:/var/log$
nagios@logrobot-04:/var/log$
nagios@logrobot-04:/var/log$ logrobot autofig /var/log/syslog 1h '.' '.' 1 2 -exceldm

frq=19,zsc=3.01441,asc=[Oct-17-(12:20)]
frq=17,zsc=2.4241,asc=[Oct-17-(12:19)]
frq=15,zsc=1.8338,asc=[Oct-17-(12:56)]
frq=15,zsc=1.8338,asc=[Oct-17-(12:23)]
frq=15,zsc=1.8338,asc=[Oct-17-(12:18)]
frq=14,zsc=1.53865,asc=[Oct-17-(12:55)]
frq=14,zsc=1.53865,asc=[Oct-17-(12:05)]
frq=13,zsc=1.2435,asc=[Oct-17-(12:50)]
frq=13,zsc=1.2435,asc=[Oct-17-(12:24)]
frq=12,zsc=0.948352,asc=[Oct-17-(12:57)]
frq=12,zsc=0.948352,asc=[Oct-17-(12:25)]
frq=12,zsc=0.948352,asc=[Oct-17-(12:15)]
frq=11,zsc=0.653201,asc=[Oct-17-(12:54)]
frq=11,zsc=0.653201,asc=[Oct-17-(12:45)]
truncated...