[OBS-3614] syslog messages appear truncated for Citrix ADC - Observium

Details

Type: Bug
Resolution: Fixed
Priority: Minor
Fix Version/s: None
Affects Version/s: Professional Edition
Component/s: Alerting
Labels:
- syslog
Environment:
Centos 7

Description

We are using rsyslog to capture the logs in Observium. The messages from our Citrix ADC virtual appliances seem to be truncated.

Original message (tcpdump on port 514):

10:49:36.461706 IP (tos 0x0, ttl 253, id 19565, offset 0, flags [none], proto UDP (17), length 228)
10.250.128.20.32730 > 10.250.32.100.syslog: [udp sum ok] SYSLOG, length: 200
Facility local0 (16), Severity notice (5)
Msg: 01/19/2021:09:49:33 GMT DCRX-ANS-P003 0-PPE-0 : default EVENT DEVICEDOWN 35996673 0 : Device "server_serviceGroup_NSSVC_SSL_10.250.65.21:8636(SVG_PRD_HTTPS_DS?DCRX-LDM-P002?8636)" - State DOWN\0x0a
0x0000: 3c31 3333 3e20 3031 2f31 392f 3230 3231
0x0010: 3a30 393a 3439 3a33 3320 474d 5420 4443
0x0020: 5258 2d41 4e53 2d50 3030 3320 302d 5050
0x0030: 452d 3020 3a20 6465 6661 756c 7420 4556
0x0040: 454e 5420 4445 5649 4345 444f 574e 2033
0x0050: 3539 3936 3637 3320 3020 3a20 2044 6576
0x0060: 6963 6520 2273 6572 7665 725f 7365 7276
0x0070: 6963 6547 726f 7570 5f4e 5353 5643 5f53
0x0080: 534c 5f31 302e 3235 302e 3635 2e32 313a
0x0090: 3836 3336 2853 5647 5f50 5244 5f48 5454
0x00a0: 5053 5f44 533f 4443 5258 2d4c 444d 2d50
0x00b0: 3030 323f 3836 3336 2922 202d 2053 7461
0x00c0: 7465 2044 4f57 4e0a

Syslog debug output (enabled #$config['syslog']['debug'] = TRUE:

[2021/01/19 10:48:22 +0100] syslog.php(84224): dcrx-ans-p003||16||5||5||||2021-01-19 10:48:22|| 01/19/2021:09:48:19 GMT DCRX-ANS-P003 0-PPE-0 : default EVENT DEVICEDOWN 35993973 0 : Device "server_serviceGroup_NSSVC_SSL_10.250.65.21:8636(SVG_PRD_HTTPS_DS?DCRX-LDM-P002?8636)" - State DOWN||
[2021/01/19 10:48:59 +0100] syslog.php(84224): dcrx-ans-p003||16||5||5||||2021-01-19 10:48:59|| 01/19/2021:09:48:56 GMT DCRX-ANS-P003 0-PPE-0 : default EVENT DEVICEUP 35995288 0 : Device "server_serviceGroup_NSSVC_SSL_10.250.65.21:8636(SVG_PRD_HTTPS_DS?DCRX-LDM-P002?8636)" - State UP||
[2021/01/19 10:49:36 +0100] syslog.php(84224): dcrx-ans-p003||16||5||5||||2021-01-19 10:49:36|| 01/19/2021:09:49:33 GMT DCRX-ANS-P003 0-PPE-0 : default EVENT DEVICEDOWN 35996673 0 : Device "server_serviceGroup_NSSVC_SSL_10.250.65.21:8636(SVG_PRD_HTTPS_DS?DCRX-LDM-P002?8636)" - State DOWN||
[2021/01/19 10:50:08 +0100] syslog.php(84224): dcrx-ans-p003||16||5||5||||2021-01-19 10:50:08|| 01/19/2021:09:50:06 GMT DCRX-ANS-P003 0-PPE-0 : default EVENT DEVICEUP 35997958 0 : Device "server_serviceGroup_NSSVC_SSL_10.250.65.21:8636(SVG_PRD_HTTPS_DS?DCRX-LDM-P002?8636)" - State UP||
[2021/01/19 10:50:24 +0100] syslog.php(84224): dcrx-ans-p003||16||5||5||||2021-01-19 10:50:24|| 01/19/2021:09:50:22 GMT DCRX-ANS-P003 0-PPE-0 : default EVENT DEVICEDOWN 35998539 0 : Device "server_serviceGroup_NSSVC_SSL_10.250.65.21:8636(SVG_PRD_HTTPS_DS?DCRX-LDM-P002?8636)" - State DOWN||
[2021/01/19 10:50:57 +0100] syslog.php(84224): dcrx-ans-p003||16||5||5||||2021-01-19 10:50:57|| 01/19/2021:09:50:54 GMT DCRX-ANS-P003 0-PPE-0 : default EVENT DEVICEUP 35999829 0 : Device "server_serviceGroup_NSSVC_SSL_10.250.65.21:8636(SVG_PRD_HTTPS_DS?DCRX-LDM-P002?8636)" - State UP||

The output in the Observium web interface:

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

image-2021-01-19-10-59-21-686.png
102 kB
2021/01/19 09:59 AM
image-2021-01-22-08-27-49-284.png
21 kB
2021/01/22 07:27 AM
image-2021-01-22-08-29-18-695.png
46 kB
2021/01/22 07:29 AM
Observium - Syslog 2021-02-10 01-56-33.png
389 kB
2021/02/09 10:57 PM

Issue Links

relates to

OBS-523 Can we have netscaler syslog parsed by syslog.php?

Closed

Activity

[OBS-3614] syslog messages appear truncated for Citrix ADC

Mike Stupalov added a comment - 2021/02/09 11:53 PM

Fixed in r10995.

For rsyslog need restart service after update.

Mike Stupalov added a comment - 2021/02/09 11:53 PM Fixed in r10995. For rsyslog need restart service after update.

Mike Stupalov added a comment - 2021/02/09 11:02 PM

Hrm, but your messages stored differently

Mike Stupalov added a comment - 2021/02/09 11:02 PM Hrm, but your messages stored differently

Mike Stupalov added a comment - 2021/02/09 11:00 PM

Just tested your debug syslog entries:

I only can said - observium store full message which received from rsyslog :/

Mike Stupalov added a comment - 2021/02/09 11:00 PM Just tested your debug syslog entries: I only can said - observium store full message which received from rsyslog :/

mdwnn added a comment - 2021/02/09 02:50 PM

I added the debug output. I assume this is what you needed?

mdwnn added a comment - 2021/02/09 02:50 PM I added the debug output. I assume this is what you needed?

mdwnn added a comment - 2021/01/26 01:33 PM

This is the debug output:

[2021/01/26 13:47:08 +0100] syslog.php(99197): dcrx-ans-n004||16||5||5||||2021-01-26 13:47:07|| 01/26/2021:12:47:07 GMT DCRX-ANS-N004 0-PPE-0 : default EVENT DEVICEDOWN 62431870 0 :  Device "server_serviceGroup_NSSVC_TCP_172.16.200.150:636(SVG_TEST_LDAPS_ADMB?DC-BRU-150?636)" - State DOWN||

[2021/01/26 13:47:08 +0100] syslog.php(99197): dcrx-ans-n004||16||5||5||||2021-01-26 13:47:08|| 01/26/2021:12:47:08 GMT DCRX-ANS-N004 0-PPE-0 : default EVENT DEVICEDOWN 62431875 0 :  Device "server_serviceGroup_NSSVC_TCP_10.250.64.12:389(SVG_TEST_LDAP_LIANTIS?DCRX-WDC-P002?389)" - State DOWN||

[2021/01/26 13:47:09 +0100] syslog.php(99197): dcrx-ans-n004||16||5||5||||2021-01-26 13:47:09|| 01/26/2021:12:47:09 GMT DCRX-ANS-N004 0-PPE-0 : default EVENT DEVICEDOWN 62431923 0 :  Device "server_serviceGroup_NSSVC_TCP_172.16.200.150:636(SVG_DEV_LDAPS_ADMB?DC-BRU-150?636)" - State DOWN||

[2021/01/26 13:47:38 +0100] syslog.php(99197): dcrx-ans-n004||16||5||5||||2021-01-26 13:47:38|| 01/26/2021:12:47:38 GMT DCRX-ANS-N004 0-PPE-0 : default EVENT DEVICEUP 62432797 0 :  Device "server_serviceGroup_NSSVC_TCP_172.16.200.150:636(SVG_TEST_LDAPS_ADMB?DC-BRU-150?636)" - State UP||

[2021/01/26 13:47:38 +0100] syslog.php(99197): dcrx-ans-n004||16||5||5||||2021-01-26 13:47:38|| 01/26/2021:12:47:38 GMT DCRX-ANS-N004 0-PPE-0 : default EVENT DEVICEUP 62432807 0 :  Device "server_serviceGroup_NSSVC_TCP_10.250.64.12:389(SVG_TEST_LDAP_LIANTIS?DCRX-WDC-P002?389)" - State UP||

[2021/01/26 13:47:40 +0100] syslog.php(99197): dcrx-ans-n004||16||5||5||||2021-01-26 13:47:40|| 01/26/2021:12:47:40 GMT DCRX-ANS-N004 0-PPE-0 : default EVENT DEVICEUP 62432846 0 :  Device "server_serviceGroup_NSSVC_TCP_172.16.200.150:636(SVG_DEV_LDAPS_ADMB?DC-BRU-150?636)" - State UP||

mdwnn added a comment - 2021/01/26 01:33 PM This is the debug output: [2021/01/26 13:47:08 +0100] syslog.php(99197): dcrx-ans-n004||16||5||5||||2021-01-26 13:47:07|| 01/26/2021:12:47:07 GMT DCRX-ANS-N004 0-PPE-0 : default EVENT DEVICEDOWN 62431870 0 : Device "server_serviceGroup_NSSVC_TCP_172.16.200.150:636(SVG_TEST_LDAPS_ADMB?DC-BRU-150?636)" - State DOWN|| [2021/01/26 13:47:08 +0100] syslog.php(99197): dcrx-ans-n004||16||5||5||||2021-01-26 13:47:08|| 01/26/2021:12:47:08 GMT DCRX-ANS-N004 0-PPE-0 : default EVENT DEVICEDOWN 62431875 0 : Device "server_serviceGroup_NSSVC_TCP_10.250.64.12:389(SVG_TEST_LDAP_LIANTIS?DCRX-WDC-P002?389)" - State DOWN|| [2021/01/26 13:47:09 +0100] syslog.php(99197): dcrx-ans-n004||16||5||5||||2021-01-26 13:47:09|| 01/26/2021:12:47:09 GMT DCRX-ANS-N004 0-PPE-0 : default EVENT DEVICEDOWN 62431923 0 : Device "server_serviceGroup_NSSVC_TCP_172.16.200.150:636(SVG_DEV_LDAPS_ADMB?DC-BRU-150?636)" - State DOWN|| [2021/01/26 13:47:38 +0100] syslog.php(99197): dcrx-ans-n004||16||5||5||||2021-01-26 13:47:38|| 01/26/2021:12:47:38 GMT DCRX-ANS-N004 0-PPE-0 : default EVENT DEVICEUP 62432797 0 : Device "server_serviceGroup_NSSVC_TCP_172.16.200.150:636(SVG_TEST_LDAPS_ADMB?DC-BRU-150?636)" - State UP|| [2021/01/26 13:47:38 +0100] syslog.php(99197): dcrx-ans-n004||16||5||5||||2021-01-26 13:47:38|| 01/26/2021:12:47:38 GMT DCRX-ANS-N004 0-PPE-0 : default EVENT DEVICEUP 62432807 0 : Device "server_serviceGroup_NSSVC_TCP_10.250.64.12:389(SVG_TEST_LDAP_LIANTIS?DCRX-WDC-P002?389)" - State UP|| [2021/01/26 13:47:40 +0100] syslog.php(99197): dcrx-ans-n004||16||5||5||||2021-01-26 13:47:40|| 01/26/2021:12:47:40 GMT DCRX-ANS-N004 0-PPE-0 : default EVENT DEVICEUP 62432846 0 : Device "server_serviceGroup_NSSVC_TCP_172.16.200.150:636(SVG_DEV_LDAPS_ADMB?DC-BRU-150?636)" - State UP||

mdwnn added a comment - 2021/01/26 08:27 AM

I restarted it (both restart and stop/start). I enabled debugging and I will update the case when I have some messages.

mdwnn added a comment - 2021/01/26 08:27 AM I restarted it (both restart and stop/start). I enabled debugging and I will update the case when I have some messages.

Mike Stupalov added a comment - 2021/01/23 10:14 AM

did you restart rsyslog service?

Show lines in Syslog debug output.

Mike Stupalov added a comment - 2021/01/23 10:14 AM did you restart rsyslog service? Show lines in Syslog debug output.

mdwnn added a comment - 2021/01/22 07:30 AM

Hi Mike,

I tried the change to the rsyslog template, but it does not seem to work:

08:28:36.181757 IP (tos 0x0, ttl 252, id 1443, offset 0, flags [none], proto UDP (17), length 225)
10.251.160.21.34320 > 10.250.32.100.syslog: [udp sum ok] SYSLOG, length: 197
Facility local0 (16), Severity notice (5)
Msg: 01/22/2021:07:28:36 GMT DCRX-ANS-N004 0-PPE-0 : default EVENT DEVICEUP 51974872 0 : Device "server_serviceGroup_NSSVC_TCP_10.211.67.69:1636(SVG_TEST_LDAPS_DS?DCRX-LDM-T002?1636)" - State UP\0x0a
0x0000: 3c31 3333 3e20 3031 2f32 322f 3230 3231
0x0010: 3a30 373a 3238 3a33 3620 474d 5420 4443
0x0020: 5258 2d41 4e53 2d4e 3030 3420 302d 5050
0x0030: 452d 3020 3a20 6465 6661 756c 7420 4556
0x0040: 454e 5420 4445 5649 4345 5550 2035 3139
0x0050: 3734 3837 3220 3020 3a20 2044 6576 6963
0x0060: 6520 2273 6572 7665 725f 7365 7276 6963
0x0070: 6547 726f 7570 5f4e 5353 5643 5f54 4350
0x0080: 5f31 302e 3231 312e 3637 2e36 393a 3136
0x0090: 3336 2853 5647 5f54 4553 545f 4c44 4150
0x00a0: 535f 4453 3f44 4352 582d 4c44 4d2d 5430
0x00b0: 3032 3f31 3633 3629 2220 2d20 5374 6174
0x00c0: 6520 5550 0a
08:28:46.648224 IP (tos 0x0, ttl 252, id 1444, offset 0, flags [none], proto UDP (17), length 224)
10.251.160.21.34320 > 10.250.32.100.syslog: [udp sum ok] SYSLOG, length: 196
Facility local0 (16), Severity notice (5)
Msg: 01/22/2021:07:28:46 GMT DCRX-ANS-N004 0-PPE-0 : default EVENT DEVICEUP 51975230 0 : Device "server_serviceGroup_NSSVC_TCP_10.253.97.16:1636(SVG_OPL_LDAPS_DS?DCRX-LDM-O002?1636)" - State UP\0x0a
0x0000: 3c31 3333 3e20 3031 2f32 322f 3230 3231
0x0010: 3a30 373a 3238 3a34 3620 474d 5420 4443
0x0020: 5258 2d41 4e53 2d4e 3030 3420 302d 5050
0x0030: 452d 3020 3a20 6465 6661 756c 7420 4556
0x0040: 454e 5420 4445 5649 4345 5550 2035 3139
0x0050: 3735 3233 3020 3020 3a20 2044 6576 6963
0x0060: 6520 2273 6572 7665 725f 7365 7276 6963
0x0070: 6547 726f 7570 5f4e 5353 5643 5f54 4350
0x0080: 5f31 302e 3235 332e 3937 2e31 363a 3136
0x0090: 3336 2853 5647 5f4f 504c 5f4c 4441 5053
0x00a0: 5f44 533f 4443 5258 2d4c 444d 2d4f 3030
0x00b0: 323f 3136 3336 2922 202d 2053 7461 7465
0x00c0: 2055 500a

#---------------------------------------------------------

# send remote logs to observium# provides UDP syslog reception

module(load="imudp")input(type="imudp"

      port="514"

      ruleset="observium")## provides TCP syslog reception (uncomment if required)

#module(load="imptcp")

#input(type="imptcp"

#      port="514"

#      ruleset="observium")module(load="omprog")# observium syslog template

template(name="observium"

         type="string"

         string="%fromhost%||%syslogfacility%||%syslogpriority%||%syslogseverity%||%syslogtag%||%$year%-%$month%-%$day% %timereported:8:25%||%msg:::space-cc%||%programname%\n")# observium RuleSets

ruleset(name="observium") {

    action(type="omprog"

           binary="/data/observium/syslog.php"

           template="observium")

    stop

}#---------------------------------------------------------

mdwnn added a comment - 2021/01/22 07:30 AM Hi Mike, I tried the change to the rsyslog template, but it does not seem to work: 08:28:36.181757 IP (tos 0x0, ttl 252, id 1443, offset 0, flags [none] , proto UDP (17), length 225) 10.251.160.21.34320 > 10.250.32.100.syslog: [udp sum ok] SYSLOG, length: 197 Facility local0 (16), Severity notice (5) Msg: 01/22/2021:07:28:36 GMT DCRX-ANS-N004 0-PPE-0 : default EVENT DEVICEUP 51974872 0 : Device "server_serviceGroup_NSSVC_TCP_10.211.67.69:1636(SVG_TEST_LDAPS_DS?DCRX-LDM-T002?1636)" - State UP\0x0a 0x0000: 3c31 3333 3e20 3031 2f32 322f 3230 3231 0x0010: 3a30 373a 3238 3a33 3620 474d 5420 4443 0x0020: 5258 2d41 4e53 2d4e 3030 3420 302d 5050 0x0030: 452d 3020 3a20 6465 6661 756c 7420 4556 0x0040: 454e 5420 4445 5649 4345 5550 2035 3139 0x0050: 3734 3837 3220 3020 3a20 2044 6576 6963 0x0060: 6520 2273 6572 7665 725f 7365 7276 6963 0x0070: 6547 726f 7570 5f4e 5353 5643 5f54 4350 0x0080: 5f31 302e 3231 312e 3637 2e36 393a 3136 0x0090: 3336 2853 5647 5f54 4553 545f 4c44 4150 0x00a0: 535f 4453 3f44 4352 582d 4c44 4d2d 5430 0x00b0: 3032 3f31 3633 3629 2220 2d20 5374 6174 0x00c0: 6520 5550 0a 08:28:46.648224 IP (tos 0x0, ttl 252, id 1444, offset 0, flags [none] , proto UDP (17), length 224) 10.251.160.21.34320 > 10.250.32.100.syslog: [udp sum ok] SYSLOG, length: 196 Facility local0 (16), Severity notice (5) Msg: 01/22/2021:07:28:46 GMT DCRX-ANS-N004 0-PPE-0 : default EVENT DEVICEUP 51975230 0 : Device "server_serviceGroup_NSSVC_TCP_10.253.97.16:1636(SVG_OPL_LDAPS_DS?DCRX-LDM-O002?1636)" - State UP\0x0a 0x0000: 3c31 3333 3e20 3031 2f32 322f 3230 3231 0x0010: 3a30 373a 3238 3a34 3620 474d 5420 4443 0x0020: 5258 2d41 4e53 2d4e 3030 3420 302d 5050 0x0030: 452d 3020 3a20 6465 6661 756c 7420 4556 0x0040: 454e 5420 4445 5649 4345 5550 2035 3139 0x0050: 3735 3233 3020 3020 3a20 2044 6576 6963 0x0060: 6520 2273 6572 7665 725f 7365 7276 6963 0x0070: 6547 726f 7570 5f4e 5353 5643 5f54 4350 0x0080: 5f31 302e 3235 332e 3937 2e31 363a 3136 0x0090: 3336 2853 5647 5f4f 504c 5f4c 4441 5053 0x00a0: 5f44 533f 4443 5258 2d4c 444d 2d4f 3030 0x00b0: 323f 3136 3336 2922 202d 2053 7461 7465 0x00c0: 2055 500a #--------------------------------------------------------- # send remote logs to observium# provides UDP syslog reception module(load="imudp")input(type="imudp" port="514" ruleset="observium")## provides TCP syslog reception (uncomment if required) #module(load="imptcp") # #input(type="imptcp" # port="514" # ruleset="observium")module(load="omprog")# observium syslog template template(name="observium" type="string" string="%fromhost%||%syslogfacility%||%syslogpriority%||%syslogseverity%||%syslogtag%||%$year%-%$month%-%$day% %timereported:8:25%||%msg:::space-cc%||%programname%\n")# observium RuleSets ruleset(name="observium") { action(type="omprog" binary="/data/observium/syslog.php" template="observium") stop }#---------------------------------------------------------

People

Assignee:: Mike Stupalov

Reporter:: mdwnn

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Dates

Due:: 2021/02/11

Created:: 2021/01/19 10:01 AM

Updated:: 2021/03/12 06:10 PM

Resolved:: 2021/02/09 11:53 PM