Uploaded image for project: 'Observium'
  1. Observium
  2. OBS-3614

syslog messages appear truncated for Citrix ADC

Details

    • Bug
    • Resolution: Fixed
    • Minor
    • None
    • Professional Edition
    • Alerting
    • Centos 7

    Description

      We are using rsyslog to capture the logs in Observium.  The messages from our Citrix ADC virtual appliances seem to be truncated.

      Original message (tcpdump on port 514):

      10:49:36.461706 IP (tos 0x0, ttl 253, id 19565, offset 0, flags [none], proto UDP (17), length 228)
      10.250.128.20.32730 > 10.250.32.100.syslog: [udp sum ok] SYSLOG, length: 200
      Facility local0 (16), Severity notice (5)
      Msg: 01/19/2021:09:49:33 GMT DCRX-ANS-P003 0-PPE-0 : default EVENT DEVICEDOWN 35996673 0 : Device "server_serviceGroup_NSSVC_SSL_10.250.65.21:8636(SVG_PRD_HTTPS_DS?DCRX-LDM-P002?8636)" - State DOWN\0x0a
      0x0000: 3c31 3333 3e20 3031 2f31 392f 3230 3231
      0x0010: 3a30 393a 3439 3a33 3320 474d 5420 4443
      0x0020: 5258 2d41 4e53 2d50 3030 3320 302d 5050
      0x0030: 452d 3020 3a20 6465 6661 756c 7420 4556
      0x0040: 454e 5420 4445 5649 4345 444f 574e 2033
      0x0050: 3539 3936 3637 3320 3020 3a20 2044 6576
      0x0060: 6963 6520 2273 6572 7665 725f 7365 7276
      0x0070: 6963 6547 726f 7570 5f4e 5353 5643 5f53
      0x0080: 534c 5f31 302e 3235 302e 3635 2e32 313a
      0x0090: 3836 3336 2853 5647 5f50 5244 5f48 5454
      0x00a0: 5053 5f44 533f 4443 5258 2d4c 444d 2d50
      0x00b0: 3030 323f 3836 3336 2922 202d 2053 7461
      0x00c0: 7465 2044 4f57 4e0a

      Syslog debug output (enabled #$config['syslog']['debug'] = TRUE:

      [2021/01/19 10:48:22 +0100] syslog.php(84224): dcrx-ans-p003||16||5||5||||2021-01-19 10:48:22|| 01/19/2021:09:48:19 GMT DCRX-ANS-P003 0-PPE-0 : default EVENT DEVICEDOWN 35993973 0 : Device "server_serviceGroup_NSSVC_SSL_10.250.65.21:8636(SVG_PRD_HTTPS_DS?DCRX-LDM-P002?8636)" - State DOWN||
      [2021/01/19 10:48:59 +0100] syslog.php(84224): dcrx-ans-p003||16||5||5||||2021-01-19 10:48:59|| 01/19/2021:09:48:56 GMT DCRX-ANS-P003 0-PPE-0 : default EVENT DEVICEUP 35995288 0 : Device "server_serviceGroup_NSSVC_SSL_10.250.65.21:8636(SVG_PRD_HTTPS_DS?DCRX-LDM-P002?8636)" - State UP||
      [2021/01/19 10:49:36 +0100] syslog.php(84224): dcrx-ans-p003||16||5||5||||2021-01-19 10:49:36|| 01/19/2021:09:49:33 GMT DCRX-ANS-P003 0-PPE-0 : default EVENT DEVICEDOWN 35996673 0 : Device "server_serviceGroup_NSSVC_SSL_10.250.65.21:8636(SVG_PRD_HTTPS_DS?DCRX-LDM-P002?8636)" - State DOWN||
      [2021/01/19 10:50:08 +0100] syslog.php(84224): dcrx-ans-p003||16||5||5||||2021-01-19 10:50:08|| 01/19/2021:09:50:06 GMT DCRX-ANS-P003 0-PPE-0 : default EVENT DEVICEUP 35997958 0 : Device "server_serviceGroup_NSSVC_SSL_10.250.65.21:8636(SVG_PRD_HTTPS_DS?DCRX-LDM-P002?8636)" - State UP||
      [2021/01/19 10:50:24 +0100] syslog.php(84224): dcrx-ans-p003||16||5||5||||2021-01-19 10:50:24|| 01/19/2021:09:50:22 GMT DCRX-ANS-P003 0-PPE-0 : default EVENT DEVICEDOWN 35998539 0 : Device "server_serviceGroup_NSSVC_SSL_10.250.65.21:8636(SVG_PRD_HTTPS_DS?DCRX-LDM-P002?8636)" - State DOWN||
      [2021/01/19 10:50:57 +0100] syslog.php(84224): dcrx-ans-p003||16||5||5||||2021-01-19 10:50:57|| 01/19/2021:09:50:54 GMT DCRX-ANS-P003 0-PPE-0 : default EVENT DEVICEUP 35999829 0 : Device "server_serviceGroup_NSSVC_SSL_10.250.65.21:8636(SVG_PRD_HTTPS_DS?DCRX-LDM-P002?8636)" - State UP||

       

      The output in the Observium web interface:

       

      Attachments

        Issue Links

          Activity

            [OBS-3614] syslog messages appear truncated for Citrix ADC

            Fixed in r10995.

             

            For rsyslog need restart service after update.

            landy Mike Stupalov added a comment - Fixed in r10995.   For rsyslog need restart service after update.

            Hrm, but your messages stored differently

             

            landy Mike Stupalov added a comment - Hrm, but your messages stored differently  

            Just tested your debug syslog entries:

            I only can said - observium store full message which received from rsyslog :/

            landy Mike Stupalov added a comment - Just tested your debug syslog entries: I only can said - observium store full message which received from rsyslog :/
            mdwnn mdwnn added a comment -

            I added the debug output. I assume this is what you needed?

            mdwnn mdwnn added a comment - I added the debug output. I assume this is what you needed?
            mdwnn mdwnn added a comment -

            This is the debug output:

            [2021/01/26 13:47:08 +0100] syslog.php(99197): dcrx-ans-n004||16||5||5||||2021-01-26 13:47:07|| 01/26/2021:12:47:07 GMT DCRX-ANS-N004 0-PPE-0 : default EVENT DEVICEDOWN 62431870 0 :  Device "server_serviceGroup_NSSVC_TCP_172.16.200.150:636(SVG_TEST_LDAPS_ADMB?DC-BRU-150?636)" - State DOWN||
            [2021/01/26 13:47:08 +0100] syslog.php(99197): dcrx-ans-n004||16||5||5||||2021-01-26 13:47:08|| 01/26/2021:12:47:08 GMT DCRX-ANS-N004 0-PPE-0 : default EVENT DEVICEDOWN 62431875 0 :  Device "server_serviceGroup_NSSVC_TCP_10.250.64.12:389(SVG_TEST_LDAP_LIANTIS?DCRX-WDC-P002?389)" - State DOWN||
            [2021/01/26 13:47:09 +0100] syslog.php(99197): dcrx-ans-n004||16||5||5||||2021-01-26 13:47:09|| 01/26/2021:12:47:09 GMT DCRX-ANS-N004 0-PPE-0 : default EVENT DEVICEDOWN 62431923 0 :  Device "server_serviceGroup_NSSVC_TCP_172.16.200.150:636(SVG_DEV_LDAPS_ADMB?DC-BRU-150?636)" - State DOWN||
            [2021/01/26 13:47:38 +0100] syslog.php(99197): dcrx-ans-n004||16||5||5||||2021-01-26 13:47:38|| 01/26/2021:12:47:38 GMT DCRX-ANS-N004 0-PPE-0 : default EVENT DEVICEUP 62432797 0 :  Device "server_serviceGroup_NSSVC_TCP_172.16.200.150:636(SVG_TEST_LDAPS_ADMB?DC-BRU-150?636)" - State UP||
            [2021/01/26 13:47:38 +0100] syslog.php(99197): dcrx-ans-n004||16||5||5||||2021-01-26 13:47:38|| 01/26/2021:12:47:38 GMT DCRX-ANS-N004 0-PPE-0 : default EVENT DEVICEUP 62432807 0 :  Device "server_serviceGroup_NSSVC_TCP_10.250.64.12:389(SVG_TEST_LDAP_LIANTIS?DCRX-WDC-P002?389)" - State UP||
            [2021/01/26 13:47:40 +0100] syslog.php(99197): dcrx-ans-n004||16||5||5||||2021-01-26 13:47:40|| 01/26/2021:12:47:40 GMT DCRX-ANS-N004 0-PPE-0 : default EVENT DEVICEUP 62432846 0 :  Device "server_serviceGroup_NSSVC_TCP_172.16.200.150:636(SVG_DEV_LDAPS_ADMB?DC-BRU-150?636)" - State UP||
            

            mdwnn mdwnn added a comment - This is the debug output: [2021/01/26 13:47:08 +0100] syslog.php(99197): dcrx-ans-n004||16||5||5||||2021-01-26 13:47:07|| 01/26/2021:12:47:07 GMT DCRX-ANS-N004 0-PPE-0 : default EVENT DEVICEDOWN 62431870 0 : Device "server_serviceGroup_NSSVC_TCP_172.16.200.150:636(SVG_TEST_LDAPS_ADMB?DC-BRU-150?636)" - State DOWN|| [2021/01/26 13:47:08 +0100] syslog.php(99197): dcrx-ans-n004||16||5||5||||2021-01-26 13:47:08|| 01/26/2021:12:47:08 GMT DCRX-ANS-N004 0-PPE-0 : default EVENT DEVICEDOWN 62431875 0 : Device "server_serviceGroup_NSSVC_TCP_10.250.64.12:389(SVG_TEST_LDAP_LIANTIS?DCRX-WDC-P002?389)" - State DOWN|| [2021/01/26 13:47:09 +0100] syslog.php(99197): dcrx-ans-n004||16||5||5||||2021-01-26 13:47:09|| 01/26/2021:12:47:09 GMT DCRX-ANS-N004 0-PPE-0 : default EVENT DEVICEDOWN 62431923 0 : Device "server_serviceGroup_NSSVC_TCP_172.16.200.150:636(SVG_DEV_LDAPS_ADMB?DC-BRU-150?636)" - State DOWN|| [2021/01/26 13:47:38 +0100] syslog.php(99197): dcrx-ans-n004||16||5||5||||2021-01-26 13:47:38|| 01/26/2021:12:47:38 GMT DCRX-ANS-N004 0-PPE-0 : default EVENT DEVICEUP 62432797 0 : Device "server_serviceGroup_NSSVC_TCP_172.16.200.150:636(SVG_TEST_LDAPS_ADMB?DC-BRU-150?636)" - State UP|| [2021/01/26 13:47:38 +0100] syslog.php(99197): dcrx-ans-n004||16||5||5||||2021-01-26 13:47:38|| 01/26/2021:12:47:38 GMT DCRX-ANS-N004 0-PPE-0 : default EVENT DEVICEUP 62432807 0 : Device "server_serviceGroup_NSSVC_TCP_10.250.64.12:389(SVG_TEST_LDAP_LIANTIS?DCRX-WDC-P002?389)" - State UP|| [2021/01/26 13:47:40 +0100] syslog.php(99197): dcrx-ans-n004||16||5||5||||2021-01-26 13:47:40|| 01/26/2021:12:47:40 GMT DCRX-ANS-N004 0-PPE-0 : default EVENT DEVICEUP 62432846 0 : Device "server_serviceGroup_NSSVC_TCP_172.16.200.150:636(SVG_DEV_LDAPS_ADMB?DC-BRU-150?636)" - State UP||
            mdwnn mdwnn added a comment -

            I restarted it (both restart and stop/start). I enabled debugging and I will update the case when I have some messages.

            mdwnn mdwnn added a comment - I restarted it (both restart and stop/start). I enabled debugging and I will update the case when I have some messages.

            did you restart rsyslog service?

            Show lines in Syslog debug output.

            landy Mike Stupalov added a comment - did you restart rsyslog service? Show lines in Syslog debug output.
            mdwnn mdwnn added a comment -

            Hi Mike,

            I tried the change to the rsyslog template, but it does not seem to work:

             

            08:28:36.181757 IP (tos 0x0, ttl 252, id 1443, offset 0, flags [none], proto UDP (17), length 225)
            10.251.160.21.34320 > 10.250.32.100.syslog: [udp sum ok] SYSLOG, length: 197
            Facility local0 (16), Severity notice (5)
            Msg: 01/22/2021:07:28:36 GMT DCRX-ANS-N004 0-PPE-0 : default EVENT DEVICEUP 51974872 0 : Device "server_serviceGroup_NSSVC_TCP_10.211.67.69:1636(SVG_TEST_LDAPS_DS?DCRX-LDM-T002?1636)" - State UP\0x0a
            0x0000: 3c31 3333 3e20 3031 2f32 322f 3230 3231
            0x0010: 3a30 373a 3238 3a33 3620 474d 5420 4443
            0x0020: 5258 2d41 4e53 2d4e 3030 3420 302d 5050
            0x0030: 452d 3020 3a20 6465 6661 756c 7420 4556
            0x0040: 454e 5420 4445 5649 4345 5550 2035 3139
            0x0050: 3734 3837 3220 3020 3a20 2044 6576 6963
            0x0060: 6520 2273 6572 7665 725f 7365 7276 6963
            0x0070: 6547 726f 7570 5f4e 5353 5643 5f54 4350
            0x0080: 5f31 302e 3231 312e 3637 2e36 393a 3136
            0x0090: 3336 2853 5647 5f54 4553 545f 4c44 4150
            0x00a0: 535f 4453 3f44 4352 582d 4c44 4d2d 5430
            0x00b0: 3032 3f31 3633 3629 2220 2d20 5374 6174
            0x00c0: 6520 5550 0a
            08:28:46.648224 IP (tos 0x0, ttl 252, id 1444, offset 0, flags [none], proto UDP (17), length 224)
            10.251.160.21.34320 > 10.250.32.100.syslog: [udp sum ok] SYSLOG, length: 196
            Facility local0 (16), Severity notice (5)
            Msg: 01/22/2021:07:28:46 GMT DCRX-ANS-N004 0-PPE-0 : default EVENT DEVICEUP 51975230 0 : Device "server_serviceGroup_NSSVC_TCP_10.253.97.16:1636(SVG_OPL_LDAPS_DS?DCRX-LDM-O002?1636)" - State UP\0x0a
            0x0000: 3c31 3333 3e20 3031 2f32 322f 3230 3231
            0x0010: 3a30 373a 3238 3a34 3620 474d 5420 4443
            0x0020: 5258 2d41 4e53 2d4e 3030 3420 302d 5050
            0x0030: 452d 3020 3a20 6465 6661 756c 7420 4556
            0x0040: 454e 5420 4445 5649 4345 5550 2035 3139
            0x0050: 3735 3233 3020 3020 3a20 2044 6576 6963
            0x0060: 6520 2273 6572 7665 725f 7365 7276 6963
            0x0070: 6547 726f 7570 5f4e 5353 5643 5f54 4350
            0x0080: 5f31 302e 3235 332e 3937 2e31 363a 3136
            0x0090: 3336 2853 5647 5f4f 504c 5f4c 4441 5053
            0x00a0: 5f44 533f 4443 5258 2d4c 444d 2d4f 3030
            0x00b0: 323f 3136 3336 2922 202d 2053 7461 7465
            0x00c0: 2055 500a

             

             

            #---------------------------------------------------------
            # send remote logs to observium# provides UDP syslog reception
            module(load="imudp")input(type="imudp"
                  port="514"
                  ruleset="observium")## provides TCP syslog reception (uncomment if required)
            #module(load="imptcp")
            #
            #input(type="imptcp"
            #      port="514"
            #      ruleset="observium")module(load="omprog")# observium syslog template
            template(name="observium"
                     type="string"
                     string="%fromhost%||%syslogfacility%||%syslogpriority%||%syslogseverity%||%syslogtag%||%$year%-%$month%-%$day% %timereported:8:25%||%msg:::space-cc%||%programname%\n")# observium RuleSets
            ruleset(name="observium") {
                action(type="omprog"
                       binary="/data/observium/syslog.php"
                       template="observium")
                stop
            }#---------------------------------------------------------
            

             

            mdwnn mdwnn added a comment - Hi Mike, I tried the change to the rsyslog template, but it does not seem to work:   08:28:36.181757 IP (tos 0x0, ttl 252, id 1443, offset 0, flags [none] , proto UDP (17), length 225) 10.251.160.21.34320 > 10.250.32.100.syslog: [udp sum ok] SYSLOG, length: 197 Facility local0 (16), Severity notice (5) Msg: 01/22/2021:07:28:36 GMT DCRX-ANS-N004 0-PPE-0 : default EVENT DEVICEUP 51974872 0 : Device "server_serviceGroup_NSSVC_TCP_10.211.67.69:1636(SVG_TEST_LDAPS_DS?DCRX-LDM-T002?1636)" - State UP\0x0a 0x0000: 3c31 3333 3e20 3031 2f32 322f 3230 3231 0x0010: 3a30 373a 3238 3a33 3620 474d 5420 4443 0x0020: 5258 2d41 4e53 2d4e 3030 3420 302d 5050 0x0030: 452d 3020 3a20 6465 6661 756c 7420 4556 0x0040: 454e 5420 4445 5649 4345 5550 2035 3139 0x0050: 3734 3837 3220 3020 3a20 2044 6576 6963 0x0060: 6520 2273 6572 7665 725f 7365 7276 6963 0x0070: 6547 726f 7570 5f4e 5353 5643 5f54 4350 0x0080: 5f31 302e 3231 312e 3637 2e36 393a 3136 0x0090: 3336 2853 5647 5f54 4553 545f 4c44 4150 0x00a0: 535f 4453 3f44 4352 582d 4c44 4d2d 5430 0x00b0: 3032 3f31 3633 3629 2220 2d20 5374 6174 0x00c0: 6520 5550 0a 08:28:46.648224 IP (tos 0x0, ttl 252, id 1444, offset 0, flags [none] , proto UDP (17), length 224) 10.251.160.21.34320 > 10.250.32.100.syslog: [udp sum ok] SYSLOG, length: 196 Facility local0 (16), Severity notice (5) Msg: 01/22/2021:07:28:46 GMT DCRX-ANS-N004 0-PPE-0 : default EVENT DEVICEUP 51975230 0 : Device "server_serviceGroup_NSSVC_TCP_10.253.97.16:1636(SVG_OPL_LDAPS_DS?DCRX-LDM-O002?1636)" - State UP\0x0a 0x0000: 3c31 3333 3e20 3031 2f32 322f 3230 3231 0x0010: 3a30 373a 3238 3a34 3620 474d 5420 4443 0x0020: 5258 2d41 4e53 2d4e 3030 3420 302d 5050 0x0030: 452d 3020 3a20 6465 6661 756c 7420 4556 0x0040: 454e 5420 4445 5649 4345 5550 2035 3139 0x0050: 3735 3233 3020 3020 3a20 2044 6576 6963 0x0060: 6520 2273 6572 7665 725f 7365 7276 6963 0x0070: 6547 726f 7570 5f4e 5353 5643 5f54 4350 0x0080: 5f31 302e 3235 332e 3937 2e31 363a 3136 0x0090: 3336 2853 5647 5f4f 504c 5f4c 4441 5053 0x00a0: 5f44 533f 4443 5258 2d4c 444d 2d4f 3030 0x00b0: 323f 3136 3336 2922 202d 2053 7461 7465 0x00c0: 2055 500a     #--------------------------------------------------------- # send remote logs to observium# provides UDP syslog reception module(load="imudp")input(type="imudp" port="514" ruleset="observium")## provides TCP syslog reception (uncomment if required) #module(load="imptcp") # #input(type="imptcp" # port="514" # ruleset="observium")module(load="omprog")# observium syslog template template(name="observium" type="string" string="%fromhost%||%syslogfacility%||%syslogpriority%||%syslogseverity%||%syslogtag%||%$year%-%$month%-%$day% %timereported:8:25%||%msg:::space-cc%||%programname%\n")# observium RuleSets ruleset(name="observium") { action(type="omprog" binary="/data/observium/syslog.php" template="observium") stop }#---------------------------------------------------------  

            Which syslog server you use (rsyslog/syslog-ng)?

            Anyway, this is not possible to fix on syslog parser.
            But need change syslog server template.

            I.e. for Rsyslog 8.x need change message string in template.
            If you configured as written in our docs here, than change template file /etc/rsyslog.d/30-observium.conf to:

            #---------------------------------------------------------
            # send remote logs to observium
             
            # provides UDP syslog reception
            module(load="imudp")
             
            input(type="imudp"
                  port="514"
                  ruleset="observium")
             
            ## provides TCP syslog reception (uncomment if required)
            #module(load="imptcp")
            #
            #input(type="imptcp"
            #      port="514"
            #      ruleset="observium")
             
            module(load="omprog")
             
            # observium syslog template
            template(name="observium"
                     type="string"
                     string="%fromhost%||%syslogfacility%||%syslogpriority%||%syslogseverity%||%syslogtag%||%$year%-%$month%-%$day% %timereported:8:25%||%msg:::space-cc%||%programname%\n")
             
            # observium RuleSets
            ruleset(name="observium") {
                action(type="omprog"
                       binary="/opt/observium/syslog.php"
                       template="observium")
                stop
            }
             
            #---------------------------------------------------------

            There is only chnaged %msg% tag to %msg:::space-cc%.
            See rsyslog description for this property here.
            (All new lines will replaced with spaces in message.

            After change template need restart rsyslogd:

            sudo service rsyslog restart

            landy Mike Stupalov added a comment - Which syslog server you use (rsyslog/syslog-ng)? Anyway, this is not possible to fix on syslog parser. But need change syslog server template. I.e. for Rsyslog 8.x need change message string in template. If you configured as written in our docs here , than change template file /etc/rsyslog.d/30-observium.conf to: #--------------------------------------------------------- # send remote logs to observium   # provides UDP syslog reception module(load="imudp")   input(type="imudp" port="514" ruleset="observium")   ## provides TCP syslog reception (uncomment if required) #module(load="imptcp") # #input(type="imptcp" # port="514" # ruleset="observium")   module(load="omprog")   # observium syslog template template(name="observium" type="string" string="%fromhost%||%syslogfacility%||%syslogpriority%||%syslogseverity%||%syslogtag%||%$year%-%$month%-%$day% %timereported:8:25%||%msg:::space-cc%||%programname%\n")   # observium RuleSets ruleset(name="observium") { action(type="omprog" binary="/opt/observium/syslog.php" template="observium") stop }   #--------------------------------------------------------- There is only chnaged %msg% tag to %msg:::space-cc% . See rsyslog description for this property here . (All new lines will replaced with spaces in message. After change template need restart rsyslogd: sudo service rsyslog restart
            mdwnn mdwnn added a comment - - edited

            no snmp involved, Observium version 20.11.10814 (11th November 2020)

            Citrix Netscaler NS12.1 (Build 59.16.nc)

            mdwnn mdwnn added a comment - - edited no snmp involved, Observium version 20.11.10814 (11th November 2020) Citrix Netscaler NS12.1 (Build 59.16.nc)

            People

              landy Mike Stupalov
              mdwnn mdwnn
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: