Details

    • Help
    • Resolution: Fixed
    • Major
    • None
    • Professional Edition
    • Alerting, Poller

    Description

      Hello all,

      Since moving to 13056 today, we detected that from a lot of EXOS devices their syslog is not been processed by observium, is like observium stopped collecting or writing, we can see old ones, but no new ones.

      We can see the syslog packet arrives correctly to the server, the demon is listening in right port 514 and the rsyslog and observium syslog process are running, but in the GUI we can not see it:

      root@observium:/opt/observium# tcpdump -i any port 514 and '(host 172.19.8.248 or host 192.168.130.250)' -n
      tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
      listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
      13:43:12.969469 ethertype IPv4, IP 172.19.8.248.514 > 172.19.8.182.514: SYSLOG local4.info, length: 106
      13:43:12.969469 IP 172.19.8.248.514 > 172.19.8.182.514: SYSLOG local4.info, length: 106
      13:43:12.970676 ethertype IPv4, IP 172.19.8.248.514 > 172.19.8.182.514: SYSLOG local4.info, length: 129
      13:43:12.970676 IP 172.19.8.248.514 > 172.19.8.182.514: SYSLOG local4.info, length: 129
      13:43:12.971010 ethertype IPv4, IP 172.19.8.248.514 > 172.19.8.182.514: SYSLOG local4.info, length: 127
      13:43:12.971010 IP 172.19.8.248.514 > 172.19.8.182.514: SYSLOG local4.info, length: 127
      13:43:13.141950 ethertype IPv4, IP 172.19.8.248.514 > 172.19.8.182.514: SYSLOG local4.warning, length: 168
      13:43:13.141950 IP 172.19.8.248.514 > 172.19.8.182.514: SYSLOG local4.warning, length: 168
      13:43:13.183165 ethertype IPv4, IP 172.19.8.248.514 > 172.19.8.182.514: SYSLOG local4.warning, length: 168
      13:43:13.183165 IP 172.19.8.248.514 > 172.19.8.182.514: SYSLOG local4.warning, length: 168
      13:43:14.745574 ethertype IPv4, IP 172.19.8.248.514 > 172.19.8.182.514: SYSLOG local4.info, length: 118
      13:43:14.745574 IP 172.19.8.248.514 > 172.19.8.182.514: SYSLOG local4.info, length: 118
      ^C
      12 packets captured
      50 packets received by filter
      0 packets dropped by kernel
      root@observium:/opt/observium# netstat -anp | grep 514
      udp        0      0 0.0.0.0:514             0.0.0.0:*                           674119/rsyslogd     
      root@observium:/opt/observium# ps -edaf | grep syslog
      root      674119       1  0 13:19 ?        00:00:01 /usr/sbin/rsyslogd -n -iNONE
      root      674131  674119  0 13:19 ?        00:00:06 php /opt/observium/syslog.php
       
      root@observium:/opt/observium# service rsyslog status
      ● rsyslog.service - System Logging Service
           Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
           Active: active (running) since Fri 2023-10-13 13:19:33 CEST; 39min ago
      TriggeredBy: ● syslog.socket
             Docs: man:rsyslogd(8)
                   https://www.rsyslog.com/doc/
         Main PID: 674119 (rsyslogd)
            Tasks: 6 (limit: 154018)
           Memory: 47.9M
           CGroup: /system.slice/rsyslog.service
                   ├─674119 /usr/sbin/rsyslogd -n -iNONE
                   └─674131 php /opt/observium/syslog.php
      Oct 13 13:19:33 observium systemd[1]: Starting System Logging Service...
      Oct 13 13:19:33 observium rsyslogd[674119]: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd.  [v8.2001.0]
      Oct 13 13:19:33 observium systemd[1]: Started System Logging Service.
      Oct 13 13:19:33 observium rsyslogd[674119]: [origin software="rsyslogd" swVersion="8.2001.0" x-pid="674119" x-info="https://www.rsyslog.com"] start

       

       

       

      How I can debug if is observium that is not processing the syslog, or if is something related to the linux server?

       

      Thanks,

      EM

      Attachments

        1. 30-observium.conf
          1 kB
        2. observium-syslog.png
          observium-syslog.png
          109 kB
        3. rsyslog.conf
          2 kB
        4. syslog-exos1.png
          syslog-exos1.png
          370 kB

        Activity

          [OBS-4637] Syslog for EXOS in 13056

          Docs fixed.

          landy Mike Stupalov added a comment - Docs fixed.

          Oh... no, seems as in doc this ruleset ended up there by mistake.

          By default it should be exactly as (without any filters):

          # observium RuleSets
          ruleset(name="observium") {
             action(type="omprog"
                    binary="/opt/observium/syslog.php"
                    template="observium")
             stop
          }

          landy Mike Stupalov added a comment - Oh... no, seems as in doc this ruleset ended up there by mistake. By default it should be exactly as (without any filters): # observium RuleSets ruleset(name="observium") { action(type="omprog" binary="/opt/observium/syslog.php" template="observium") stop }

          Hi @Mike,

          For the ruleset, we just followed the doc of observium for rsyslog 8.x, https://docs.observium.org/syslog/

          Add the following to /etc/rsyslog.d/30-observium.conf

          We configured your new suggestion and now it works, observium is processing correctly the syslog for all devices:

          ruleset(name="observium"){
            if ( $syslogseverity <= '7' ) then {
              action(type="omprog" binary="/opt/observium/syslog.php" template="observium")
            }
          }

          Perhaps is need it to update the online doc?

           

          Thanks @Mike.

          Greetings.

          EM

          emiliomejias Emilio MEJIAS added a comment - Hi @Mike, For the ruleset, we just followed the doc of observium for rsyslog 8.x, https://docs.observium.org/syslog/ Add the following to /etc/rsyslog.d/ 30 -observium.conf We configured your new suggestion and now it works, observium is processing correctly the syslog for all devices: ruleset(name= "observium" ){ if ( $syslogseverity <= '7' ) then { action(type= "omprog" binary= "/opt/observium/syslog.php" template= "observium" ) } } Perhaps is need it to update the online doc?   Thanks @Mike. Greetings. EM

          As I see, you created own ruleset with if case:

          ruleset(name="observium") {
              if $syslogseverity <= '7' then action(type="omprog" binary="/opt/observium/syslog.php" template="observium")
              stop
          }
          

          I'm not a big expert of rsyslog, but based on google examples, you should use something like this:

          ruleset(name="observium"){
            if ( $syslogseverity <= '7' ) then {
              action(type="omprog" binary="/opt/observium/syslog.php" template="observium")
            }
          }
          

          anyway.. try remove your own ruleset and use our default - if the problem disappears, then the problem is these ruleset.

          landy Mike Stupalov added a comment - As I see, you created own ruleset with if case: ruleset(name="observium") { if $syslogseverity <= '7' then action(type="omprog" binary="/opt/observium/syslog.php" template="observium") stop } I'm not a big expert of rsyslog, but based on google examples, you should use something like this: ruleset(name="observium"){ if ( $syslogseverity <= '7' ) then { action(type="omprog" binary="/opt/observium/syslog.php" template="observium") } } anyway.. try remove your own ruleset and use our default - if the problem disappears, then the problem is these ruleset.

          Hi Mike,

          show your versions: 
           
          root@observium:/opt/observium# ./discovery.php -VV
          Observium 23.11.13153
          #####  Software versions  #####
           o OS                   Linux 5.4.0-77-generic [amd64] (Ubuntu 20.04)
           o Apache               2.4.41
           o PHP                  7.4.3-4ubuntu2.19 (OPcache: DISABLED)
           o Python               3.8.10
           o MySQL                8.0.35-0ubuntu0.20.04.1 (extension: mysqli 7.4.3-4ubuntu2.19)
           o SNMP                 NET-SNMP 5.8
           o RRDtool              1.7.2 (rrdcached 1.7.2: unix:/var/run/rrdcached.sock)
           o Fping                4.2 (IPv4 and IPv6)
           o Fetch                cURL 7.68.0 (OpenSSL/1.1.1f, LibZ 1.2.11, LibIDN 2.2.0)
          #####  Memory Limit  #####
           o PHP                  Unlimited
          #####  DB info  #####
           o DB schema            491
           o MySQL binlog         ON
           o MySQL mode           NO_ENGINE_SUBSTITUTION
          #####  Charset info  #####
           o PHP                  UTF-8
           o MySQL                utf8mb3
          #####  Timezones info  #####
           o Date                 Friday, 10-Nov-23 11:09:13 CET
           o PHP                  +01:00
           o MySQL                +01:00
           
          root@observium:/opt/observium# rsyslogd -v
          rsyslogd  8.2001.0 (aka 2020.01) compiled with:
          	PLATFORM:				x86_64-pc-linux-gnu
          	PLATFORM (lsb_release -d):		
          	FEATURE_REGEXP:				Yes
          	GSSAPI Kerberos 5 support:		Yes
          	FEATURE_DEBUG (debug build, slow code):	No
          	32bit Atomic operations supported:	Yes
          	64bit Atomic operations supported:	Yes
          	memory allocator:			system default
          	Runtime Instrumentation (slow code):	No
          	uuid support:				Yes
          	systemd support:			Yes
          	Config file:				/etc/rsyslog.conf
          	PID file:				/run/rsyslogd.pid
          	Number of Bits in RainerScript integers: 64
          See https://www.rsyslog.com for more information.

          Attached /etc/rsyslog.d/30-observium.conf and /etc/rsyslog.conf.

          For syslog.php, I ran for 10minutes and no echo and errors:

          try run syslog.php manually from cmd: 
           
           
          root@observium:/opt/observium# time ./syslog.php 
          ^C
           
          real	9m46.256s
          user	0m0.109s
          sys	0m0.054s
          root@observium:/opt/observium#

           

          Do you think enabling the syslog debug can provide info for certain devices that are not been processed?

          Enable Syslog DEBUG
          Enable store RAW syslog lines into logs/debug.log file. Make sure that logs dir and debug.log file have write permission for your syslog server user. For example in Ubuntu rsyslog uses user syslog, add write permission for it: `sudo chmod o+w /opt/observium/logs/debug.log`
          

           

          Attached the current config for syslog in UI page.

           

          Thanks,

          EM

          emiliomejias Emilio MEJIAS added a comment - Hi Mike, show your versions:   root @observium :/opt/observium# ./discovery.php -VV Observium 23.11 . 13153 ##### Software versions ##### o OS Linux 5.4 . 0 - 77 -generic [amd64] (Ubuntu 20.04 ) o Apache 2.4 . 41 o PHP 7.4 . 3 -4ubuntu2. 19 (OPcache: DISABLED) o Python 3.8 . 10 o MySQL 8.0 . 35 -0ubuntu0. 20.04 . 1 (extension: mysqli 7.4 . 3 -4ubuntu2. 19 ) o SNMP NET-SNMP 5.8 o RRDtool 1.7 . 2 (rrdcached 1.7 . 2 : unix:/var/run/rrdcached.sock) o Fping 4.2 (IPv4 and IPv6) o Fetch cURL 7.68 . 0 (OpenSSL/ 1.1 .1f, LibZ 1.2 . 11 , LibIDN 2.2 . 0 ) ##### Memory Limit ##### o PHP Unlimited ##### DB info ##### o DB schema 491 o MySQL binlog ON o MySQL mode NO_ENGINE_SUBSTITUTION ##### Charset info ##### o PHP UTF- 8 o MySQL utf8mb3 ##### Timezones info ##### o Date Friday, 10 -Nov- 23 11 : 09 : 13 CET o PHP + 01 : 00 o MySQL + 01 : 00   root @observium :/opt/observium# rsyslogd -v rsyslogd 8.2001 . 0 (aka 2020.01 ) compiled with: PLATFORM: x86_64-pc-linux-gnu PLATFORM (lsb_release -d): FEATURE_REGEXP: Yes GSSAPI Kerberos 5 support: Yes FEATURE_DEBUG (debug build, slow code): No 32bit Atomic operations supported: Yes 64bit Atomic operations supported: Yes memory allocator: system default Runtime Instrumentation (slow code): No uuid support: Yes systemd support: Yes Config file: /etc/rsyslog.conf PID file: /run/rsyslogd.pid Number of Bits in RainerScript integers: 64 See https: //www.rsyslog.com for more information. Attached /etc/rsyslog.d/30-observium.conf and /etc/rsyslog.conf. For syslog.php, I ran for 10minutes and no echo and errors: try run syslog.php manually from cmd:     root @observium :/opt/observium# time ./syslog.php ^C   real 9m46.256s user 0m0.109s sys 0m0.054s root @observium :/opt/observium#   Do you think enabling the syslog debug can provide info for certain devices that are not been processed? Enable Syslog DEBUG Enable store RAW syslog lines into logs/debug.log file. Make sure that logs dir and debug.log file have write permission for your syslog server user. For example in Ubuntu rsyslog uses user syslog, add write permission for it: `sudo chmod o+w /opt/observium/logs/debug.log`   Attached the current config for syslog in UI page.   Thanks, EM

          Hi, not sure..

          • show your versions:

            ./discovery.php -VV

          • attach rsyslog observium related conf: /etc/rsyslog.d/30-observium.conf
          • try run syslog.php manually from cmd:

            ./syslog.php

            (normally it's should run without any echo and errors). not sure why your program was terminated.

          landy Mike Stupalov added a comment - Hi, not sure.. show your versions: ./discovery.php -VV attach rsyslog observium related conf: /etc/rsyslog.d/30-observium.conf try run syslog.php manually from cmd: ./syslog.php (normally it's should run without any echo and errors). not sure why your program was terminated.

          Hello,

          Do you have any news?

          Do you know how I can debug why observium is not processing the syslog he is receiving?

          observium rsyslogd[2765196]: child process (pid 1085578) terminated by signal 1 [v8.2001.0]
          observium rsyslogd[2765196]: omprog: program '/opt/observium/syslog.php' (pid 1085578) terminated; will be restarted [v8.2001.0 try https://www.rsyslog.com/e/2119 ]
          observium rsyslogd[2765196]: action 'action-1-omprog' suspended (module 'omprog'), retry 0. There should be messages before this one giving the reason for suspension. [v8.20>
          

          emiliomejias Emilio MEJIAS added a comment - Hello, Do you have any news? Do you know how I can debug why observium is not processing the syslog he is receiving? observium rsyslogd[ 2765196 ]: child process (pid 1085578 ) terminated by signal 1 [v8. 2001.0 ] observium rsyslogd[ 2765196 ]: omprog: program '/opt/observium/syslog.php' (pid 1085578 ) terminated; will be restarted [v8. 2001.0 try https: //www.rsyslog.com/e/2119 ] observium rsyslogd[ 2765196 ]: action 'action-1-omprog' suspended (module 'omprog' ), retry 0 . There should be messages before this one giving the reason for suspension. [v8. 20 >

          General questions and device support can be discussed in our Discord channel, click here to join.


          Please make and attach additional information about the device:

          • full snmp dump from device:

            snmpwalk -v2c -c <community> -t 3 -Cc --hexOutputLength=0 -Ih -ObentxU <hostname> .1 > myagent.snmpwalk
            snmpwalk -v2c -c <community> -t 3 -Cc --hexOutputLength=0 -Ih -ObentxU <hostname> .1.3.6.1.4.1 >> myagent.snmpwalk

            If device not support SNMP version 2c, replace -v2c with -v1.

          • If you have problems with discovery or poller processes, please do and attach these debugs:

            ./discovery.php -d -h <device>
            ./poller.php -d -h <device>

          • additionally attach device and/or vendor specific MIB files

          This comment is added automatically.

          bot Observium Bot added a comment - General questions and device support can be discussed in our Discord channel, click here to join . Please make and attach additional information about the device: full snmp dump from device: snmpwalk -v2c -c <community> -t 3 -Cc --hexOutputLength=0 -Ih -ObentxU <hostname> .1 > myagent.snmpwalk snmpwalk -v2c -c <community> -t 3 -Cc --hexOutputLength=0 -Ih -ObentxU <hostname> .1.3.6.1.4.1 >> myagent.snmpwalk If device not support SNMP version 2c, replace -v2c with -v1. If you have problems with discovery or poller processes, please do and attach these debugs: ./discovery.php -d -h <device> ./poller.php -d -h <device> additionally attach device and/or vendor specific MIB files This comment is added automatically.

          People

            landy Mike Stupalov
            emiliomejias Emilio MEJIAS
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: