Uploaded image for project: 'Observium'
  1. Observium
  2. OBS-3922

Failure on OS detection with Proxmox 6.3

Details

    • Library Bug
    • Resolution: Fixed
    • Minor
    • None
    • CE-21.10
    • Discovery, OS
    • None

    Description

      I have 3 identical servers running Proxmox VE 6.3 conforming a Cluster.
      Discovery.php fails to detect Node 1' OS and Proxmox and set it as Standar Linux. Also discovery takes too long to work.
      Detection of Nodes 2 and 3 works great and the running time of discovery is considerably short than node 1

      # /opt/observium/discovery.php -m os -h node1
       
        ___   _                              _
       / _ \ | |__   ___   ___  _ __ __   __(_) _   _  _ __ ___
      | | | || '_ \ / __| / _ \| '__|\ \ / /| || | | || '_ ` _ \
      | |_| || |_) |\__ \|  __/| |    \ V / | || |_| || | | | | |
       \___/ |_.__/ |___/ \___||_|     \_/  |_| \__,_||_| |_| |_|
                          Observium Community Edition 21.10.11666
                                        https://www.observium.org
       
      #####  Starting discovery run at 2021-10-26 16:26:22  #####
       
      #####  node1 [299]  #####
       
       o OS Type              linux
       o OS Group             unix
       o SNMP Version         v3
       o Last discovery       2021-10-26 16:02:11
       o Last duration        13.01 seconds
       
      #####  Module Start: os  #####
       
      +-------------+--------------------------------------------------------------------------------------------+
      | OID         |                                                                                            |
      +-------------+--------------------------------------------------------------------------------------------+
      | sysDescr    | Linux node1 5.4.78-2-pve #1 SMP PVE 5.4.78-2 (Thu, 03 Dec 2020 14:26:17 +0100) x86_64 |
      | sysObjectID | .1.3.6.1.4.1.8072.3.2.10                                                                   |
      +-------------+--------------------------------------------------------------------------------------------+
       
       o Detect OS matched (linux: Linux): 
      +------+----------------------------------------------------+--+
      | OID  | Matched definition                                 |  |
      +------+----------------------------------------------------+--+
      | file | /opt/observium/includes/discovery/os/linux.inc.php |  |
      +------+----------------------------------------------------+--+
       
       
       o Duration             13.2857s
       
      #####  node1 [299] completed discovery modules at 2021-10-26 16:26:35  #####
       
       o Discovery time       13.2907 seconds
       
      #####  Finished discovery run at 2021-10-26 16:26:35  #####
       
       o Devices Discovered   1
       o Discovery Time       13.33 secs
       o Memory usage         20MB (peak: 22.55MB)
       o MySQL Usage          Cell[2/0s] Row[4/0.001s] Rows[5/0.002s] Column[0/0s] Update[1/0.004s] Insert[1/0.004s] Delete[1/0.003s] 
       o RRDTool Usage        
      

      # /opt/observium/discovery.php -m os -h node2
       
        ___   _                              _
       / _ \ | |__   ___   ___  _ __ __   __(_) _   _  _ __ ___
      | | | || '_ \ / __| / _ \| '__|\ \ / /| || | | || '_ ` _ \
      | |_| || |_) |\__ \|  __/| |    \ V / | || |_| || | | | | |
       \___/ |_.__/ |___/ \___||_|     \_/  |_| \__,_||_| |_| |_|
                          Observium Community Edition 21.10.11666
                                        https://www.observium.org
       
      #####  Starting discovery run at 2021-10-26 16:30:55  #####
       
      #####  node2 [293]  #####
       
       o OS Type              pve
       o OS Group             unix
       o SNMP Version         v3
       o Last discovery       2021-10-26 16:01:04
       o Last duration        0.13 seconds
       
      #####  Module Start: os  #####
       
      +-------------+--------------------------------------------------------------------------------------------+
      | OID         |                                                                                            |
      +-------------+--------------------------------------------------------------------------------------------+
      | sysDescr    | Linux node2 5.4.78-2-pve #1 SMP PVE 5.4.78-2 (Thu, 03 Dec 2020 14:26:17 +0100) x86_64 |
      | sysObjectID | .1.3.6.1.4.1.8072.3.2.10                                                                   |
      +-------------+--------------------------------------------------------------------------------------------+
       
       o Re-Detect OS matched (pve: Proxmox VE): 
      +-------------------+---------------------------+--------------------------------------------------------------------------------------------+
      | OID               | Matched definition        |                                                                                            |
      +-------------------+---------------------------+--------------------------------------------------------------------------------------------+
      | sysObjectID       | .1.3.6.1.4.1.8072.3.2.10  | .1.3.6.1.4.1.8072.3.2.10                                                                   |
      | sysDescr          | /^Linux .*\d\-pve | PVE / | Linux node2 5.4.78-2-pve #1 SMP PVE 5.4.78-2 (Thu, 03 Dec 2020 14:26:17 +0100) x86_64 |
      | hrSWInstalledName | /^pve\-manager[\-_]\d/    | pve-manager-6.3-3                                                                          |
      +-------------------+---------------------------+--------------------------------------------------------------------------------------------+
       
       
       o Duration             0.1202s
       
      #####  node2 [293] completed discovery modules at 2021-10-26 16:30:55  #####
       
       o Discovery time       0.1255 seconds
       
      #####  Finished discovery run at 2021-10-26 16:30:55  #####
       
       o Devices Discovered   1
       o Discovery Time       0.156 secs
       o Memory usage         20MB (peak: 22.55MB)
       o MySQL Usage          Cell[2/0s] Row[4/0.001s] Rows[5/0.001s] Column[0/0s] Update[1/0.004s] Insert[1/0.004s] Delete[1/0.004s] 
       o RRDTool Usage  
      

      They have the same sysObjectID and sysDescr.
      SNMP config is the same on all (managed by puppet). SNMPD service was restarted several times with no change in behavior of discovery. 

      BTW: when try to debug the output of discovery I get:

      # /opt/observium/discovery.php -d -m os -h node
      DEBUG!
      DEFINITIONS loaded by: 0.0258241 ms
       
      DEFINITIONS in memory: 15.2MB
       
       
      SQL[PHP Parse error:  syntax error, unexpected '?', expecting variable (T_VARIABLE) in /opt/observium_new/libs/Doctrine/SqlFormatter/SqlFormatter.php on line 37
       
      Parse error: syntax error, unexpected '?', expecting variable (T_VARIABLE) in /opt/observium_new/libs/Doctrine/SqlFormatter/SqlFormatter.php on line 37
      
      

      Attachments

        1. discovery_node1_fixed.txt
          47 kB
        2. discovery_node1.txt
          22 kB
        3. myagent.snmpwalk
          1.44 MB
        4. Screenshot_20211027_155621.png
          Screenshot_20211027_155621.png
          191 kB
        5. Screenshot_20211027_161124.png
          Screenshot_20211027_161124.png
          276 kB

        Activity

          [OBS-3922] Failure on OS detection with Proxmox 6.3

          For future releases snmpbulkwalk complete disabled in os detect.
          Fixed in r11691.

          landy Mike Stupalov added a comment - For future releases snmpbulkwalk complete disabled in os detect. Fixed in r11691.

          Ok, this is still strange.
          By unknown reason, your server stop respond with timeout on snmpbulkwalk.
          We not seen before such troubles on any linux hosts.

          Now I can only advise set different max-rep value (ie around 30) or complete disable snmpbulk by set maxrep to 0 (as you done for node1).

          landy Mike Stupalov added a comment - Ok, this is still strange. By unknown reason, your server stop respond with timeout on snmpbulkwalk. We not seen before such troubles on any linux hosts. Now I can only advise set different max-rep value (ie around 30) or complete disable snmpbulk by set maxrep to 0 (as you done for node1).

          Ahh, no.. this is our definitions.

          landy Mike Stupalov added a comment - Ahh, no.. this is our definitions.

          Hrm, you was set maxrep 100 manually before?
          (we not use such value for linux hosts).

          I think than this is main trouble for you, remove this value for devices (default is 10 and this value is quite enough for linux hosts).

          landy Mike Stupalov added a comment - Hrm, you was set maxrep 100 manually before? (we not use such value for linux hosts). I think than this is main trouble for you, remove this value for devices (default is 10 and this value is quite enough for linux hosts).
          nico.loba Nicolas Lobariñas added a comment - - edited

          Finally worked!!. Looking on the device's event log I found this error

          ERROR! snmpbulkwalk exit by timeout. Try to decrease SNMP Max Repetitions on the device properties page or set to 0 to not use bulk snmp commands. (repeated 2 times)

          Then setting "Max Repetitions" to 0 on device properties and triggering the discovery again it set the OS to PVE.

          But still some issues, like:
          node1: no storage discovery, vendor module detect brand.
          node2: no vendor detected.

          Behavior is strange...

          nico.loba Nicolas Lobariñas added a comment - - edited Finally worked!!. Looking on the device's event log I found this error ERROR! snmpbulkwalk exit by timeout. Try to decrease SNMP Max Repetitions on the device properties page or set to 0 to not use bulk snmp commands. (repeated 2 times) Then setting "Max Repetitions" to 0 on device properties and triggering the discovery again it set the OS to PVE. But still some issues, like: node1: no storage discovery, vendor module detect brand. node2: no vendor detected. Behavior is strange...

          Thanks for the quick reply!! 
          I've upgraded PHP and Debug now works! I uploaded the output of "discovery -d -m os  -h node1"

           

           

          nico.loba Nicolas Lobariñas added a comment - Thanks for the quick reply!!  I've upgraded PHP and Debug now works! I uploaded the output of "discovery -d -m os  -h node1"    

          I still need to see discovery debug.
          Proxmox for detect os, required to fetch packages from device.
          Not sure why, but seems here packages not fetched or required package not found.

          That why I need to see debug, as minimum:

          ./discovery.php -d -m os -h node1
          

          But for fix debug trouble, update your php on host to 7.2 or 7.4 pls.
          Here you can read how upgrade php:
          https://computingforgeeks.com/how-to-install-latest-php-on-debian/

          landy Mike Stupalov added a comment - I still need to see discovery debug. Proxmox for detect os, required to fetch packages from device. Not sure why, but seems here packages not fetched or required package not found. That why I need to see debug, as minimum: ./discovery.php -d -m os -h node1 But for fix debug trouble, update your php on host to 7.2 or 7.4 pls. Here you can read how upgrade php: https://computingforgeeks.com/how-to-install-latest-php-on-debian/

          People

            landy Mike Stupalov
            nico.loba Nicolas Lobariñas
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: