Uploaded image for project: 'Observium'
  1. Observium
  2. OBS-5135

Device Polling & Discovery stuck on several devices

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Major
    • None
    • Enterprise Edition
    • Discovery, Poller
    • None

    Description

      For some time now, polling and discovery seem to be stuck on several devices.

       

      When I run this command:

      /opt/observium/poller.php -h merit.fw.prod.ds.icorr.io

      or this

      /opt/observium/discovery.php -h merit.fw.prod.ds.icorr.io

       

      I get the following message

      /opt/observium/poller.php -h merit.fw.prod.ds.icorr.io  ___   _                              _
       / _ \ | |__   ___   ___  _ __ __   __(_) _   _  _ __ ___
      | | | || '_ \ / __| / _ \| '__|\ \ / /| || | | || '_ ` _ \
      | |_| || |_) |\__ \|  __/| |    \ V / | || |_| || | | | | |
       \___/ |_.__/ |___/ \___||_|     \_/  |_| \__,_||_| |_| |_|
                                Observium Professional 25.8.14182
                                        https://www.observium.org#####  Starting polling run at 2025-11-18 15:22:55  ##########  merit.fw.prod.ds.icorr.io [3026]  ##### o OS                   asa
       o OS Group             cisco
       o Last poll duration   51.91 seconds
       o Last Polled          2025-11-05 08:56:55
       o SNMP Version         v2c
       o Device status        Device is reachable by SNMP (204.57ms)    ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣀⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣠⡶⠋⠁⠙⣦⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      ⠀⣠⡶⠛⠛⠲⣤⣀⠀⠀⠀⠀⡀⠀⠀⠀⠀⠀⠀⠀⣠⠄⣠⣾⣡⠀⠀⠀⠀⠸⡆⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      ⢰⡏⠀⠀⠀⠀⠈⣿⣷⣤⡐⣄⣽⡾⠧⠤⠤⣤⣤⣾⣿⣾⣿⣯⣀⡀⠀⠀⠀⢠⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      ⠀⢷⡀⠀⠀⠀⠀⣿⣿⣿⣿⣿⣯⣤⣤⡀⣠⣈⣹⣽⣿⣿⣿⣿⣷⣦⣄⠀⢀⡾⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      ⠀⠈⠻⣆⠀⠀⠀⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡟⠀⣠⠞⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      ⠀⠀⠀⠈⢳⣆⠘⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣧⣾⣷⣶⣤⣀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      ⠀⠀⠀⠀⠀⣘⣿⣿⣿⣿⣿⣟⣿⣿⣿⣿⣿⣿⣿⣿⢡⡼⣿⣿⣿⣿⣿⣿⣿⣿⣿⣷⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      ⠀⠀⠀⠀⣾⣿⣿⣿⣿⣿⡏⣯⡇⣿⣿⣿⡏⣿⣿⣿⣜⣛⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡷⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      ⠀⠀⠀⠀⣿⣿⣿⣿⣿⣿⣿⣭⣾⣿⣿⣿⣇⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡿⠃⠀⠘⢷⣄⠀⠀⠀⠀⠀⠀⠀⠀
      ⠀⠀⠀⢠⣿⣿⡏⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡏⢿⣿⣿⡿⠁⠀⠀⠀⠸⣿⣷⡀⠀⠀⠀⠀⠀⠀
      ⠀⠀⠀⢸⣿⣿⡇⠈⢻⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠻⣿⣿⡟⢠⣿⣿⡟⠁⠀⠀⠀⠀⠙⣻⣿⣿⣆⠀⠀⠀⠀⠀
      ⠀⠀⠀⠈⣿⣿⣷⡈⠛⠿⣿⣿⡆⠻⢿⣿⣽⣿⣯⣿⠿⠃⠀⠻⠛⣰⣿⣿⠏⠀⠀⠀⠀⢀⣙⣿⣿⣿⣿⣿⣧⡀⠀⠀⠀
      ⠀⠀⠀⠀⢿⠈⠙⠻⠷⠤⠄⢉⡁⠀⠀⠙⢉⣵⣌⠀⠀⠀⣀⠄⠘⠛⠋⠁⢀⣀⣤⣤⣭⣭⣽⣿⣿⣿⣿⣿⣿⣷⡀⠀⠀
      ⠀⠀⠀⢀⣿⡄⠀⠀⠀⠀⠀⠀⠉⠓⢦⣄⡉⠉⢉⣩⠶⠋⠁⠀⠀⠀⣠⣾⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣷⡀⠀
      ⠀⠀⠀⣼⣿⣿⣷⣶⣶⣆⣀⡀⠀⠀⠀⠀⠉⠉⠉⠀⠀⠀⠀⠀⣠⣾⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣧⠀
      ⠀⠀⠀⣿⣿⣿⣿⣿⣿⣿⣥⣄⡀⠀⠀⠀⠀⠀⠀⠀⠀⣸⣦⣾⣿⣿⣿⣿⣿⣿⡿⠿⠿⡟⠿⣿⣟⡿⢿⡛⠀⣘⣿⣿⡆
      ⠀⠀⠀⢿⣿⣿⠿⠿⠿⠿⢿⣿⣿⣷⣿⡆⠀⠀⣶⣶⣾⡿⠛⠛⠛⠛⠻⣿⣿⣿⡟⠉⠀⠀⠀⠈⠁⠁⢀⢀⠲⣬⣿⣿⡇
      ⠀⠀⠀⠘⡏⠳⣤⡄⠀⡄⠀⠀⠈⠙⢿⣧⠀⢀⣿⣿⠏⠀⠀⠀⠀⢀⣠⣿⣿⠋⠀⠀⠀⠀⠀⠀⠀⠀⠈⣯⣷⣿⣿⣿⡇
      ⠀⠀⠀⠀⢿⠀⠀⠉⠳⣇⠀⠀⠀⠀⠀⢻⠀⢸⣿⣿⣆⢠⠀⡆⣰⡿⠟⠉⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⡾⣿⣿⣿⣿⣿⡇
      ⠀⠀⠀⠀⠘⣧⠀⠀⠀⠹⣷⣤⣄⠀⡈⣾⡆⢸⡿⠋⠿⠾⠴⠷⠋⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣠⠞⠉⣶⣾⣿⣿⣿⣿⠀
      ⠀⠀⠀⠀⠀⠈⠳⣄⠀⠀⠀⠉⠻⣿⣿⣿⣇⠸⡷⠀⠀⠀⢀⣠⢄⣠⣶⣤⣤⣤⡤⠤⠖⠛⠉⠀⠠⣤⣼⣿⣿⣿⣿⡏⠀
      ⠀⠀⠀⠀⠀⠀⠀⠉⢻⣷⠶⠒⠀⠀⠁⠈⢿⡄⣷⣆⣤⣶⡿⠛⠛⠛⠉⠉⠉⠁⠀⠀⠀⠀⠀⠀⠀⠈⢿⣿⣿⣿⣿⠇⠀
      ⠀⠀⠀⠀⠀⠀⠀⠀⠘⣏⠀⠀⠀⠀⠀⠀⠘⣧⣿⣿⡿⢋⡤⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠰⣾⣿⣿⣿⣿⠀⠀
      ⠀⠀⠀⠀⠀⠀⠀⠀⠀⢻⡄⠀⠀⠀⠀⠀⠀⢸⣿⣿⣶⣫⠆⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡀⢦⣵⣿⣿⣿⣿⠇⠀⠀
      ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢻⣆⠀⠀⠀⠀⠀⠀⣿⣿⣿⠇⠀⣀⠀⠀⠀⠀⠀⠀⠀⠀⠳⠦⣴⣿⣶⣿⣿⣿⣿⡿⠀⠀⠀
      ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠛⢷⣄⠀⠀⠀⠀⢹⣿⣿⣾⣿⣋⣤⡄⠀⠀⠀⠀⠀⠀⠀⠠⣈⠻⣿⣿⣿⣿⣿⡇⠀⠀⠀
      ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠉⠛⣦⣤⣤⣾⣿⣿⣿⣿⣿⣯⡴⠂⣀⡀⠀⡀⡀⡀⢠⣬⣻⣿⣿⣿⣿⡿⠀⠀⠀⠀
      ⠀⠀⠀⠀⠀⠀⠀⢀⣀⣀⣀⣀⣠⡴⠊⠉⠀⠈⢹⠿⣿⣿⣿⣿⣿⣷⣾⣿⠾⡿⡿⠿⣷⣦⣿⣿⣿⣿⣿⡟⠁⠀⠀⠀⠀
      ⠀⠀⠀⠀⠀⠀⠀⠛⠷⠦⠤⠤⠴⠶⠶⠶⠶⠚⠉⠀⠀⠈⣉⣩⠽⠟⠋⠁⠀⠀⣁⣠⠿⠛⠋⠉⠉⠉⠁⠀⠀⠀⠀⠀⠀
      ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠉⠉⠉⠉⠉⠉⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      Oops! Something went wrong! You may want to report this to the Observium developers.fwrite(): Argument #1 ($stream) must be of type resource, null given in /opt/observium/includes/rrdtool.inc.php:330Stack trace:#0: (unknown function) called at [/opt/observium/includes/rrdtool.inc.php:330]
      325: logfile('rrd.log', "RRD " . $exec_status['stderr'] . ", CMD: $cmd");
      326: }
      327: } else {
      328: // FIXME, need add check if pipes exist
      329: $start = microtime(TRUE);
      330: fwrite($rrd_pipes[0], $cmd . "\n");
      331: // FIXME, complete not sure why sleep required here.. @mike
      332: usleep(1000);
      333:
      334: $stdout  = trim(stream_get_contents($rrd_pipes[1]));
      335: $stderr  = trim(stream_get_contents($rrd_pipes[2]));#1: fwrite called at [/opt/observium/includes/rrdtool.inc.php:330]
      325: logfile('rrd.log', "RRD " . $exec_status['stderr'] . ", CMD: $cmd");
      326: }
      327: } else {
      328: // FIXME, need add check if pipes exist
      329: $start = microtime(TRUE);
      330: fwrite($rrd_pipes[0], $cmd . "\n");
      331: // FIXME, complete not sure why sleep required here.. @mike
      332: usleep(1000);
      333:
      334: $stdout  = trim(stream_get_contents($rrd_pipes[1]));
      335: $stderr  = trim(stream_get_contents($rrd_pipes[2]));#2: rrdtool called at [/opt/observium/includes/rrdtool.inc.php:553]
      548: $step = $options['step'] ?? $config['rrd']['step'];
      549:
      550: // Create DS parameter based on the definition
      551: $ds = rrdtool_def_ds('create', $definition, $index, $options);
      552:
      553: return rrdtool('create', $fsfilename, implode(' ', $ds) . " --step $step $rra");
      554:
      555: }
      556:
      557: /**
      558: * Updates an rrd database at $filename using $options#3: rrdtool_create_ng called at [/opt/observium/includes/rrdtool.inc.php:597]
      592:
      593: $fsfilename = get_rrd_path($device, $filename);
      594:
      595: // Create the file if missing (if we have permission to create it)
      596: if ($create) {
      597: rrdtool_create_ng($device, $type, $index, $options);
      598: }
      599: // Update DSes when requested
      600: if (isset($options['update_max']) || isset($options['update_min'])) {
      601: print_debug_vars($options);
      602: rrdtool_update_ds($device, $type, $index, $options);#4: rrdtool_update_ng called at [/opt/observium/includes/polling/functions.inc.php:213]
      208: log_event('Device status changed to Down (' . $device['status_type'] . ' -> ' . $status_type . ')', $device, 'device', $device['device_id'], 3);
      209: }
      210: $device['status_type'] = $status_type;
      211: }
      212:
      213: rrdtool_update_ng($device, 'status', ['status' => $status]);
      214: //print_vars(rrdtool_export_ng($device, 'status'));
      215:
      216: if (!$attribs['ping_skip']) {
      217: // Ping response RRD database.
      218: rrdtool_update_ng($device, 'ping', ['ping' => ($device['status_pingable'] ?: 'U')]);#5: poll_device called at [/opt/observium/poller.php:211]
      206: $params[] = $config['poller_id'];
      207: }
      208:
      209: foreach (dbFetchColumn($query, $params) as $device_id) {
      210: $device = dbFetchRow("SELECT * FROM `devices` WHERE `device_id` = ?", [ $device_id ]);
      211: poll_device($device, $options);
      212: $polled_devices++;
      213: }
      214:
      215: $poller_time = elapsed_time($poller_start, 4);
      216:
      

       

       

      /opt/observium/discovery.php -h merit.fw.prod.ds.icorr.io  ___   _                              _
       / _ \ | |__   ___   ___  _ __ __   __(_) _   _  _ __ ___
      | | | || '_ \ / __| / _ \| '__|\ \ / /| || | | || '_ ` _ \
      | |_| || |_) |\__ \|  __/| |    \ V / | || |_| || | | | | |
       \___/ |_.__/ |___/ \___||_|     \_/  |_| \__,_||_| |_| |_|
                                Observium Professional 25.8.14182
                                        https://www.observium.org#####  Starting discovery run at 2025-11-18 15:24:32  ##########  merit.fw.prod.ds.icorr.io [3026]  ##### o OS Type              asa
       o OS Group             cisco
       o SNMP Version         v2c
       o Last discovery       2025-11-05 06:15:18
       o Last duration        79.13 seconds#####  Module Start: os  #####+-------------+---------------------------------------------------+
      | OID         |                                                   |
      +-------------+---------------------------------------------------+
      | sysDescr    | Cisco Adaptive Security Appliance Version 9.14(2) |
      | sysObjectID | .1.3.6.1.4.1.9.1.2663                             |
      +-------------+---------------------------------------------------+ o Re-Detect OS matched (asa: Cisco ASA):
      +-------------+-------------------------------------+---------------------------------------------------+
      | OID         | Matched definition                  |                                                   |
      +-------------+-------------------------------------+---------------------------------------------------+
      | sysObjectID | .1.3.6.1.4.1.9.1                    | .1.3.6.1.4.1.9.1.2663                             |
      | sysDescr    | /Cisco Adaptive Security Appliance/ | Cisco Adaptive Security Appliance Version 9.14(2) |
      +-------------+-------------------------------------+---------------------------------------------------+
       o Duration             0.4441s#####  Module Start: mibs  ##### o MIBs discovered      <empty>
      +----------+--------------------+-----------+----------------------------------+
      | Identity | MIB                | Source    | Status                           |
      +----------+--------------------+-----------+----------------------------------+
      |          | MPLS-L3VPN-STD-MIB | Discovery | MIB defined for os but not found |
      |          | MPLS-VPN-MIB       | Discovery | MIB defined for os but not found |
      |          | CISCO-FIREWALL-MIB | Discovery | MIB defined for os but not found |
      +----------+--------------------+-----------+----------------------------------+
       o Duration             2.8253s#####  Module Start: vrf  ##### oMPLS-L3VPN-STD-MIB
       oMPLS-VPN-MIB
       oCISCO-VRF-MIB o Duration             2.116s#####  Module Start: ports  ##### oCaching OIDs ifDescr ifAlias ifName ifType ifOperStatus
       o Caching DB           15 ports
       o Discovering ports    ...............
      +---------+--------------------------------+------------------+----------------------+----------------+-------------+---------+
      | ifIndex | ifDescr                        | ifName           | ifAlias              | ifType         | Oper Status | Ignored |
      +---------+--------------------------------+------------------+----------------------+----------------+-------------+---------+
      | 2       | Adaptive Security Appliance... | Internal-Data0/0 | ...                  | ethernetCsmacd | up          | no      |
      | 3       | Adaptive Security Appliance... | management       | ...                  | ethernetCsmacd | up          | no      |
      | 4       | Adaptive Security Appliance... | nlp_int_tap      | ...                  | ethernetCsmacd | up          | no      |
      | 5       | Adaptive Security Appliance... | Corr_Int         | Corrugator network   | other          | up          | no      |
      | 6       | Adaptive Security Appliance... | DMZ_Int          | DMZ network          | other          | up          | no      |
      | 7       | Adaptive Security Appliance... | Remote_Int       | ...                  | other          | up          | no      |
      | 8       | Adaptive Security Appliance... | Ethernet1/4      | ...                  | other          | down        | no      |
      | 9       | Adaptive Security Appliance... | WLAN_Int         | BHS Corrugator WL... | other          | down        | no      |
      | 10      | Adaptive Security Appliance... | Ethernet1/6      | ...                  | other          | down        | no      |
      | 11      | Adaptive Security Appliance... | Ethernet1/7      | ...                  | other          | down        | no      |
      | 12      | Adaptive Security Appliance... | OnSite_Int       | Customer network     | other          | up          | no      |
      | 13      | Adaptive Security Appliance... | LWL_Int          | LWL transfer network | other          | down        | no      |
      | 14      | Adaptive Security Appliance... | Ethernet1/10     | ...                  | other          | down        | no      |
      | 15      | Adaptive Security Appliance... | Ethernet1/11     | ...                  | other          | down        | no      |
      | 16      | Adaptive Security Appliance... | Ethernet1/12     | ...                  | other          | down        | no      |
      +---------+--------------------------------+------------------+----------------------+----------------+-------------+---------+
       o Duration             3.8011s#####  Module Start: ports-stack  ##### o Duration             0.5962s#####  Module Start: vlans  ##### oCISCO-VTP-MIB oQ-BRIDGE-MIB
       o Duration             0.2371s#####  Module Start: oids  #####
       o Duration             0.0006s#####  Module Start: ip-addresses  ##### oIP-MIB
          ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣀⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣠⡶⠋⠁⠙⣦⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      ⠀⣠⡶⠛⠛⠲⣤⣀⠀⠀⠀⠀⡀⠀⠀⠀⠀⠀⠀⠀⣠⠄⣠⣾⣡⠀⠀⠀⠀⠸⡆⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      ⢰⡏⠀⠀⠀⠀⠈⣿⣷⣤⡐⣄⣽⡾⠧⠤⠤⣤⣤⣾⣿⣾⣿⣯⣀⡀⠀⠀⠀⢠⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      ⠀⢷⡀⠀⠀⠀⠀⣿⣿⣿⣿⣿⣯⣤⣤⡀⣠⣈⣹⣽⣿⣿⣿⣿⣷⣦⣄⠀⢀⡾⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      ⠀⠈⠻⣆⠀⠀⠀⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡟⠀⣠⠞⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      ⠀⠀⠀⠈⢳⣆⠘⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣧⣾⣷⣶⣤⣀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      ⠀⠀⠀⠀⠀⣘⣿⣿⣿⣿⣿⣟⣿⣿⣿⣿⣿⣿⣿⣿⢡⡼⣿⣿⣿⣿⣿⣿⣿⣿⣿⣷⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      ⠀⠀⠀⠀⣾⣿⣿⣿⣿⣿⡏⣯⡇⣿⣿⣿⡏⣿⣿⣿⣜⣛⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡷⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      ⠀⠀⠀⠀⣿⣿⣿⣿⣿⣿⣿⣭⣾⣿⣿⣿⣇⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡿⠃⠀⠘⢷⣄⠀⠀⠀⠀⠀⠀⠀⠀
      ⠀⠀⠀⢠⣿⣿⡏⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡏⢿⣿⣿⡿⠁⠀⠀⠀⠸⣿⣷⡀⠀⠀⠀⠀⠀⠀
      ⠀⠀⠀⢸⣿⣿⡇⠈⢻⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠻⣿⣿⡟⢠⣿⣿⡟⠁⠀⠀⠀⠀⠙⣻⣿⣿⣆⠀⠀⠀⠀⠀
      ⠀⠀⠀⠈⣿⣿⣷⡈⠛⠿⣿⣿⡆⠻⢿⣿⣽⣿⣯⣿⠿⠃⠀⠻⠛⣰⣿⣿⠏⠀⠀⠀⠀⢀⣙⣿⣿⣿⣿⣿⣧⡀⠀⠀⠀
      ⠀⠀⠀⠀⢿⠈⠙⠻⠷⠤⠄⢉⡁⠀⠀⠙⢉⣵⣌⠀⠀⠀⣀⠄⠘⠛⠋⠁⢀⣀⣤⣤⣭⣭⣽⣿⣿⣿⣿⣿⣿⣷⡀⠀⠀
      ⠀⠀⠀⢀⣿⡄⠀⠀⠀⠀⠀⠀⠉⠓⢦⣄⡉⠉⢉⣩⠶⠋⠁⠀⠀⠀⣠⣾⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣷⡀⠀
      ⠀⠀⠀⣼⣿⣿⣷⣶⣶⣆⣀⡀⠀⠀⠀⠀⠉⠉⠉⠀⠀⠀⠀⠀⣠⣾⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣧⠀
      ⠀⠀⠀⣿⣿⣿⣿⣿⣿⣿⣥⣄⡀⠀⠀⠀⠀⠀⠀⠀⠀⣸⣦⣾⣿⣿⣿⣿⣿⣿⡿⠿⠿⡟⠿⣿⣟⡿⢿⡛⠀⣘⣿⣿⡆
      ⠀⠀⠀⢿⣿⣿⠿⠿⠿⠿⢿⣿⣿⣷⣿⡆⠀⠀⣶⣶⣾⡿⠛⠛⠛⠛⠻⣿⣿⣿⡟⠉⠀⠀⠀⠈⠁⠁⢀⢀⠲⣬⣿⣿⡇
      ⠀⠀⠀⠘⡏⠳⣤⡄⠀⡄⠀⠀⠈⠙⢿⣧⠀⢀⣿⣿⠏⠀⠀⠀⠀⢀⣠⣿⣿⠋⠀⠀⠀⠀⠀⠀⠀⠀⠈⣯⣷⣿⣿⣿⡇
      ⠀⠀⠀⠀⢿⠀⠀⠉⠳⣇⠀⠀⠀⠀⠀⢻⠀⢸⣿⣿⣆⢠⠀⡆⣰⡿⠟⠉⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⡾⣿⣿⣿⣿⣿⡇
      ⠀⠀⠀⠀⠘⣧⠀⠀⠀⠹⣷⣤⣄⠀⡈⣾⡆⢸⡿⠋⠿⠾⠴⠷⠋⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣠⠞⠉⣶⣾⣿⣿⣿⣿⠀
      ⠀⠀⠀⠀⠀⠈⠳⣄⠀⠀⠀⠉⠻⣿⣿⣿⣇⠸⡷⠀⠀⠀⢀⣠⢄⣠⣶⣤⣤⣤⡤⠤⠖⠛⠉⠀⠠⣤⣼⣿⣿⣿⣿⡏⠀
      ⠀⠀⠀⠀⠀⠀⠀⠉⢻⣷⠶⠒⠀⠀⠁⠈⢿⡄⣷⣆⣤⣶⡿⠛⠛⠛⠉⠉⠉⠁⠀⠀⠀⠀⠀⠀⠀⠈⢿⣿⣿⣿⣿⠇⠀
      ⠀⠀⠀⠀⠀⠀⠀⠀⠘⣏⠀⠀⠀⠀⠀⠀⠘⣧⣿⣿⡿⢋⡤⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠰⣾⣿⣿⣿⣿⠀⠀
      ⠀⠀⠀⠀⠀⠀⠀⠀⠀⢻⡄⠀⠀⠀⠀⠀⠀⢸⣿⣿⣶⣫⠆⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡀⢦⣵⣿⣿⣿⣿⠇⠀⠀
      ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢻⣆⠀⠀⠀⠀⠀⠀⣿⣿⣿⠇⠀⣀⠀⠀⠀⠀⠀⠀⠀⠀⠳⠦⣴⣿⣶⣿⣿⣿⣿⡿⠀⠀⠀
      ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠛⢷⣄⠀⠀⠀⠀⢹⣿⣿⣾⣿⣋⣤⡄⠀⠀⠀⠀⠀⠀⠀⠠⣈⠻⣿⣿⣿⣿⣿⡇⠀⠀⠀
      ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠉⠛⣦⣤⣤⣾⣿⣿⣿⣿⣿⣯⡴⠂⣀⡀⠀⡀⡀⡀⢠⣬⣻⣿⣿⣿⣿⡿⠀⠀⠀⠀
      ⠀⠀⠀⠀⠀⠀⠀⢀⣀⣀⣀⣀⣠⡴⠊⠉⠀⠈⢹⠿⣿⣿⣿⣿⣿⣷⣾⣿⠾⡿⡿⠿⣷⣦⣿⣿⣿⣿⣿⡟⠁⠀⠀⠀⠀
      ⠀⠀⠀⠀⠀⠀⠀⠛⠷⠦⠤⠤⠴⠶⠶⠶⠶⠚⠉⠀⠀⠈⣉⣩⠽⠟⠋⠁⠀⠀⣁⣠⠿⠛⠋⠉⠉⠉⠁⠀⠀⠀⠀⠀⠀
      ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠉⠉⠉⠉⠉⠉⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      Oops! Something went wrong! You may want to report this to the Observium developers.Failed opening required 'PEAR.php' (include_path='/opt/observium/libs/pear:/opt/observium/libs:.:/usr/share/php') in /opt/observium/libs/pear/Net/IPv4.php:24Stack trace:#0: (unknown function) called at [/opt/observium/libs/pear/Net/IPv4.php:24]
      19: * @license    http://www.php.net/license/3_01.txt  PHP License 3.01
      20: * @version    CVS: $Id: IPv4.php 302879 2010-08-30 06:52:41Z bate $
      21: * @link       http://pear.php.net/package/Net_IPv4
      22: */
      23:
      24: require_once 'PEAR.php';
      25:
      26: // {{{ Net_IPv4
      27:
      28: /**
      29: * Class to provide IPv4 calculations#1: include_once called at [/opt/observium/includes/autoloader.inc.php:133]
      128: //}
      129: }
      130: $full_path = $base_dir . $class_file;
      131:
      132: if ($status = is_file($full_path)) {
      133: $status = include_once($full_path);
      134: }
      135: if (defined('OBS_DEBUG') && OBS_DEBUG > 1 &&
      136: function_exists('print_message')) {
      137: print_message("%WLoad class '$class_name' from '$full_path': " . ($status ? '%gOK' : '%rFAIL'), 'console');
      138: }#2: observium_autoload called at [/opt/observium/includes/entities/ip-address.inc.php:238]
      233: * @param string $netmask
      234: *
      235: * @return string|null
      236: */
      237: function netmask2cidr($netmask) {
      238: $addr = Net_IPv4::parseAddress("1.2.3.4/$netmask");
      239: return is_intnum($addr->bitmask) ? $addr->bitmask : NULL;
      240: }
      241:
      242: /**
      243: * Convert CIDR to IPv4 netmask#3: netmask2cidr called at [/opt/observium/includes/entities/ip-address.inc.php:319]
      314: switch (get_ip_version($entry['ip'])) {
      315: case 4:
      316: if (!safe_empty($entry['prefix']) && $entry['prefix'] !== 'zeroDotZero') {
      317: $prefix = netmask2cidr($entry['prefix']);
      318: } elseif (!safe_empty($entry['mask']) && $entry['mask'] !== 'zeroDotZero') {
      319: $prefix = netmask2cidr($entry['mask']);
      320: }
      321:
      322: if (!is_intnum($prefix)) {
      323: // When gateway not exists, return 32
      324: $prefix = gateway2prefix($entry['ip'], $entry['gateway']);#4: get_ip_prefix called at [/opt/observium/includes/entities/ip-address.inc.php:165]
      160: $ifIndex = $entry['ifIndex'];
      161:
      162: switch ($ip_version) {
      163: case 'ipv4':
      164: // IPv4
      165: $prefix = get_ip_prefix($entry);
      166: if (!is_ipv4_valid($ip, $prefix)) {
      167: print_debug("Address '$ip/$prefix' skipped as invalid.");
      168: return;
      169: }
      170: $entry['prefix'] = $prefix;#5: discover_add_ip_address called at [/opt/observium/includes/discovery/ip-addresses/ip-mib.inc.php:93]
      88: $data = [
      89: 'ifIndex' => $ifIndex,
      90: 'ip'      => $ip_address,
      91: 'mask'    => $entry['ipAdEntNetMask']
      92: ];
      93: discover_add_ip_address($device, $mib, $data);
      94: }
      95:
      96: // Get IP addresses from IP-MIB (new table, both IPv4/IPv6)
      97: $flags    = OBS_SNMP_ALL ^ OBS_QUOTES_STRIP;
      98: $oid_data = snmpwalk_cache_twopart_oid($device, 'ipAddressIfIndex', [], 'IP-MIB', NULL, $flags);#6: include called at [/opt/observium/includes/include-dir-mib.inc.php:32]
      27:
      28: if (is_file($inc_file)) {
      29: print_cli_data_field("$mib ");
      30:
      31: $inc_start  = microtime(TRUE); // MIB timing start
      32: $inc_status = include($inc_file);
      33: echo(PHP_EOL);
      34:
      35: if ($include_lib && is_file($inc_dir . '.lib.php')) {
      36: // separated functions include, for exclude fatal redeclare errors
      37: include_once($inc_dir . '.lib.php');#7: include called at [/opt/observium/includes/discovery/ip-addresses.inc.php:28]
      23: //$valid['ip-addresses'] = array();
      24:
      25: $include_dir   = 'includes/discovery/ip-addresses';
      26: $include_order = 'default'; // Use MIBs from default os definitions by first!
      27:
      28: include($config['install_dir'] . "/includes/include-dir-mib.inc.php");
      29:
      30: foreach (get_device_mibs_permitted($device) as $mib) {
      31: // Detect addresses by definitions
      32: if (is_array($config['mibs'][$mib]['ip-address'])) {
      33: print_cli_data_field($mib);#8: include called at [/opt/observium/includes/discovery/functions.inc.php:842]
      837: $GLOBALS['module_stats'][$module] = [];
      838: $valid[$module]                   = [];
      839:
      840: print_cli_heading("Module Start: %R" . $module);
      841:
      842: include("includes/discovery/$module.inc.php");
      843:
      844: $GLOBALS['module_stats'][$module]['time'] = elapsed_time($m_start, 4);
      845: print_module_stats($device, $module);
      846: echo(PHP_EOL);
      847: //print_cli_heading("Module End: %R".$module."");#9: discover_device called at [/opt/observium/discovery.php:263]
      258: $total_devices++;
      259:
      260: // Additional check if device SNMPable, because during
      261: // discovery many devices (long time), some device can be switched off
      262: if ($options['h'] === 'new' || is_snmpable($device)) {
      263: $discover_status = discover_device($device, $options);
      264: } else {
      265: $string = "Device '" . $device['hostname'] . "' skipped, because switched off during runtime discovery process.";
      266: print_debug($string);
      267: logfile($argv[0] . ": $string");
      268: $discover_status = FALSE; 

      For some inexplicable reason, the force_discovery flag was also active on several devices.

      Was there an update for this in the last version?

      I tried to correct this in the database with the SQL statement:

      UPDATE devices SET force_discovery = 0 WHERE force_discovery = 1;
      Query OK, 1008 rows affected (0.02 sec)
      Rows matched: 1008  Changed: 1008  Warnings: 0 

       

      Attachments

        Activity

          People

            landy Mike Stupalov
            ds-ops DS-Operations
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: