Uploaded image for project: 'Observium'
  1. Observium
  2. OBS-5317

Polling Times Significantly Increased After Upgrade from 26.7.14966 to 26.10.15295

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • None
    • None
    • Discovery, Poller
    • None

    Description

      Hello,

      We recently updated our Observium installation from 26.7.14966 to 26.10.15295 and noticed an impact to the polling times for our devices, both in our regular 5 minute polling cycles and in our quarter-daily discovery cycles.

      Pre-upgrade our total polling times averaged about 180 seconds. After the upgrade our polling times are now averaging about 550 seconds. No devices have been added or removed between these two time periods. Some of the outliers for long poll times are listed below:

      • mgmt00.auz1: ~150 seconds before upgrade to ~550 seconds after
      • mgmt01.auz1: ~90 seconds before upgrade to ~190 seconds after
      • mgmt02.auz1: ~80 seconds before upgrade to ~180 seconds after
      • mgmt01.dca1: ~130 seconds before upgrade to ~190 seconds after
      • mgmt03.dca1: ~70 seconds before upgrade to ~80 seconds after

      I've run the debug poll command on two of these devices, mgmt00.auz1 & mgmt03.dca1 and for both devices the longest section was the port polling.

      mgmt00.auz1:

       

      CMD[/usr/bin/snmpbulkwalk -Cr'1' -t '4' -v3 -l 'authPriv' -n '' -x *** -X *** -a *** -A *** -u *** -Pud -Ir -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'mgmt00.auz1':'161' ifEntry]
       
      CMD EXITCODE[0]
      CMD RUNTIME[213.7908s]
       
      --- output cut ---
       
      CMD[/usr/bin/snmpbulkwalk -Cr'1' -t '4' -v3 -l 'authPriv' -n '' -x *** -X *** -a *** -A *** -u *** -Pud -Ir -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'mgmt00.auz1.':'161' ifXEntry]
       
      CMD EXITCODE[0]
      CMD RUNTIME[193.5658s]

       

       

      mgmt03.dca1

       

      CMD[/usr/bin/snmpbulkwalk -Cr'2' -v3 -l 'authPriv' -n '' -x *** -X *** -a *** -A *** -u *** -Pud -Ir -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'mgmt03.dca1':'161' ifEntry]
       
      CMD EXITCODE[0]
      CMD RUNTIME[41.2908s]
       
      --- output cut ---
       
      CMD[/usr/bin/snmpbulkwalk -Cr'2' -v3 -l 'authPriv' -n '' -x *** -X *** -a *** -A *** -u *** -Pud -Ir -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'mgmt03.dca1':'161' ifXEntry]
       
      CMD EXITCODE[0]
      CMD RUNTIME[38.7049s]

       

       

      Additionally we also started to receive the following message in the Event Log for Observium after the upgrade when we had not received it at all prior to the upgrade:

      ERROR! snmpbulkwalk exit by timeout. If the device responds to other SNMP requests, try to decrease SNMP Max Repetitions on the device properties page or set to 0 to not use bulk snmp commands.

      We attempted to reduce the maximum repetitions from the default (5) down to 2, and even 1, but that did not stop the error from occurring.

      Can you please advise on what we can do to troubleshoot and resolve these significantly increased polling times? discovery.php -VV is attached in the Environment section of the form.

      Attachments

        Activity

          People

            landy Mike Stupalov
            wesgeorge Wes George
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: