Details

    • Bug
    • Resolution: Fixed
    • Major
    • None
    • Professional Edition
    • Alerting, Graphs
    • None

    Description

      Hello,

      We are currently running version 25.2.13910 (rolling) and have encountered an issue with our BGP peers. For the majority of our peers, the prefix count is not displayed correctly—only a small number (3 out of 90) show the proper prefix information.

       

       

      Initially, we suspected the issue might be related to VRF configuration, but we found that the three peers displaying correct prefix counts are also configured within VRFs.

      Furthermore, the issue is occurring across devices from different vendors and models, and the displayed prefix count on the affected peers is showing as “-nan” (Not a Number). We have also observed that only the newly added peers are working correctly and showing the proper prefix counts.

      Could you please investigate this matter and advise on a solution?

      Thank you,

      Attachments

        1. poller.txt
          2.97 MB
        2. image-2025-02-25-12-41-00-503.png
          image-2025-02-25-12-41-00-503.png
          143 kB
        3. image-2025-02-24-15-44-11-983.png
          image-2025-02-24-15-44-11-983.png
          152 kB
        4. image-2025-02-21-17-10-13-597.png
          image-2025-02-21-17-10-13-597.png
          165 kB
        5. discovery.txt
          3.47 MB

        Activity

          [OBS-4987] BGP Neighbor prefixes
          emiliano.rosico@transatel.com Emiliano added a comment -

          Fixed Mike, thx so much!

          emiliano.rosico@transatel.com Emiliano added a comment - Fixed Mike, thx so much!

          VRF related issue for AFI/SAFI counters.
          Should be fixed in r13915. (rolling updates)

          landy Mike Stupalov added a comment - VRF related issue for AFI/SAFI counters. Should be fixed in r13915. (rolling updates)
          emiliano.rosico@transatel.com Emiliano added a comment -

          And in the poller I can see the insert is empty:

          Peer: 10.1.20.108 (State = established, AdminStatus = start)
          [ polled 1740411589 -> period 123 ]
          RRD /data/observium/rrd/core-pa2-2/bgp-10.1.20.108.rrd already exists - no need to create.RRD CMD[update /data/observium/rrd/core-pa2-2/bgp-10.1.20.108.rrd N:11465:38:601962:674714:539017]
          RRD RUNTIME[0.0011s]
          RRD STDOUT[OK u:0.02 s:0.04 r:21.22]
          RRD_STATUS[TRUE]
           
          SQL[UPDATE `bgpPeers` SET `bgpPeerInUpdates` ='38',`bgpPeerInUpdates_delta` ='0',`bgpPeerInUpdates_rate` ='0',`bgpPeerOutUpdates` ='11465',`bgpPeerOutUpdates_delta` ='0',`bgpPeerOutUpdates_rate` ='0',`bgpPeerInTotalMessages` ='674714',`bgpPeerInTotalMessages_delta` ='3',`bgpPeerInTotalMessages_rate` ='0.024390243902439',`bgpPeerOutTotalMessages` ='601962',`bgpPeerOutTotalMessages_delta` ='2',`bgpPeerOutTotalMessages_rate` ='0.016260162601626',`bgpPeerFsmEstablishedTime` ='539017',`bgpPeerInUpdateElapsedTime` ='538992',`bgpPeer_polled` ='1740411589' WHERE `bgpPeer_id` = '9761']
          SQL RUNTIME[0.00368595s]
           
          SQL[SELECT * FROM `bgpPeers_cbgp` WHERE `device_id` = '6' AND `bgpPeer_id` = '9761']
          SQL RUNTIME[0.00087786s]
          ipv4 unicast
          Unknown AFI/SAFI index
           
          SQL[UPDATE `bgpPeers_cbgp` SET `AcceptedPrefixes` ='',`DeniedPrefixes` ='',`PrefixAdminLimit` ='',`PrefixThreshold` ='',`PrefixClearThreshold` ='',`AdvertisedPrefixes` ='',`SuppressedPrefixes` ='',`WithdrawnPrefixes` ='' WHERE `cbgp_id` = '11360']
          SQL RUNTIME[0.00073314s]
          RRD /data/observium/rrd/core-pa2-2/cbgp-10.1.20.108.ipv4.unicast.rrd already exists - no need to create.
           
          RRD CMD[update /data/observium/rrd/core-pa2-2/cbgp-10.1.20.108.ipv4.unicast.rrd N:U:U:U:U:U]
          RRD RUNTIME[0.0011s]
          RRD STDOUT[OK u:0.02 s:0.04 r:21.23]
          RRD_STATUS[TRUE] 

          But the number of prefixes is correctly collected from the device:

          CMD[/usr/bin/snmpbulkwalk -v3 -l 'authPriv' -n '' -x *** -X *** -a *** -A *** -u *** -Pud -OQUsb -m ARISTA-BGP4V2-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp:/opt/observium/mibs/arista 'udp':'core-pa2-2':'161' aristaBgp4V2PrefixInPrefixesAccepted]CMD EXITCODE[0]
          CMD RUNTIME[0.0249s]
          STDOUT[
          aristaBgp4V2PrefixInPrefixesAccepted.1.1.4.10.1.3.187.1.1 = 155
          aristaBgp4V2PrefixInPrefixesAccepted.1.1.4.10.1.3.188.1.1 = 154
          aristaBgp4V2PrefixInPrefixesAccepted.1.1.4.10.1.20.91.1.1 = 1
          aristaBgp4V2PrefixInPrefixesAccepted.1.1.4.10.1.20.108.1.1 = 4
          aristaBgp4V2PrefixInPrefixesAccepted.1.1.4.10.1.27.201.1.1 = 0
          aristaBgp4V2PrefixInPrefixesAccepted.1.1.4.10.1.27.203.1.1 = 155
          aristaBgp4V2PrefixInPrefixesAccepted.1.1.4.10.1.27.204.1.1 = 154
          aristaBgp4V2PrefixInPrefixesAccepted.1.1.4.10.1.127.9.1.1 = 77
          aristaBgp4V2PrefixInPrefixesAccepted.1.1.4.10.4.11.10.1.1 = 11
          aristaBgp4V2PrefixInPrefixesAccepted.1.1.4.10.4.11.22.1.1 = 11
          aristaBgp4V2PrefixInPrefixesAccepted.1.1.4.10.4.13.10.1.1 = 143
          aristaBgp4V2PrefixInPrefixesAccepted.1.1.4.10.4.13.22.1.1 = 143
          aristaBgp4V2PrefixInPrefixesAccepted.1.1.4.10.250.16.123.1.1 = 45
          aristaBgp4V2PrefixInPrefixesAccepted.1.1.4.10.250.16.163.1.1 = 13
          aristaBgp4V2PrefixInPrefixesAccepted.1.1.4.10.250.99.33.1.1 = 81
          aristaBgp4V2PrefixInPrefixesAccepted.1.1.4.10.250.99.34.1.1 = 80
          aristaBgp4V2PrefixInPrefixesAccepted.1.1.4.10.255.16.123.1.1 = 55
          aristaBgp4V2PrefixInPrefixesAccepted.1.1.4.10.255.16.163.1.1 = 13 

          emiliano.rosico@transatel.com Emiliano added a comment - And in the poller I can see the insert is empty: Peer: 10.1 . 20.108 (State = established, AdminStatus = start) [ polled 1740411589 -> period 123 ] RRD /data/observium/rrd/core-pa2- 2 /bgp- 10.1 . 20.108 .rrd already exists - no need to create.RRD CMD[update /data/observium/rrd/core-pa2- 2 /bgp- 10.1 . 20.108 .rrd N: 11465 : 38 : 601962 : 674714 : 539017 ] RRD RUNTIME[ 0 .0011s] RRD STDOUT[OK u: 0.02 s: 0.04 r: 21.22 ] RRD_STATUS[TRUE]   SQL[UPDATE `bgpPeers` SET `bgpPeerInUpdates` = '38' ,`bgpPeerInUpdates_delta` = '0' ,`bgpPeerInUpdates_rate` = '0' ,`bgpPeerOutUpdates` = '11465' ,`bgpPeerOutUpdates_delta` = '0' ,`bgpPeerOutUpdates_rate` = '0' ,`bgpPeerInTotalMessages` = '674714' ,`bgpPeerInTotalMessages_delta` = '3' ,`bgpPeerInTotalMessages_rate` = '0.024390243902439' ,`bgpPeerOutTotalMessages` = '601962' ,`bgpPeerOutTotalMessages_delta` = '2' ,`bgpPeerOutTotalMessages_rate` = '0.016260162601626' ,`bgpPeerFsmEstablishedTime` = '539017' ,`bgpPeerInUpdateElapsedTime` = '538992' ,`bgpPeer_polled` = '1740411589' WHERE `bgpPeer_id` = '9761' ] SQL RUNTIME[ 0 .00368595s]   SQL[SELECT * FROM `bgpPeers_cbgp` WHERE `device_id` = '6' AND `bgpPeer_id` = '9761' ] SQL RUNTIME[ 0 .00087786s] ipv4 unicast Unknown AFI/SAFI index   SQL[UPDATE `bgpPeers_cbgp` SET `AcceptedPrefixes` = '' ,`DeniedPrefixes` = '' ,`PrefixAdminLimit` = '' ,`PrefixThreshold` = '' ,`PrefixClearThreshold` = '' ,`AdvertisedPrefixes` = '' ,`SuppressedPrefixes` = '' ,`WithdrawnPrefixes` = '' WHERE `cbgp_id` = '11360' ] SQL RUNTIME[ 0 .00073314s] RRD /data/observium/rrd/core-pa2- 2 /cbgp- 10.1 . 20.108 .ipv4.unicast.rrd already exists - no need to create.   RRD CMD[update /data/observium/rrd/core-pa2- 2 /cbgp- 10.1 . 20.108 .ipv4.unicast.rrd N:U:U:U:U:U] RRD RUNTIME[ 0 .0011s] RRD STDOUT[OK u: 0.02 s: 0.04 r: 21.23 ] RRD_STATUS[TRUE] But the number of prefixes is correctly collected from the device: CMD[/usr/bin/snmpbulkwalk -v3 -l 'authPriv' -n '' -x *** -X *** -a *** -A *** -u *** -Pud -OQUsb -m ARISTA-BGP4V2-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp:/opt/observium/mibs/arista 'udp' : 'core-pa2-2' : '161' aristaBgp4V2PrefixInPrefixesAccepted]CMD EXITCODE[ 0 ] CMD RUNTIME[ 0 .0249s] STDOUT[ aristaBgp4V2PrefixInPrefixesAccepted. 1.1 . 4.10 . 1.3 . 187.1 . 1 = 155 aristaBgp4V2PrefixInPrefixesAccepted. 1.1 . 4.10 . 1.3 . 188.1 . 1 = 154 aristaBgp4V2PrefixInPrefixesAccepted. 1.1 . 4.10 . 1.20 . 91.1 . 1 = 1 aristaBgp4V2PrefixInPrefixesAccepted. 1.1 . 4.10 . 1.20 . 108.1 . 1 = 4 aristaBgp4V2PrefixInPrefixesAccepted. 1.1 . 4.10 . 1.27 . 201.1 . 1 = 0 aristaBgp4V2PrefixInPrefixesAccepted. 1.1 . 4.10 . 1.27 . 203.1 . 1 = 155 aristaBgp4V2PrefixInPrefixesAccepted. 1.1 . 4.10 . 1.27 . 204.1 . 1 = 154 aristaBgp4V2PrefixInPrefixesAccepted. 1.1 . 4.10 . 1.127 . 9.1 . 1 = 77 aristaBgp4V2PrefixInPrefixesAccepted. 1.1 . 4.10 . 4.11 . 10.1 . 1 = 11 aristaBgp4V2PrefixInPrefixesAccepted. 1.1 . 4.10 . 4.11 . 22.1 . 1 = 11 aristaBgp4V2PrefixInPrefixesAccepted. 1.1 . 4.10 . 4.13 . 10.1 . 1 = 143 aristaBgp4V2PrefixInPrefixesAccepted. 1.1 . 4.10 . 4.13 . 22.1 . 1 = 143 aristaBgp4V2PrefixInPrefixesAccepted. 1.1 . 4.10 . 250.16 . 123.1 . 1 = 45 aristaBgp4V2PrefixInPrefixesAccepted. 1.1 . 4.10 . 250.16 . 163.1 . 1 = 13 aristaBgp4V2PrefixInPrefixesAccepted. 1.1 . 4.10 . 250.99 . 33.1 . 1 = 81 aristaBgp4V2PrefixInPrefixesAccepted. 1.1 . 4.10 . 250.99 . 34.1 . 1 = 80 aristaBgp4V2PrefixInPrefixesAccepted. 1.1 . 4.10 . 255.16 . 123.1 . 1 = 55 aristaBgp4V2PrefixInPrefixesAccepted. 1.1 . 4.10 . 255.16 . 163.1 . 1 = 13
          emiliano.rosico@transatel.com Emiliano added a comment -

          Hi Mike, sry for the delay.

          For me the problem is after some upgrade anything it's wrong with the DB.... The problem it's present over 3 different vendors (Juniper, Cisco, Arista) and in different hardware and version.

          The strange is, the same equipment with NaN in some peers, show the new peers with the right information.

          CMD RUNTIME[0.0259s]
          STDOUT[
          aristaBgp4V2PrefixInPrefixes.1.1.4.10.1.3.187.1.1 = 154
          aristaBgp4V2PrefixInPrefixes.1.1.4.10.1.3.188.1.1 = 153
          aristaBgp4V2PrefixInPrefixes.1.1.4.10.1.20.91.1.1 = 1
          aristaBgp4V2PrefixInPrefixes.1.1.4.10.1.20.108.1.1 = 4
          aristaBgp4V2PrefixInPrefixes.1.1.4.10.1.27.201.1.1 = 0
          aristaBgp4V2PrefixInPrefixes.1.1.4.10.1.27.203.1.1 = 154
          aristaBgp4V2PrefixInPrefixes.1.1.4.10.1.27.204.1.1 = 153
          aristaBgp4V2PrefixInPrefixes.1.1.4.10.1.127.9.1.1 = 76
          aristaBgp4V2PrefixInPrefixes.1.1.4.10.4.11.10.1.1 = 77
          aristaBgp4V2PrefixInPrefixes.1.1.4.10.4.11.22.1.1 = 77
          aristaBgp4V2PrefixInPrefixes.1.1.4.10.4.13.10.1.1 = 305
          aristaBgp4V2PrefixInPrefixes.1.1.4.10.4.13.22.1.1 = 305
          aristaBgp4V2PrefixInPrefixes.1.1.4.10.250.16.123.1.1 = 45
          aristaBgp4V2PrefixInPrefixes.1.1.4.10.250.16.163.1.1 = 13
          aristaBgp4V2PrefixInPrefixes.1.1.4.10.250.99.33.1.1 = 81
          aristaBgp4V2PrefixInPrefixes.1.1.4.10.250.99.34.1.1 = 80
          aristaBgp4V2PrefixInPrefixes.1.1.4.10.255.16.123.1.1 = 55
          aristaBgp4V2PrefixInPrefixes.1.1.4.10.255.16.163.1.1 = 13
          

          gethostbyaddr6(10.1.20.108) failed: DNS request failed: The domain name referenced in the query does not exist.bgpPeerLocalAddr: 10.1.20.108, bgpPeerRemoteAddr: 10.1.20.107SQL[SELECT * FROM `devices` LEFT JOIN `devices_locations` USING (`device_id`) WHERE `device_id` = '455']
          SQL RUNTIME[0.0015142s]SQL[SELECT * FROM `device_graphs` WHERE `device_id` = '455']
          SQL RUNTIME[0.00096989s]SQL[SELECT * FROM `bgpPeers` WHERE `device_id` = '6' AND `bgpPeerRemoteAddr` = '10.1.20.108']
          SQL RUNTIME[0.00137186s]
          INDEX: 1.1.4.10.1.20.108, AS: 59576, IP: 10.1.20.108, AFI: ipv4, SAFI: unicastSQL[SELECT EXISTS (SELECT 1 FROM `bgpPeers_cbgp` WHERE `device_id` = '6' AND `bgpPeer_id` = '9761' AND `afi` = 'ipv4' AND `safi` = 'unicast')]
          SQL RUNTIME[0.00084591s]SQL[UPDATE `bgpPeers_cbgp` SET `bgpPeerIndex` ='1.1.4.10.1.20.108' WHERE device_id = '6' AND `bgpPeer_id` = '9761' AND `afi` = 'ipv4' AND `safi` = 'unicast']
           

           

           

          emiliano.rosico@transatel.com Emiliano added a comment - Hi Mike, sry for the delay. For me the problem is after some upgrade anything it's wrong with the DB.... The problem it's present over 3 different vendors (Juniper, Cisco, Arista) and in different hardware and version. The strange is, the same equipment with NaN in some peers, show the new peers with the right information. CMD RUNTIME[ 0 .0259s] STDOUT[ aristaBgp4V2PrefixInPrefixes. 1.1 . 4.10 . 1.3 . 187.1 . 1 = 154 aristaBgp4V2PrefixInPrefixes. 1.1 . 4.10 . 1.3 . 188.1 . 1 = 153 aristaBgp4V2PrefixInPrefixes. 1.1 . 4.10 . 1.20 . 91.1 . 1 = 1 aristaBgp4V2PrefixInPrefixes. 1.1 . 4.10 . 1.20 . 108.1 . 1 = 4 aristaBgp4V2PrefixInPrefixes. 1.1 . 4.10 . 1.27 . 201.1 . 1 = 0 aristaBgp4V2PrefixInPrefixes. 1.1 . 4.10 . 1.27 . 203.1 . 1 = 154 aristaBgp4V2PrefixInPrefixes. 1.1 . 4.10 . 1.27 . 204.1 . 1 = 153 aristaBgp4V2PrefixInPrefixes. 1.1 . 4.10 . 1.127 . 9.1 . 1 = 76 aristaBgp4V2PrefixInPrefixes. 1.1 . 4.10 . 4.11 . 10.1 . 1 = 77 aristaBgp4V2PrefixInPrefixes. 1.1 . 4.10 . 4.11 . 22.1 . 1 = 77 aristaBgp4V2PrefixInPrefixes. 1.1 . 4.10 . 4.13 . 10.1 . 1 = 305 aristaBgp4V2PrefixInPrefixes. 1.1 . 4.10 . 4.13 . 22.1 . 1 = 305 aristaBgp4V2PrefixInPrefixes. 1.1 . 4.10 . 250.16 . 123.1 . 1 = 45 aristaBgp4V2PrefixInPrefixes. 1.1 . 4.10 . 250.16 . 163.1 . 1 = 13 aristaBgp4V2PrefixInPrefixes. 1.1 . 4.10 . 250.99 . 33.1 . 1 = 81 aristaBgp4V2PrefixInPrefixes. 1.1 . 4.10 . 250.99 . 34.1 . 1 = 80 aristaBgp4V2PrefixInPrefixes. 1.1 . 4.10 . 255.16 . 123.1 . 1 = 55 aristaBgp4V2PrefixInPrefixes. 1.1 . 4.10 . 255.16 . 163.1 . 1 = 13 gethostbyaddr6( 10.1 . 20.108 ) failed: DNS request failed: The domain name referenced in the query does not exist.bgpPeerLocalAddr: 10.1 . 20.108 , bgpPeerRemoteAddr: 10.1 . 20 .107SQL[SELECT * FROM `devices` LEFT JOIN `devices_locations` USING (`device_id`) WHERE `device_id` = '455' ] SQL RUNTIME[ 0 .0015142s]SQL[SELECT * FROM `device_graphs` WHERE `device_id` = '455' ] SQL RUNTIME[ 0 .00096989s]SQL[SELECT * FROM `bgpPeers` WHERE `device_id` = '6' AND `bgpPeerRemoteAddr` = '10.1.20.108' ] SQL RUNTIME[ 0 .00137186s] INDEX: 1.1 . 4.10 . 1.20 . 108 , AS: 59576 , IP: 10.1 . 20.108 , AFI: ipv4, SAFI: unicastSQL[SELECT EXISTS (SELECT 1 FROM `bgpPeers_cbgp` WHERE `device_id` = '6' AND `bgpPeer_id` = '9761' AND `afi` = 'ipv4' AND `safi` = 'unicast' )] SQL RUNTIME[ 0 .00084591s]SQL[UPDATE `bgpPeers_cbgp` SET `bgpPeerIndex` = '1.1.4.10.1.20.108' WHERE device_id = '6' AND `bgpPeer_id` = '9761' AND `afi` = 'ipv4' AND `safi` = 'unicast' ]    

          You should not ignore the requested information.
          These counters only show the values ​​that the device transmitted via snmp.
          If they do not match the actual values, you need to check the latest firmware of the device and contact the device vendor.

          We can only help by pointing out the data that needs to be checked according to the devel information (requested in previous comment).

          landy Mike Stupalov added a comment - You should not ignore the requested information. These counters only show the values ​​that the device transmitted via snmp. If they do not match the actual values, you need to check the latest firmware of the device and contact the device vendor. We can only help by pointing out the data that needs to be checked according to the devel information (requested in previous comment).

          General questions and device support can be discussed in our Discord channel, click here to join.


          Please make and attach additional information about the device:

          • full snmp dump from device:

            snmpwalk -v2c -c <community> -t 3 -Cc --hexOutputLength=0 -Ih -ObentxU <hostname> .1 > myagent.snmpwalk
            snmpwalk -v2c -c <community> -t 3 -Cc --hexOutputLength=0 -Ih -ObentxU <hostname> .1.3.6.1.4.1 >> myagent.snmpwalk

            If device not support SNMP version 2c, replace -v2c with -v1.

          • If you have problems with discovery or poller processes, please do and attach these debugs:

            ./discovery.php -d -h <device>
            ./poller.php -d -h <device>

          • additionally attach device and/or vendor specific MIB files

          This comment is added automatically.

          bot Observium Bot added a comment - General questions and device support can be discussed in our Discord channel, click here to join . Please make and attach additional information about the device: full snmp dump from device: snmpwalk -v2c -c <community> -t 3 -Cc --hexOutputLength=0 -Ih -ObentxU <hostname> .1 > myagent.snmpwalk snmpwalk -v2c -c <community> -t 3 -Cc --hexOutputLength=0 -Ih -ObentxU <hostname> .1.3.6.1.4.1 >> myagent.snmpwalk If device not support SNMP version 2c, replace -v2c with -v1. If you have problems with discovery or poller processes, please do and attach these debugs: ./discovery.php -d -h <device> ./poller.php -d -h <device> additionally attach device and/or vendor specific MIB files This comment is added automatically.

          People

            landy Mike Stupalov
            emiliano.rosico@transatel.com Emiliano
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: