Uploaded image for project: 'Observium'
  1. Observium
  2. OBS-4134

BGP peers uptime 27 years

Details

    • Vendor Bug
    • Resolution: Won't Fix
    • Minor
    • None
    • Enterprise Edition
    • Discovery

    Description

      Dell os10 bgp peers are showing incorrect uptimes. Eventlog is also flooded every 5min about BGP session flap even though there has been no flaps. It looks like that this started happening after upgrade from 22.4.11943 to 22.5.12060. DB schema was also upgraded from 462 to 467.

       

      Attachments

        1. discover526.txt
          305 kB
        2. poller526.txt
          426 kB
        3. Screenshot 2022-06-02 at 15.21.38.png
          Screenshot 2022-06-02 at 15.21.38.png
          141 kB
        4. Screenshot 2022-06-02 at 16.12.10.png
          Screenshot 2022-06-02 at 16.12.10.png
          208 kB

        Issue Links

          relates to

          Improvement - An improvement or enhancement to an existing feature or task. OBS-3101 BGP monitoring on Dell OS10

          • Major - Major loss of function.
          • Closed

          Vendor Bug - This is vendor specific issue, not possible to be fixed inside our project. OBS-4280 BGP session in shutdown state on Dell NOS10 reported as down

          • Major - Major loss of function.
          • Resolved

          Activity

            [OBS-4134] BGP peers uptime 27 years
            landy Mike Stupalov added a comment -

            No answer from Dell support?

            landy Mike Stupalov added a comment - No answer from Dell support?
            jsalo Jari Salo added a comment -

            I have created a ticket to Dell about this issue.

            jsalo Jari Salo added a comment - I have created a ticket to Dell about this issue.
            landy Mike Stupalov added a comment -

            In r12079 I fixed uptime value scale, but I can't fix uptime reduce in time, because this is firmware issue.

             

            And added workaround for os10bgp4V2PeerAdminStatus, but it's still issue in firmware.

            landy Mike Stupalov added a comment - In r12079 I fixed uptime value scale, but I can't fix uptime reduce in time, because this is firmware issue.   And added workaround for os10bgp4V2PeerAdminStatus , but it's still issue in firmware.
            landy Mike Stupalov added a comment - - edited

            Another trouble which I see, seems as device incorrectly report all peers in halted adminstate (os10bgp4V2PeerAdminStatus).

            This is also reason for write to Dell support.

            landy Mike Stupalov added a comment - - edited Another trouble which I see, seems as device incorrectly report all peers in halted adminstate ( os10bgp4V2PeerAdminStatus ). This is also reason for write to Dell support.
            landy Mike Stupalov added a comment - - edited

            Ok, what I understand from this output:

            1. time reported with x100 scale (this can be simply fixed in our code)
            2. but uptime anyway reduced intime (instead increase), ie for 10.10.11.1:
            • 18:12:49 - 14w:3d:16:45:23
            • 18:12:54 - 14w:3d:16:45:20
            • 18:13:24 - 14w:3d:16:44:49

            I not know what this times should mean than, anyway need ask support for this.

            landy Mike Stupalov added a comment - - edited Ok, what I understand from this output: time reported with x100 scale (this can be simply fixed in our code) but uptime anyway reduced intime (instead increase), ie for 10.10.11.1: 18:12:49 - 14w:3d:16:45:23 18:12:54 - 14w:3d:16:45:20 18:13:24 - 14w:3d:16:44:49 I not know what this times should mean than, anyway need ask support for this.
            jsalo Jari Salo added a comment -

            These show commands do look like there's a bug with the firmware. I'll raise a ticket to Dell.

            jsalo Jari Salo added a comment - These show commands do look like there's a bug with the firmware. I'll raise a ticket to Dell.
            jsalo Jari Salo added a comment -

            1120-rcsp1-t5# show clock
            2022-06-02T18:12:48.97+03:00
            1120-rcsp1-t5# show ip bgp summary
            BGP router identifier 172.16.100.1 local AS number 65501
            Global BFD is enabled
            Neighbor                                      AS          MsgRcvd     MsgSent     Up/Down           State/Pfx
            10.10.11.1                                    65511       8536843     8536971     14w:3d:16:45:23   3
            10.10.11.3                                    65511       8535985     8536072     14w:3d:17:18:19   3
            10.10.11.5                                    65512       8536810     8536886     14w:3d:16:09:39   3
            10.10.11.7                                    65512       8537025     8536654     14w:3d:15:30:14   3
            10.10.11.9                                    65513       8217370     8217515     4d:12:01:48       3
            10.10.11.11                                   65513       8217210     8217526     4d:12:01:01       3
            1120-rcsp1-t5# show clock
            2022-06-02T18:12:53.56+03:00
            1120-rcsp1-t5# show ip bgp summary
            BGP router identifier 172.16.100.1 local AS number 65501
            Global BFD is enabled
            Neighbor                                      AS          MsgRcvd     MsgSent     Up/Down           State/Pfx
            10.10.11.1                                    65511       8536844     8536972     14w:3d:16:45:20   3
            10.10.11.3                                    65511       8535986     8536073     14w:3d:17:18:16   3
            10.10.11.5                                    65512       8536812     8536887     14w:3d:16:09:36   3
            10.10.11.7                                    65512       8537027     8536655     14w:3d:15:30:11   3
            10.10.11.9                                    65513       8217371     8217516     4d:12:01:45       3
            10.10.11.11                                   65513       8217211     8217527     4d:12:00:58       3
            1120-rcsp1-t5# show clock
            2022-06-02T18:13:24.28+03:00
            1120-rcsp1-t5# show ip bgp summary
            BGP router identifier 172.16.100.1 local AS number 65501
            Global BFD is enabled
            Neighbor                                      AS          MsgRcvd     MsgSent     Up/Down           State/Pfx
            10.10.11.1                                    65511       8536856     8536984     14w:3d:16:44:49   3
            10.10.11.3                                    65511       8535999     8536086     14w:3d:17:17:45   3
            10.10.11.5                                    65512       8536824     8536899     14w:3d:16:09:05   3
            10.10.11.7                                    65512       8537039     8536667     14w:3d:15:29:40   3
            10.10.11.9                                    65513       8217383     8217528     4d:12:01:14       3
            10.10.11.11                                   65513       8217223     8217539     4d:12:00:27       3

            jsalo Jari Salo added a comment - 1120-rcsp1-t5# show clock 2022-06-02T18:12:48.97+03:00 1120-rcsp1-t5# show ip bgp summary BGP router identifier 172.16.100.1 local AS number 65501 Global BFD is enabled Neighbor                                      AS          MsgRcvd     MsgSent     Up/Down           State/Pfx 10.10.11.1                                    65511       8536843     8536971     14w:3d:16:45:23   3 10.10.11.3                                    65511       8535985     8536072     14w:3d:17:18:19   3 10.10.11.5                                    65512       8536810     8536886     14w:3d:16:09:39   3 10.10.11.7                                    65512       8537025     8536654     14w:3d:15:30:14   3 10.10.11.9                                    65513       8217370     8217515     4d:12:01:48       3 10.10.11.11                                   65513       8217210     8217526     4d:12:01:01       3 1120-rcsp1-t5# show clock 2022-06-02T18:12:53.56+03:00 1120-rcsp1-t5# show ip bgp summary BGP router identifier 172.16.100.1 local AS number 65501 Global BFD is enabled Neighbor                                      AS          MsgRcvd     MsgSent     Up/Down           State/Pfx 10.10.11.1                                    65511       8536844     8536972     14w:3d:16:45:20   3 10.10.11.3                                    65511       8535986     8536073     14w:3d:17:18:16   3 10.10.11.5                                    65512       8536812     8536887     14w:3d:16:09:36   3 10.10.11.7                                    65512       8537027     8536655     14w:3d:15:30:11   3 10.10.11.9                                    65513       8217371     8217516     4d:12:01:45       3 10.10.11.11                                   65513       8217211     8217527     4d:12:00:58       3 1120-rcsp1-t5# show clock 2022-06-02T18:13:24.28+03:00 1120-rcsp1-t5# show ip bgp summary BGP router identifier 172.16.100.1 local AS number 65501 Global BFD is enabled Neighbor                                      AS          MsgRcvd     MsgSent     Up/Down           State/Pfx 10.10.11.1                                    65511       8536856     8536984     14w:3d:16:44:49   3 10.10.11.3                                    65511       8535999     8536086     14w:3d:17:17:45   3 10.10.11.5                                    65512       8536824     8536899     14w:3d:16:09:05   3 10.10.11.7                                    65512       8537039     8536667     14w:3d:15:29:40   3 10.10.11.9                                    65513       8217383     8217528     4d:12:01:14       3 10.10.11.11                                   65513       8217223     8217539     4d:12:00:27       3
            jsalo Jari Salo added a comment -

            Thu 02 Jun 2022 06:11:19 PM EEST

            DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.1" = Gauge32: 878681500 seconds
            DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.3" = Gauge32: 878879100 seconds
            DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.5" = Gauge32: 878467100 seconds
            DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.7" = Gauge32: 878230600 seconds
            DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.9" = Gauge32: 38900000 seconds
            DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.11" = Gauge32: 38895300 seconds

             

            Thu 02 Jun 2022 06:11:22 PM EEST

            DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.1" = Gauge32: 878681200 seconds
            DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.3" = Gauge32: 878878800 seconds
            DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.5" = Gauge32: 878466800 seconds
            DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.7" = Gauge32: 878230300 seconds
            DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.9" = Gauge32: 38899700 seconds
            DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.11" = Gauge32: 38895000 seconds

             

            Thu 02 Jun 2022 06:11:24 PM EEST

            DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.1" = Gauge32: 878680900 seconds
            DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.3" = Gauge32: 878878600 seconds
            DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.5" = Gauge32: 878466600 seconds
            DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.7" = Gauge32: 878230100 seconds
            DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.9" = Gauge32: 38899500 seconds
            DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.11" = Gauge32: 38894700 seconds

             

            Thu 02 Jun 2022 06:11:26 PM EEST

            DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.1" = Gauge32: 878680800 seconds
            DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.3" = Gauge32: 878878400 seconds
            DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.5" = Gauge32: 878466400 seconds
            DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.7" = Gauge32: 878229900 seconds
            DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.9" = Gauge32: 38899300 seconds
            DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.11" = Gauge32: 38894600 seconds

             

            Thu 02 Jun 2022 06:11:28 PM EEST

            DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.1" = Gauge32: 878680600 seconds
            DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.3" = Gauge32: 878878200 seconds
            DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.5" = Gauge32: 878466200 seconds
            DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.7" = Gauge32: 878229700 seconds
            DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.9" = Gauge32: 38899100 seconds
            DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.11" = Gauge32: 38894300 seconds

            jsalo Jari Salo added a comment - Thu 02 Jun 2022 06:11:19 PM EEST DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.1" = Gauge32: 878681500 seconds DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.3" = Gauge32: 878879100 seconds DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.5" = Gauge32: 878467100 seconds DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.7" = Gauge32: 878230600 seconds DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.9" = Gauge32: 38900000 seconds DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.11" = Gauge32: 38895300 seconds   Thu 02 Jun 2022 06:11:22 PM EEST DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.1" = Gauge32: 878681200 seconds DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.3" = Gauge32: 878878800 seconds DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.5" = Gauge32: 878466800 seconds DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.7" = Gauge32: 878230300 seconds DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.9" = Gauge32: 38899700 seconds DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.11" = Gauge32: 38895000 seconds   Thu 02 Jun 2022 06:11:24 PM EEST DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.1" = Gauge32: 878680900 seconds DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.3" = Gauge32: 878878600 seconds DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.5" = Gauge32: 878466600 seconds DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.7" = Gauge32: 878230100 seconds DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.9" = Gauge32: 38899500 seconds DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.11" = Gauge32: 38894700 seconds   Thu 02 Jun 2022 06:11:26 PM EEST DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.1" = Gauge32: 878680800 seconds DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.3" = Gauge32: 878878400 seconds DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.5" = Gauge32: 878466400 seconds DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.7" = Gauge32: 878229900 seconds DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.9" = Gauge32: 38899300 seconds DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.11" = Gauge32: 38894600 seconds   Thu 02 Jun 2022 06:11:28 PM EEST DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.1" = Gauge32: 878680600 seconds DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.3" = Gauge32: 878878200 seconds DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.5" = Gauge32: 878466200 seconds DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.7" = Gauge32: 878229700 seconds DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.9" = Gauge32: 38899100 seconds DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime.1.ipv4."10.10.11.11" = Gauge32: 38894300 seconds
            landy Mike Stupalov added a comment - - edited

            Can you try run multiple times (4-5) this command (change <community> to your correct snmp community):

            date; echo; snmpbulkwalk -v2c -c <community> -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp:/opt/observium/mibs/dell 1120-rcsp1-t5abr2r8-kiv.mgmt.fi.ficore.net DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime

            I want to try understand how this values changed.

            Additionally if possible show extended information for one of this peer from device console (with Established Time).

            landy Mike Stupalov added a comment - - edited Can you try run multiple times (4-5) this command (change <community> to your correct snmp community): date; echo; snmpbulkwalk -v2c -c <community> -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp:/opt/observium/mibs/dell 1120-rcsp1-t5abr2r8-kiv.mgmt.fi.ficore.net DELLEMC-OS10-BGP4V2-MIB::os10bgp4V2PeerFsmEstablishedTime I want to try understand how this values changed. Additionally if possible show extended information for one of this peer from device console (with Established Time).
            landy Mike Stupalov added a comment -

            Just polling bgp peers by DELLEMC-OS10-BGP4V2-MIB was added recently (in r11955).

            landy Mike Stupalov added a comment - Just polling bgp peers by DELLEMC-OS10-BGP4V2-MIB was added recently (in r11955).
            jsalo Jari Salo added a comment -

            Hmm, this is interesting. We've had these devices for a couple of years and only now this issue has surfaced. We do have some switches with newer software (10.5.2.7), but they behave exactly the same. However, only one of these devices (the one, where the debugs are from) is reporting bgp flaps all the time to eventlog.

            jsalo Jari Salo added a comment - Hmm, this is interesting. We've had these devices for a couple of years and only now this issue has surfaced. We do have some switches with newer software (10.5.2.7), but they behave exactly the same. However, only one of these devices (the one, where the debugs are from) is reporting bgp flaps all the time to eventlog.

            People

              landy Mike Stupalov
              jsalo Jari Salo
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: