Uploaded image for project: 'Observium'
  1. Observium
  2. OBS-4468

/api/v0/devices/ returns incomplete list of devices since latest update

Details

    • Bug
    • Resolution: Fixed
    • Major
    • None
    • Professional Edition
    • API
    • None

    Description

      Today I have run {{}}

      svn update .

      }} after seeing an out-of-date notice. The previous update was on 23 Feb. Version post update is {{{}23.4.12622 (4th April 2023).

      I am now observing that the device list obtained from /api/v0/devices/ is incomplete. A couple of devices are not listed. In contrast, the{{ /api/v0/inventory/}} is complete, specifically the devices missing from {{/api/v0/devices/ }}are included.

      I am reproducibly seeing this in our two instances. In one case 10 out of 431 devices are missing, in the other case 34 out of 623. There is no obvious hint towards specific models - the missing devices are wireless, firewalls, and F5's but also access switches. Same models with other serial numbers/names sow up fine. Also it does not seem to be related to a location or device name. The only thing that is constant is that it's always the same devices missing upon repeated API calls.

      Seems like a glitch in html/api/v0/includes/devices.inc.php perhaps?

      Attachments

        Activity

          [OBS-4468] /api/v0/devices/ returns incomplete list of devices since latest update

          Fixed in r12710.

          landy Mike Stupalov added a comment - Fixed in r12710.

          Confirmed that the fix is working fine. Thanks!

          O Helge Oldach added a comment - Confirmed that the fix is working fine. Thanks!

          It should be fixed in the latest trunk revision.

          Thanks,

          adam.

          adama Adam Armstrong added a comment - It should be fixed in the latest trunk revision. Thanks, adam.

          Actually I had updated this morning (version is now "23.4.12651 (13th April 2023)") but the issue persists.

          I briefly walked through devices.inc.php and noticed that the code paths are quite different depending on whether the URI has a query string (e.g., device ID or some option), or it's just the plain /devices/ API request.

          As an experiment, I have reverted devices.inc.php to r12621 temporarily and that indeed fixes my issue. I'm now getting the complete list of devices again. This file had also been changed with r12633 however this change seems minor and has no impact.

          I'm afraid however that I'm neither PHP nor Observium experienced enough to troubleshoot further... Sorry for that.

          O Helge Oldach added a comment - Actually I had updated this morning (version is now "23.4.12651 (13th April 2023)") but the issue persists. I briefly walked through devices.inc.php and noticed that the code paths are quite different depending on whether the URI has a query string (e.g., device ID or some option), or it's just the plain /devices/ API request. As an experiment, I have reverted devices.inc.php to r12621 temporarily and that indeed fixes my issue. I'm now getting the complete list of devices again. This file had also been changed with r12633 however this change seems minor and has no impact. I'm afraid however that I'm neither PHP nor Observium experienced enough to troubleshoot further... Sorry for that.

          latest changes is "minimal" and not related with such troubles.
          if you can get individual device by API, than in all devices it's should be too.

          Try with latest revision please!

          landy Mike Stupalov added a comment - latest changes is "minimal" and not related with such troubles. if you can get individual device by API, than in all devices it's should be too. Try with latest revision please!

          Administrator = 10. (This has been working fine with the same scripting until a few weeks ago. I suspect recent changes to html/api/v0/includes/devices.inc.php are the cause?) 

          O Helge Oldach added a comment - Administrator = 10. (This has been working fine with the same scripting until a few weeks ago. I suspect recent changes to html/api/v0/includes/devices.inc.php are the cause?) 

          Which user level of user XXX?

          landy Mike Stupalov added a comment - Which user level of user XXX?

          What I tried to say is: The devices in question are there, they are in the DB, they are properly handled and managed, they can be accessed individually through the API, however when I retrieve the full list of devices some few devices don't show up.

          BTW, same thing with the 'inventory' API.

          O Helge Oldach added a comment - What I tried to say is: The devices in question are there, they are in the DB, they are properly handled and managed, they can be accessed individually through the API, however when I retrieve the full list of devices some few devices don't show up. BTW, same thing with the 'inventory' API.
          O Helge Oldach added a comment - - edited

          I was afraid you would be saying so

          Here is an example, device ID 1789. I can retrieve it when I ask for the ID explicitly, but when I walk the list of devices, it's not included:

          # curl --no-progress-meter -u XXX:YYY https://apin008.ap-rdc01.nexperia.com/api/v0/devices/1789
          

          {"status":"ok","device":\{"device_id":"1789","poller_id":"0","hostname":"demucnwa0007.de-muc01.nexperia.com","sysName":"demucnwa0007","label":null,"ip":"172.17.66.12","snmp_community":"nexpublic","snmp_authlevel":null,"snmp_authname":null,"snmp_authpass":null,"snmp_authalgo":null,"snmp_cryptopass":null,"snmp_cryptoalgo":null,"snmp_context":null,"snmpable":null,"snmp_version":"v2c","snmp_port":"161","snmp_timeout":null,"snmp_retries":null,"snmp_maxrep":null,"ssh_port":"22","agent_version":null,"snmp_transport":"udp","bgpLocalAs":null,"snmpEngineID":"","sysObjectID":".1.3.6.1.4.1.29671.2.43","sysDescr":"Meraki MR44 Cloud Managed AP","sysContact":"","version":null,"hardware":"MR44","vendor":"Cisco","features":null,"location":"","os":"meraki-ap","status":"1","status_type":"ok","ignore":"0","ignore_until":null,"asset_tag":null,"disabled":"0","uptime":null,"last_rebooted":null,"force_discovery":"0","last_polled":"2023-04-21 06:36:32","last_discovered":"2023-04-21 05:29:57","last_alerter":"2023-04-21 06:36:32","last_polled_timetaken":"17.80","last_discovered_timetaken":"7.34","purpose":null,"type":"wireless","serial":null,"icon":null,"distro":null,"distro_ver":null,"kernel":null,"arch":null,"location_id":"2063","location_lat":null,"location_lon":null,"location_country":"unknown","location_state":"Unknown","location_county":"Unknown","location_city":"Unknown","location_geoapi":"geocodefarm","location_status":"Geocoding ENABLED, try detect device coordinates:\nNOT REQUESTED\n  GEO API REQUEST: \n  QUERY DATE: Fri, 21 Apr 2023 05:31:04 +0000","location_manual":"0","location_updated":"2023-04-21 05:31:04","state":{"poller_mod_perf":{"system":1.0926,"os":0.0011,"sensors":0.0005,"status":0.0002,"counter":0.0002,"processors":0.0004,"mempools":0.0004,"storage":0.0027,"netstats":1.0839,"ucd-mib":0.0003,"ipSystemStats":0.43,"ports":4.6764,"ucd-diskio":0.0092,"wifi":0.0031,"p2p-radios":0.0006,"ospf":0.0001,"sla":0.0013,"pseudowires":0.0004,"mac-accounting":0.0004,"loadbalancer":0.0002,"entity-physical":0.0001,"applications":10.011,"graphs":0.0004,"oids":0.0004,"probes":0.0003},"poller_ports_perf":\{"IF-MIB":2.502121925354,"EtherLike-MIB":0.42258787155151,"Q-BRIDGE-MIB":1.3024079799652,"IP-MIB":0.43505191802979},"poller_history":\{"1682058974":17.8016,"1682058674":18.0325,"1682058367":18.0716,"1682058065":17.9358,"1682057760":18.1698,"1682057459":17.9337,"1682057151":17.8016,"1682056888":18.9871,"1682056590":9.754,"1682056289":9.8054,"1682055988":9.3644,"1682055688":10.0494,"1682055398":9.8505,"1682055111":8.2786,"1682055062":10.1984},"discovery_history":\{"1682054990":7.3438},"discovery_mod_perf":\{"os":0.4825,"mibs":0.9378,"vrf":0.0008,"ports":1.1594,"ports-stack":0.2342,"vlans":0.4422,"ports_vlans":null,"oids":0.0004,"ip-addresses":1.3218,"processors":0.0006,"mempools":0.0006,"inventory":0.2418,"sensors":0.236,"storage":0.0005,"neighbours":0.6842,"arp-table":1.1065,"mac-accounting":0.0002,"sla":0.0005,"pseudowires":0.0004,"virtual-machines":0.0003,"cisco-cbqos":0.2535,"ucd-diskio":0.2299,"wifi":0.0008,"p2p-radios":0.0001,"graphs":0.0002}},"row_class":"","html_row_class":"up","os_text":"Cisco Meraki AP","human_local_as":null,"humanized":true,"graphs":\{"availability":{"device_graph_id":"891076","device_id":"1789","graph":"availability","enabled":"1"},"ping":\{"device_graph_id":"891077","device_id":"1789","graph":"ping","enabled":"1"},"ping_snmp":\{"device_graph_id":"891078","device_id":"1789","graph":"ping_snmp","enabled":"1"},"bits":\{"device_graph_id":"891079","device_id":"1789","graph":"bits","enabled":"1"},"poller_perf":\{"device_graph_id":"891080","device_id":"1789","graph":"poller_perf","enabled":"1"},"pollersnmp_count":\{"device_graph_id":"891081","device_id":"1789","graph":"pollersnmp_count","enabled":"1"},"pollersnmp_times":\{"device_graph_id":"891082","device_id":"1789","graph":"pollersnmp_times","enabled":"1"},"pollersnmp_errors_count":\{"device_graph_id":"891083","device_id":"1789","graph":"pollersnmp_errors_count","enabled":"1"},"pollersnmp_errors_times":\{"device_graph_id":"891084","device_id":"1789","graph":"pollersnmp_errors_times","enabled":"1"},"pollerdb_count":\{"device_graph_id":"891085","device_id":"1789","graph":"pollerdb_count","enabled":"1"},"pollerdb_times":\{"device_graph_id":"891086","device_id":"1789","graph":"pollerdb_times","enabled":"1"},"pollermemory_perf":\{"device_graph_id":"891087","device_id":"1789","graph":"pollermemory_perf","enabled":"1"}}},"device_id":"1789"}
          

          # curl --no-progress-meter -u XXX:YYY https://apin008.ap-rdc01.nexperia.com/api/v0/devices/ | fgrep '"1789"'
          

          Does this help isolating the issue?

          O Helge Oldach added a comment - - edited I was afraid you would be saying so Here is an example, device ID 1789. I can retrieve it when I ask for the ID explicitly, but when I walk the list of devices, it's not included: # curl --no-progress-meter -u XXX:YYY https://apin008.ap-rdc01.nexperia.com/api/v0/devices/1789 {"status":"ok","device":\{"device_id":"1789","poller_id":"0","hostname":"demucnwa0007.de-muc01.nexperia.com","sysName":"demucnwa0007","label":null,"ip":"172.17.66.12","snmp_community":"nexpublic","snmp_authlevel":null,"snmp_authname":null,"snmp_authpass":null,"snmp_authalgo":null,"snmp_cryptopass":null,"snmp_cryptoalgo":null,"snmp_context":null,"snmpable":null,"snmp_version":"v2c","snmp_port":"161","snmp_timeout":null,"snmp_retries":null,"snmp_maxrep":null,"ssh_port":"22","agent_version":null,"snmp_transport":"udp","bgpLocalAs":null,"snmpEngineID":"","sysObjectID":".1.3.6.1.4.1.29671.2.43","sysDescr":"Meraki MR44 Cloud Managed AP","sysContact":"","version":null,"hardware":"MR44","vendor":"Cisco","features":null,"location":"","os":"meraki-ap","status":"1","status_type":"ok","ignore":"0","ignore_until":null,"asset_tag":null,"disabled":"0","uptime":null,"last_rebooted":null,"force_discovery":"0","last_polled":"2023-04-21 06:36:32","last_discovered":"2023-04-21 05:29:57","last_alerter":"2023-04-21 06:36:32","last_polled_timetaken":"17.80","last_discovered_timetaken":"7.34","purpose":null,"type":"wireless","serial":null,"icon":null,"distro":null,"distro_ver":null,"kernel":null,"arch":null,"location_id":"2063","location_lat":null,"location_lon":null,"location_country":"unknown","location_state":"Unknown","location_county":"Unknown","location_city":"Unknown","location_geoapi":"geocodefarm","location_status":"Geocoding ENABLED, try detect device coordinates:\nNOT REQUESTED\n  GEO API REQUEST: \n  QUERY DATE: Fri, 21 Apr 2023 05:31:04 +0000","location_manual":"0","location_updated":"2023-04-21 05:31:04","state":{"poller_mod_perf":{"system":1.0926,"os":0.0011,"sensors":0.0005,"status":0.0002,"counter":0.0002,"processors":0.0004,"mempools":0.0004,"storage":0.0027,"netstats":1.0839,"ucd-mib":0.0003,"ipSystemStats":0.43,"ports":4.6764,"ucd-diskio":0.0092,"wifi":0.0031,"p2p-radios":0.0006,"ospf":0.0001,"sla":0.0013,"pseudowires":0.0004,"mac-accounting":0.0004,"loadbalancer":0.0002,"entity-physical":0.0001,"applications":10.011,"graphs":0.0004,"oids":0.0004,"probes":0.0003},"poller_ports_perf":\{"IF-MIB":2.502121925354,"EtherLike-MIB":0.42258787155151,"Q-BRIDGE-MIB":1.3024079799652,"IP-MIB":0.43505191802979},"poller_history":\{"1682058974":17.8016,"1682058674":18.0325,"1682058367":18.0716,"1682058065":17.9358,"1682057760":18.1698,"1682057459":17.9337,"1682057151":17.8016,"1682056888":18.9871,"1682056590":9.754,"1682056289":9.8054,"1682055988":9.3644,"1682055688":10.0494,"1682055398":9.8505,"1682055111":8.2786,"1682055062":10.1984},"discovery_history":\{"1682054990":7.3438},"discovery_mod_perf":\{"os":0.4825,"mibs":0.9378,"vrf":0.0008,"ports":1.1594,"ports-stack":0.2342,"vlans":0.4422,"ports_vlans":null,"oids":0.0004,"ip-addresses":1.3218,"processors":0.0006,"mempools":0.0006,"inventory":0.2418,"sensors":0.236,"storage":0.0005,"neighbours":0.6842,"arp-table":1.1065,"mac-accounting":0.0002,"sla":0.0005,"pseudowires":0.0004,"virtual-machines":0.0003,"cisco-cbqos":0.2535,"ucd-diskio":0.2299,"wifi":0.0008,"p2p-radios":0.0001,"graphs":0.0002}},"row_class":"","html_row_class":"up","os_text":"Cisco Meraki AP","human_local_as":null,"humanized":true,"graphs":\{"availability":{"device_graph_id":"891076","device_id":"1789","graph":"availability","enabled":"1"},"ping":\{"device_graph_id":"891077","device_id":"1789","graph":"ping","enabled":"1"},"ping_snmp":\{"device_graph_id":"891078","device_id":"1789","graph":"ping_snmp","enabled":"1"},"bits":\{"device_graph_id":"891079","device_id":"1789","graph":"bits","enabled":"1"},"poller_perf":\{"device_graph_id":"891080","device_id":"1789","graph":"poller_perf","enabled":"1"},"pollersnmp_count":\{"device_graph_id":"891081","device_id":"1789","graph":"pollersnmp_count","enabled":"1"},"pollersnmp_times":\{"device_graph_id":"891082","device_id":"1789","graph":"pollersnmp_times","enabled":"1"},"pollersnmp_errors_count":\{"device_graph_id":"891083","device_id":"1789","graph":"pollersnmp_errors_count","enabled":"1"},"pollersnmp_errors_times":\{"device_graph_id":"891084","device_id":"1789","graph":"pollersnmp_errors_times","enabled":"1"},"pollerdb_count":\{"device_graph_id":"891085","device_id":"1789","graph":"pollerdb_count","enabled":"1"},"pollerdb_times":\{"device_graph_id":"891086","device_id":"1789","graph":"pollerdb_times","enabled":"1"},"pollermemory_perf":\{"device_graph_id":"891087","device_id":"1789","graph":"pollermemory_perf","enabled":"1"}}},"device_id":"1789"} # curl --no-progress-meter -u XXX:YYY https://apin008.ap-rdc01.nexperia.com/api/v0/devices/ | fgrep '"1789"' Does this help isolating the issue?

          I can't reproduce this, I have 119 devices, the query returns 119 devices and the count in the api return is 119 devices.

          what is the "query" string being returned by the api? what does that query return in your database?

          adama Adam Armstrong added a comment - I can't reproduce this, I have 119 devices, the query returns 119 devices and the count in the api return is 119 devices. what is the "query" string being returned by the api? what does that query return in your database?

          People

            adama Adam Armstrong
            O Helge Oldach
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: