Uploaded image for project: 'Observium'
  1. Observium
  2. OBS-4871

Problems polling OcNOS and Huawei devices

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • None
    • Professional Edition
    • Poller
    • None
    • Linux, nginx, php 7.4.33

    Description

      Hello all,

      Lately we are getting problems polling OcNOS and Huawei devices and it happens sporadically. We didn't changed anything on the device's side. The error from poller is below

       

       
        ___   _                              _
       / _ \ | |__   ___   ___  _ __ __   __(_) _   _  _ __ ___
      | | | || '_ \ / __| / _ \| '__|\ \ / /| || | | || '_ ` _ \
      | |_| || |_) |\__ \|  __/| |    \ V / | || |_| || | | | | |
       \___/ |_.__/ |___/ \___||_|     \_/  |_| \__,_||_| |_| |_|
                                Observium Professional 24.6.13582
                                        https://www.observium.org
       
      #####  Starting polling run at 2024-07-23 06:50:50  #####
       
      #####  caesium.del.eix.x3me.net [1484]  #####
       
       o OS                   ocnos
       o Last Polled
       o SNMP Version         v2c
       o Device status        Device is reachable by PING (0.30ms) and SNMP (10.28ms)
       o Modules Excluded     unix-agent, wmi, ipmi, junose-atm-vp, cisco-ipsec-flow-monitor, cisco-remote-access-monitor, cisco-cef,
                              arista-software-ip-forwarding,cipsec-tunnels, cisco-cbqos, cisco-eigrp, aruba-controller, cisco-vpdn,
                              netscaler-vsvr
       o Modules Disabled     applications, netstats, ucd-mib, ipSystemStats, printersupplies, ucd-diskio, wifi, p2p-radios, lsp, pseudowires,
                              mac-accounting,loadbalancer, entity-physical, fdb-table, oids, syslog, hr-mib
       o Modules Enabled      system, os, sensors, status, counter, processors, mempools, storage, ports, bgp-peers, ospf, sla, graphs, processes, probes
       
      #####  Module Start: system  #####
       
       o SNMPv2-MIB
       o Uptime               259 days, 19h 33m 6s
       o Last reboot          2023-11-06 11:17:44
       o Load average         0.5, 0.38, 0.42
       o sysObjectID          .1.3.6.1.4.1.36673.100.1.2.1.1.2
       o snmpEngineID         80001F880334EFB66A024C
       o sysDescr             Hardware Model:EC_AS7716-24SC, Software version: OcNOS,6.3.3.41
       o sysName              caesium.del
       
       
       
          ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣀⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣠⡶⠋⠁⠙⣦⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      ⠀⣠⡶⠛⠛⠲⣤⣀⠀⠀⠀⠀⡀⠀⠀⠀⠀⠀⠀⠀⣠⠄⣠⣾⣡⠀⠀⠀⠀⠸⡆⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      ⢰⡏⠀⠀⠀⠀⠈⣿⣷⣤⡐⣄⣽⡾⠧⠤⠤⣤⣤⣾⣿⣾⣿⣯⣀⡀⠀⠀⠀⢠⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      ⠀⢷⡀⠀⠀⠀⠀⣿⣿⣿⣿⣿⣯⣤⣤⡀⣠⣈⣹⣽⣿⣿⣿⣿⣷⣦⣄⠀⢀⡾⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      ⠀⠈⠻⣆⠀⠀⠀⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡟⠀⣠⠞⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      ⠀⠀⠀⠈⢳⣆⠘⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣧⣾⣷⣶⣤⣀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      ⠀⠀⠀⠀⠀⣘⣿⣿⣿⣿⣿⣟⣿⣿⣿⣿⣿⣿⣿⣿⢡⡼⣿⣿⣿⣿⣿⣿⣿⣿⣿⣷⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      ⠀⠀⠀⠀⣾⣿⣿⣿⣿⣿⡏⣯⡇⣿⣿⣿⡏⣿⣿⣿⣜⣛⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡷⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      ⠀⠀⠀⠀⣿⣿⣿⣿⣿⣿⣿⣭⣾⣿⣿⣿⣇⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡿⠃⠀⠘⢷⣄⠀⠀⠀⠀⠀⠀⠀⠀
      ⠀⠀⠀⢠⣿⣿⡏⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡏⢿⣿⣿⡿⠁⠀⠀⠀⠸⣿⣷⡀⠀⠀⠀⠀⠀⠀
      ⠀⠀⠀⢸⣿⣿⡇⠈⢻⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠻⣿⣿⡟⢠⣿⣿⡟⠁⠀⠀⠀⠀⠙⣻⣿⣿⣆⠀⠀⠀⠀⠀
      ⠀⠀⠀⠈⣿⣿⣷⡈⠛⠿⣿⣿⡆⠻⢿⣿⣽⣿⣯⣿⠿⠃⠀⠻⠛⣰⣿⣿⠏⠀⠀⠀⠀⢀⣙⣿⣿⣿⣿⣿⣧⡀⠀⠀⠀
      ⠀⠀⠀⠀⢿⠈⠙⠻⠷⠤⠄⢉⡁⠀⠀⠙⢉⣵⣌⠀⠀⠀⣀⠄⠘⠛⠋⠁⢀⣀⣤⣤⣭⣭⣽⣿⣿⣿⣿⣿⣿⣷⡀⠀⠀
      ⠀⠀⠀⢀⣿⡄⠀⠀⠀⠀⠀⠀⠉⠓⢦⣄⡉⠉⢉⣩⠶⠋⠁⠀⠀⠀⣠⣾⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣷⡀⠀
      ⠀⠀⠀⣼⣿⣿⣷⣶⣶⣆⣀⡀⠀⠀⠀⠀⠉⠉⠉⠀⠀⠀⠀⠀⣠⣾⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣧⠀
      ⠀⠀⠀⣿⣿⣿⣿⣿⣿⣿⣥⣄⡀⠀⠀⠀⠀⠀⠀⠀⠀⣸⣦⣾⣿⣿⣿⣿⣿⣿⡿⠿⠿⡟⠿⣿⣟⡿⢿⡛⠀⣘⣿⣿⡆
      ⠀⠀⠀⢿⣿⣿⠿⠿⠿⠿⢿⣿⣿⣷⣿⡆⠀⠀⣶⣶⣾⡿⠛⠛⠛⠛⠻⣿⣿⣿⡟⠉⠀⠀⠀⠈⠁⠁⢀⢀⠲⣬⣿⣿⡇
      ⠀⠀⠀⠘⡏⠳⣤⡄⠀⡄⠀⠀⠈⠙⢿⣧⠀⢀⣿⣿⠏⠀⠀⠀⠀⢀⣠⣿⣿⠋⠀⠀⠀⠀⠀⠀⠀⠀⠈⣯⣷⣿⣿⣿⡇
      ⠀⠀⠀⠀⢿⠀⠀⠉⠳⣇⠀⠀⠀⠀⠀⢻⠀⢸⣿⣿⣆⢠⠀⡆⣰⡿⠟⠉⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⡾⣿⣿⣿⣿⣿⡇
      ⠀⠀⠀⠀⠘⣧⠀⠀⠀⠹⣷⣤⣄⠀⡈⣾⡆⢸⡿⠋⠿⠾⠴⠷⠋⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣠⠞⠉⣶⣾⣿⣿⣿⣿⠀
      ⠀⠀⠀⠀⠀⠈⠳⣄⠀⠀⠀⠉⠻⣿⣿⣿⣇⠸⡷⠀⠀⠀⢀⣠⢄⣠⣶⣤⣤⣤⡤⠤⠖⠛⠉⠀⠠⣤⣼⣿⣿⣿⣿⡏⠀
      ⠀⠀⠀⠀⠀⠀⠀⠉⢻⣷⠶⠒⠀⠀⠁⠈⢿⡄⣷⣆⣤⣶⡿⠛⠛⠛⠉⠉⠉⠁⠀⠀⠀⠀⠀⠀⠀⠈⢿⣿⣿⣿⣿⠇⠀
      ⠀⠀⠀⠀⠀⠀⠀⠀⠘⣏⠀⠀⠀⠀⠀⠀⠘⣧⣿⣿⡿⢋⡤⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠰⣾⣿⣿⣿⣿⠀⠀
      ⠀⠀⠀⠀⠀⠀⠀⠀⠀⢻⡄⠀⠀⠀⠀⠀⠀⢸⣿⣿⣶⣫⠆⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡀⢦⣵⣿⣿⣿⣿⠇⠀⠀
      ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢻⣆⠀⠀⠀⠀⠀⠀⣿⣿⣿⠇⠀⣀⠀⠀⠀⠀⠀⠀⠀⠀⠳⠦⣴⣿⣶⣿⣿⣿⣿⡿⠀⠀⠀
      ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠛⢷⣄⠀⠀⠀⠀⢹⣿⣿⣾⣿⣋⣤⡄⠀⠀⠀⠀⠀⠀⠀⠠⣈⠻⣿⣿⣿⣿⣿⡇⠀⠀⠀
      ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠉⠛⣦⣤⣤⣾⣿⣿⣿⣿⣿⣯⡴⠂⣀⡀⠀⡀⡀⡀⢠⣬⣻⣿⣿⣿⣿⡿⠀⠀⠀⠀
      ⠀⠀⠀⠀⠀⠀⠀⢀⣀⣀⣀⣀⣠⡴⠊⠉⠀⠈⢹⠿⣿⣿⣿⣿⣿⣷⣾⣿⠾⡿⡿⠿⣷⣦⣿⣿⣿⣿⣿⡟⠁⠀⠀⠀⠀
      ⠀⠀⠀⠀⠀⠀⠀⠛⠷⠦⠤⠤⠴⠶⠶⠶⠶⠚⠉⠀⠀⠈⣉⣩⠽⠟⠋⠁⠀⠀⣁⣠⠿⠛⠋⠉⠉⠉⠁⠀⠀⠀⠀⠀⠀
      ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠉⠉⠉⠉⠉⠉⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      Oops! Something went wrong! You may want to report this to the Observium developers.
       
      Call to undefined function mb_ord() in /opt/observium/includes/common.inc.php:4972
       
      Stack trace:
       
      #0: (unknown function) called at [/opt/observium/includes/common.inc.php:4972]
      4967: * "\u00cb\u0419\u0426\u0423\u041a\u0415\u041d\u0413\u0428\u0429\u0417\u0425\u042a\u0424\u042b\u0412\u0410\u041f\u0420\u041e\u041b\u0414\u0416\u042d\u042f\u0427\u0421\u041c\u0418\u0422\u042c\u0411\u042e"
      4968: */
      4969: function fix_json_unicode($string)
      4970: {
      4971: return preg_replace_callback('/([\x\{0080}-\x\{FFFF}])/u', function ($match) {
      4972: return '\\u' . str_pad(dechex(mb_ord($match[1], 'UTF-8')), 4, '0', STR_PAD_LEFT);
      4973: }, $string);
      4974: }
      4975:
      4976: function safe_unserialize($str)
      4977: {
       
      #1: \{closure} called at [(unknown file):(unknown line)]
       
      #2: preg_replace_callback called at [/opt/observium/includes/common.inc.php:4973]
      4968: */
      4969: function fix_json_unicode($string)
      4970: {
      4971: return preg_replace_callback('/([\x\{0080}-\x\{FFFF}])/u', function ($match) {
      4972: return '\\u' . str_pad(dechex(mb_ord($match[1], 'UTF-8')), 4, '0', STR_PAD_LEFT);
      4973: }, $string);
      4974: }
      4975:
      4976: function safe_unserialize($str)
      4977: {
      4978: if (is_array($str)) {
       
      #3: fix_json_unicode called at [/opt/observium/includes/common.inc.php:4946]
      4941: // Try to fix "Control character error, possibly incorrectly encoded"
      4942: $str_fix = preg_replace('/[[:cntrl:]]/', '', smart_quotes($str));
      4943: print_debug_vars($str_fix);
      4944: } else {
      4945: // Try fix utf errors
      4946: $str_fix = fix_json_unicode(smart_quotes($str));
      4947: print_debug_vars($str_fix);
      4948: }
      4949: $json_fix = @json_decode($str_fix, TRUE, 512, $options);
      4950: if (json_last_error() === JSON_ERROR_NONE) {
      4951: //print_vars(smart_quotes(fix_json_unicode($str)));
       
      #4: safe_json_decode called at [/opt/observium/includes/http.inc.php:611]
      606:
      607: // If response return valid code and content, additional parse for specific defined tests
      608: if ($success) {
      609: // Decode if request OK
      610: if ($is_response_array) {
      611: $response = safe_json_decode($response);
      612: }
      613: // else additional formats?
      614:
      615: // Check if call succeeded
      616: if (isset($def['response_test'])) {
       
      #5: test_http_request called at [/opt/observium/includes/http.inc.php:712]
      707: $request_status = FALSE;
      708: // Send out API call and parse response
      709: for ($retry = 1; $retry <= (int)$request_retry; $retry++) {
      710: print_debug("Request [$url] #$retry:");
      711: $response = get_http_request($url, $options, $ratelimit);
      712: if ($request_status = test_http_request($def, $response)) {
      713: // stop for on success
      714: return $request_status;
      715: }
      716: // wait little time
      717: sleep($request_sleep);
       
      #6: process_http_request called at [/opt/observium/includes/geolocation.inc.php:339]
      334:
      335: // API URL to POST to
      336: $url = generate_http_url($geo_def, $tags);
      337:
      338: // First or second request
      339: if (process_http_request($geo_def, $url, $options, $mapresponse)) {
      340:
      341: switch ($geo_def['response_format']) {
      342: case 'xml':
      343: // Hrm, currently unused
      344: break;
       
      #7: get_geo_http_def called at [/opt/observium/includes/geolocation.inc.php:142]
      137: $address_second = $geo_type === 'forward' ? generate_location_alt($address) : NULL;
      138:
      139: // Generate geolocation tags, used for rewrites in definition
      140: $tags = generate_geolocation_tags($location['location_geoapi'], $location);
      141:
      142: $data = get_geo_http_def($geo_def, $geo_type, $tags, $debug_msg);
      143:
      144: if (!$data) {
      145: // Return an old array with new status (for later recheck)
      146: unset($geo_db['hostname'], $geo_db['location_updated']);
      147: $location['location_status']  = $debug_msg;
       
      #8: get_geolocation called at [/opt/observium/includes/polling/system.inc.php:311]
      306: print_debug_vars($geo_db);
      307:
      308: $geo_detect = geo_detect($device, $poll_device, $geo_db, $dns_only);
      309:
      310: if ($geo_detect || $dns_only) {
      311: $update_geo = get_geolocation($poll_device['sysLocation'], $geo_db, $dns_only);
      312: if ($update_geo) {
      313: print_debug_vars($update_geo, 1);
      314: if (is_numeric($update_geo['location_lat']) && is_numeric($update_geo['location_lon']) && $update_geo['location_country'] !== 'Unknown') {
      315: $geo_msg = 'Geolocation (' . strtoupper($update_geo['location_geoapi']) . ') -> ';
      316: $geo_msg .= '[' . sprintf('%f', $update_geo['location_lat']) . ', ' . sprintf('%f', $update_geo['location_lon']) . '] ';
       
      #9: include called at [/opt/observium/includes/polling/functions.inc.php:314]
      309:
      310: print_cli_heading("Module Start: %R" . $module);
      311:
      312: $m_start = utime();
      313:
      314: include($config['install_dir'] . "/includes/polling/$module.inc.php");
      315:
      316: $m_run                                    = elapsed_time($m_start, 4);
      317: $device_state['poller_mod_perf'][$module] = $m_run;
      318: print_cli_data("Module time", format_number_short($m_run, 4) . "s");
      319:
       
      #10: poll_device called at [/opt/observium/poller.php:189]
      184: $params[] = $config['poller_id'];
      185: }
      186:
      187: foreach (dbFetchColumn($query, $params) as $device_id) {
      188: $device = dbFetchRow("SELECT * FROM `devices` WHERE `device_id` = ?", [ $device_id ]);
      189: poll_device($device, $options);
      190: $polled_devices++;
      191: }
      192:
      193: $poller_time = elapsed_time($poller_start, 4);
      194:
      

      Attachments

        Issue Links

          Activity

            People

              landy Mike Stupalov
              ymarinov Yavor Marinov
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: