Uploaded image for project: 'Observium'
  1. Observium
  2. OBS-4942

nvidia status error

    XMLWordPrintable

Details

    • Paid Support
    • Resolution: Fixed
    • Minor
    • None
    • Professional Edition, Enterprise Edition
    • Unix Agent

    Description

      Getting the following error during polling host:

      / _ \ | |__   ___   ___  _ __ __   __(_) _   _  _ __ ___
      | | | || '_ \ / __| / _ \| '__|\ \ / /| || | | || '_ ` _ \
      | |_| || |_) |\__ \|  __/| |    \ V / | || |_| || | | | | |
       \___/ |_.__/ |___/ \___||_|     \_/  |_| \__,_||_| |_| |_|
                               Observium Professional 24.12.13776
                                        https://www.observium.org
       
      #####  Starting polling run at 2024-12-10 00:27:56  #####
       
      #####  robotech.hpc.nist.gov [1270]  #####
       
       o OS                   linux
       o OS Group             unix
       o Last poll duration   26.01 seconds
       o Last Polled          2024-12-10 00:27:27
       o SNMP Version         v2c
       o Device status        Device is reachable by PING (0.213ms) and SNMP (17.73ms)
       o Modules Excluded     wmi, junose-atm-vp, cisco-ipsec-flow-monitor, cisco-remote-access-monitor, cisco-cef, 
                              arista-software-ip-forwarding,cipsec-tunnels, cisco-cbqos, cisco-eigrp, aruba-controller, cisco-vpdn 
       o Modules Disabled     ipmi, bgp-peers, printersupplies, wifi, ospf, lsp
       o Modules Enabled      system, os, unix-agent, applications, sensors, status, counter, processors, mempools, storage, netstats, ucd-mib, 
                              ipSystemStats,ports, ucd-diskio, p2p-radios, sla, pseudowires, mac-accounting, loadbalancer, entity-physical, 
                              fdb-table,graphs, oids, processes, probes, syslog 
       
      #####  Module Start: unix-agent  #####
       
       o UNIX Agent execution time 24305ms
       
      DEB Packages: ..............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
       o Diskstats            
      L.L.L............L.L..L.......................L..................L.......#####  nvidia-smi  #####
       
      #####  Nvidia Card 1: NVIDIA GeForce RTX 3080 Ti  #####
       
      U o temperature.gpu      34C
      LU o power.draw           26.37 WW
      U o fan.speed            33 %n
      U o utilization.gpu      0 %n
      U o utilization.memory   32 %n
       
          ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣀⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣠⡶⠋⠁⠙⣦⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      ⠀⣠⡶⠛⠛⠲⣤⣀⠀⠀⠀⠀⡀⠀⠀⠀⠀⠀⠀⠀⣠⠄⣠⣾⣡⠀⠀⠀⠀⠸⡆⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      ⢰⡏⠀⠀⠀⠀⠈⣿⣷⣤⡐⣄⣽⡾⠧⠤⠤⣤⣤⣾⣿⣾⣿⣯⣀⡀⠀⠀⠀⢠⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      ⠀⢷⡀⠀⠀⠀⠀⣿⣿⣿⣿⣿⣯⣤⣤⡀⣠⣈⣹⣽⣿⣿⣿⣿⣷⣦⣄⠀⢀⡾⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      ⠀⠈⠻⣆⠀⠀⠀⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡟⠀⣠⠞⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      ⠀⠀⠀⠈⢳⣆⠘⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣧⣾⣷⣶⣤⣀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      ⠀⠀⠀⠀⠀⣘⣿⣿⣿⣿⣿⣟⣿⣿⣿⣿⣿⣿⣿⣿⢡⡼⣿⣿⣿⣿⣿⣿⣿⣿⣿⣷⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      ⠀⠀⠀⠀⣾⣿⣿⣿⣿⣿⡏⣯⡇⣿⣿⣿⡏⣿⣿⣿⣜⣛⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡷⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      ⠀⠀⠀⠀⣿⣿⣿⣿⣿⣿⣿⣭⣾⣿⣿⣿⣇⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡿⠃⠀⠘⢷⣄⠀⠀⠀⠀⠀⠀⠀⠀
      ⠀⠀⠀⢠⣿⣿⡏⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡏⢿⣿⣿⡿⠁⠀⠀⠀⠸⣿⣷⡀⠀⠀⠀⠀⠀⠀
      ⠀⠀⠀⢸⣿⣿⡇⠈⢻⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠻⣿⣿⡟⢠⣿⣿⡟⠁⠀⠀⠀⠀⠙⣻⣿⣿⣆⠀⠀⠀⠀⠀
      ⠀⠀⠀⠈⣿⣿⣷⡈⠛⠿⣿⣿⡆⠻⢿⣿⣽⣿⣯⣿⠿⠃⠀⠻⠛⣰⣿⣿⠏⠀⠀⠀⠀⢀⣙⣿⣿⣿⣿⣿⣧⡀⠀⠀⠀
      ⠀⠀⠀⠀⢿⠈⠙⠻⠷⠤⠄⢉⡁⠀⠀⠙⢉⣵⣌⠀⠀⠀⣀⠄⠘⠛⠋⠁⢀⣀⣤⣤⣭⣭⣽⣿⣿⣿⣿⣿⣿⣷⡀⠀⠀
      ⠀⠀⠀⢀⣿⡄⠀⠀⠀⠀⠀⠀⠉⠓⢦⣄⡉⠉⢉⣩⠶⠋⠁⠀⠀⠀⣠⣾⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣷⡀⠀
      ⠀⠀⠀⣼⣿⣿⣷⣶⣶⣆⣀⡀⠀⠀⠀⠀⠉⠉⠉⠀⠀⠀⠀⠀⣠⣾⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣧⠀
      ⠀⠀⠀⣿⣿⣿⣿⣿⣿⣿⣥⣄⡀⠀⠀⠀⠀⠀⠀⠀⠀⣸⣦⣾⣿⣿⣿⣿⣿⣿⡿⠿⠿⡟⠿⣿⣟⡿⢿⡛⠀⣘⣿⣿⡆
      ⠀⠀⠀⢿⣿⣿⠿⠿⠿⠿⢿⣿⣿⣷⣿⡆⠀⠀⣶⣶⣾⡿⠛⠛⠛⠛⠻⣿⣿⣿⡟⠉⠀⠀⠀⠈⠁⠁⢀⢀⠲⣬⣿⣿⡇
      ⠀⠀⠀⠘⡏⠳⣤⡄⠀⡄⠀⠀⠈⠙⢿⣧⠀⢀⣿⣿⠏⠀⠀⠀⠀⢀⣠⣿⣿⠋⠀⠀⠀⠀⠀⠀⠀⠀⠈⣯⣷⣿⣿⣿⡇
      ⠀⠀⠀⠀⢿⠀⠀⠉⠳⣇⠀⠀⠀⠀⠀⢻⠀⢸⣿⣿⣆⢠⠀⡆⣰⡿⠟⠉⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⡾⣿⣿⣿⣿⣿⡇
      ⠀⠀⠀⠀⠘⣧⠀⠀⠀⠹⣷⣤⣄⠀⡈⣾⡆⢸⡿⠋⠿⠾⠴⠷⠋⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣠⠞⠉⣶⣾⣿⣿⣿⣿⠀
      ⠀⠀⠀⠀⠀⠈⠳⣄⠀⠀⠀⠉⠻⣿⣿⣿⣇⠸⡷⠀⠀⠀⢀⣠⢄⣠⣶⣤⣤⣤⡤⠤⠖⠛⠉⠀⠠⣤⣼⣿⣿⣿⣿⡏⠀
      ⠀⠀⠀⠀⠀⠀⠀⠉⢻⣷⠶⠒⠀⠀⠁⠈⢿⡄⣷⣆⣤⣶⡿⠛⠛⠛⠉⠉⠉⠁⠀⠀⠀⠀⠀⠀⠀⠈⢿⣿⣿⣿⣿⠇⠀
      ⠀⠀⠀⠀⠀⠀⠀⠀⠘⣏⠀⠀⠀⠀⠀⠀⠘⣧⣿⣿⡿⢋⡤⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠰⣾⣿⣿⣿⣿⠀⠀
      ⠀⠀⠀⠀⠀⠀⠀⠀⠀⢻⡄⠀⠀⠀⠀⠀⠀⢸⣿⣿⣶⣫⠆⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡀⢦⣵⣿⣿⣿⣿⠇⠀⠀
      ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢻⣆⠀⠀⠀⠀⠀⠀⣿⣿⣿⠇⠀⣀⠀⠀⠀⠀⠀⠀⠀⠀⠳⠦⣴⣿⣶⣿⣿⣿⣿⡿⠀⠀⠀
      ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠛⢷⣄⠀⠀⠀⠀⢹⣿⣿⣾⣿⣋⣤⡄⠀⠀⠀⠀⠀⠀⠀⠠⣈⠻⣿⣿⣿⣿⣿⡇⠀⠀⠀
      ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠉⠛⣦⣤⣤⣾⣿⣿⣿⣿⣿⣯⡴⠂⣀⡀⠀⡀⡀⡀⢠⣬⣻⣿⣿⣿⣿⡿⠀⠀⠀⠀
      ⠀⠀⠀⠀⠀⠀⠀⢀⣀⣀⣀⣀⣠⡴⠊⠉⠀⠈⢹⠿⣿⣿⣿⣿⣿⣷⣾⣿⠾⡿⡿⠿⣷⣦⣿⣿⣿⣿⣿⡟⠁⠀⠀⠀⠀
      ⠀⠀⠀⠀⠀⠀⠀⠛⠷⠦⠤⠤⠴⠶⠶⠶⠶⠚⠉⠀⠀⠈⣉⣩⠽⠟⠋⠁⠀⠀⣁⣠⠿⠛⠋⠉⠉⠉⠁⠀⠀⠀⠀⠀⠀
      ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠉⠉⠉⠉⠉⠉⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      Oops! Something went wrong! You may want to report this to the Observium developers.
       
      Unsupported operand types: string + int in /opt/observium/includes/polling/unix-agent/nvidia.inc.php:24
       
      Stack trace:
       
      #0: (unknown function) called at [/opt/observium/includes/polling/unix-agent/nvidia.inc.php:24]
      19: if (safe_count($nvidia)) {
      20: 
      21: print_cli_heading("nvidia-smi", 3);
      22: foreach ($nvidia as $card) {
      23: 
      24: $descr_card = "Nvidia Card " . ($card['index'] + 1) . ": " . $card['name'];
      25: print_cli_heading($descr_card, 4);
      26: 
      27: if (!in_array($card['temperature.gpu'], $invalid)) {
      28: $index = 'temperature.gpu.' . $card['index'];
      29: $descr = $descr_card;
       
      #1: include called at [/opt/observium/includes/polling/unix-agent.inc.php:34]
      29: 
      30: foreach (array_keys($agent_data) as $key) {
      31: if (file_exists($config['install_dir'] . '/includes/polling/unix-agent/' . $key . '.inc.php')) {
      32: print_debug('Including: unix-agent/' . $key . '.inc.php');
      33: 
      34: include($config['install_dir'] . '/includes/polling/unix-agent/' . $key . '.inc.php');
      35: } else {
      36: //print_warning("No include: ".$key);
      37: }
      38: }
      39: 
       
      #2: include called at [/opt/observium/includes/polling/functions.inc.php:316]
      311: 
      312: print_cli_heading("Module Start: %R" . $module);
      313: 
      314: $m_start = utime();
      315: 
      316: include($config['install_dir'] . "/includes/polling/$module.inc.php");
      317: 
      318: $m_run                                    = elapsed_time($m_start, 4);
      319: $device_state['poller_mod_perf'][$module] = $m_run;
      320: print_cli_data("Module time", format_number_short($m_run, 4) . "s");
      321: 
       
      #3: poll_device called at [/opt/observium/poller.php:189]
      184: $params[] = $config['poller_id'];
      185: }
      186: 
      187: foreach (dbFetchColumn($query, $params) as $device_id) {
      188: $device = dbFetchRow("SELECT * FROM `devices` WHERE `device_id` = ?", [ $device_id ]);
      189: poll_device($device, $options);
      190: $polled_devices++;
      191: }
      192: 
      193: $poller_time = elapsed_time($poller_start, 4);
      

       

      On Host: (robotech.hpc.nist.gov)

      RUN: 

      root@Robotech:~# /usr/lib/observium_agent/scripts-available/nvidia-smi 
      <<<nvidia-smi>>>
      index, name, temperature.gpu, fan.speed [%], power.draw [W], utilization.gpu [%], utilization.memory [%]
      0, NVIDIA GeForce RTX 3080 Ti, 35, 32 %, 26.86 W, 2 %, 30 %
      
      

      Attachments

        Activity

          People

            landy Mike Stupalov
            mwilliamsr-net Mark Williams
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: