DEBUG! SQL[SELECT `attrib_value` FROM `observium_attribs` WHERE `attrib_type` = 'dbSchema';] SQL RUNTIME[0.00013900s] SQL[SELECT `attrib_value` FROM `observium_attribs` WHERE `attrib_type` = 'current_rev';] SQL RUNTIME[0.00008702s] SQL[SELECT * FROM `config`] SQL RUNTIME[0.00006080s] CMD[/bin/hostname -f]  CMD EXITCODE[0] CMD RUNTIME[0.0025s] STDOUT[ lobservium01 ]  ___ _ _ / _ \ | |__ ___ ___ _ __ __ __(_) _ _ _ __ ___ | | | || '_ \ / __| / _ \| '__|\ \ / /| || | | || '_ ` _ \ | |_| || |_) |\__ \| __/| | \ V / | || |_| || | | | | | \___/ |_.__/ |___/ \___||_| \_/ |_| \__,_||_| |_| |_| Observium Professional 0.16.3.7703 http://www.observium.org  SQL[SELECT `attrib_value` FROM `observium_attribs` WHERE `attrib_type` = 'latest_ver';] SQL RUNTIME[0.00005293s] SQL[SELECT `attrib_value` FROM `observium_attribs` WHERE `attrib_type` = 'latest_rev';] SQL RUNTIME[0.00005984s] SQL[SELECT `attrib_value` FROM `observium_attribs` WHERE `attrib_type` = 'latest_rev_date';] SQL RUNTIME[0.00119209s] There is a newer revision of Observium available! CMD[date "+%:z"]  CMD EXITCODE[0] CMD RUNTIME[0.0017s] CMD EXITDELAY[10ms] STDOUT[ +01:00 ] SQL[SELECT TIMEDIFF(NOW(), UTC_TIMESTAMP);] SQL RUNTIME[0.00009894s] Version 0.16.5.7823 (5th May 2016) is 120 revisions ahead.  CMD[/data/observium/scripts/distro]  CMD EXITCODE[0] CMD RUNTIME[0.0127s] STDOUT[ Linux|3.10.0-327.el7.x86_64|amd64|CentOS|7.2.1511 ] CMD[/usr/bin/env python --version 2>&1]  CMD EXITCODE[0] CMD RUNTIME[0.0031s] STDOUT[ Python 2.7.5 ] SQL[SELECT version();] SQL RUNTIME[0.00009584s] CMD[/usr/bin/snmpget --version 2>&1]  CMD EXITCODE[0] CMD RUNTIME[0.0055s] CMD EXITDELAY[10ms] STDOUT[ NET-SNMP version: 5.7.2 ] CMD[/usr/bin/rrdtool --version |head -n1]  CMD EXITCODE[0] CMD RUNTIME[0.007s] STDOUT[ RRDtool 1.4.8 Copyright 1997-2013 by Tobias Oetiker ] CMD[/usr/bin/env apache2 -v | awk '/Server version:/ {print $3}']  CMD EXITCODE[0] CMD RUNTIME[0.0034s] STDOUT[ ] CMD[/usr/bin/env httpd -v | awk '/Server version:/ {print $3}']  CMD EXITCODE[0] CMD RUNTIME[0.0331s] STDOUT[ Apache/2.4.6 ] SQL[SELECT @@sql_mode;] SQL RUNTIME[0.00009799s] SQL[SHOW VARIABLES LIKE 'character_set_connection'] SQL RUNTIME[0.00044608s] ##### Software versions #####   o OS  Linux 3.10.0-327.el7.x86_64 [amd64] (CentOS 7.2.1511)  o Apache  2.4.6  o PHP  7.0.6  o Python  2.7.5  o MySQL  5.5.47-MariaDB-log (extension: mysqli 5.0.12-dev)  o SNMP  NET-SNMP 5.7.2  o RRDtool  1.4.8 ##### MySQL mode #####   o MySQL  ##### Charset info #####   o PHP  UTF-8  o MySQL  utf8 ##### Timezones info #####   o Date  Thursday, 05-May-16 15:00:20 BST  o PHP  +01:00  o MySQL  +01:00 SQL[SELECT * FROM `alerts_maint` WHERE `maint_start` < '1462456820' AND `maint_end` > '1462456820'] SQL RUNTIME[0.00041604s] ##### Starting polling run at 2016-05-05 15:00:20 #####  SQL[SELECT `device_id` FROM `devices` WHERE `disabled` = 0 AND `hostname` LIKE 'lgpu003%' ORDER BY `device_id` ASC] SQL RUNTIME[0.00133300s] SQL[SELECT * FROM `devices` WHERE `device_id` = '2810'] SQL RUNTIME[0.00152111s] SQL[SELECT * FROM `entity_attribs` WHERE `entity_type` = 'device' AND `entity_id` = '2810'] SQL RUNTIME[0.00005817s] SQL[SELECT * FROM `observium_processes` WHERE `process_name` = 'poller.php' AND `device_id` = '2810'] SQL RUNTIME[0.00093699s] CMD[/bin/ps -ww -o pid,ppid,uid,gid,tty,stat,time,lstart,args -p 6528]  CMD EXITCODE[0] CMD RUNTIME[0.0186s] CMD EXITDELAY[10ms] STDOUT[ PID PPID UID GID TT STAT TIME STARTED COMMAND 6528 28608 0 0 pts/0 S+ 00:00:00 Thu May 5 15:00:19 2016 php ./poller.php -m unix-agent -d -h lgpu003* ] CMD[/bin/ps -ww -o pid,ppid,uid,gid,tty,stat,time,lstart,args -p 28608]  CMD EXITCODE[0] CMD RUNTIME[0.0252s] CMD EXITDELAY[10ms] STDOUT[ PID PPID UID GID TT STAT TIME STARTED COMMAND 28608 28598 0 0 pts/0 Ss 00:00:00 Wed Apr 27 14:44:32 2016 -bash ] SQL[INSERT INTO `observium_processes` (`process_pid`,`process_name`,`process_ppid`,`process_uid`,`process_command`,`process_start`,`device_id`) VALUES ('6528','poller.php','28608','0','php ./poller.php -m unix-agent -d -h lgpu003*','1462456819','2810')] SQL RUNTIME[0.00025511s] SQL[SELECT * FROM `alert_tests` WHERE 1] SQL RUNTIME[0.00006795s] Cached 3 alert rules. SQL[SELECT *,`alert_table`.`alert_table_id` AS `alert_table_id` FROM `alert_table` WHERE `device_id` = '2810'] SQL RUNTIME[0.00182009s] ##### lgpu003.lon.compute.pgs.com [2810] #####   o OS  linux  o OS Group  unix  o Last poll duration  3.68 seconds  o Last Polled  2016-05-05 14:58:58  o SNMP Version  v2c CMD[/usr/sbin/fping -t 500 -c 1 -q 172.21.138.56 2>&1]  CMD EXITCODE[0] CMD RUNTIME[0.0025s] STDOUT[ 172.21.138.56 : xmt/rcv/%loss = 1/1/0%, min/avg/max = 0.06/0.06/0.06 ] CMD[/usr/bin/snmpget -v2c -c *** -Pu -OQUst -m SNMPv2-MIB -M /data/observium/mibs/rfc:/data/observium/mibs/net-snmp 'udp':'lgpu003.lon.compute.pgs.com':'161' sysObjectID.0 sysUpTime.0]  CMD EXITCODE[0] CMD RUNTIME[0.0089s] STDOUT[ sysObjectID.0 = enterprises.8072.3.2.10 sysUpTime.0 = 148609986 ] SNMP STATUS[TRUE]  o Device status  Device is reachable by PING (0.06ms) and SNMP (8.87ms) RRD CMD[update /data/observium/rrd/lgpu003.lon.compute.pgs.com/status.rrd N:1] RRD RUNTIME[0.0011s] RRD STDOUT[OK u:0.00 s:0.00 r:0.09] RRD_STATUS[TRUE]  RRD CMD[update /data/observium/rrd/lgpu003.lon.compute.pgs.com/ping.rrd N:0.06] RRD RUNTIME[0.0011s] RRD STDOUT[OK u:0.00 s:0.00 r:0.09] RRD_STATUS[TRUE]  RRD CMD[update /data/observium/rrd/lgpu003.lon.compute.pgs.com/ping_snmp.rrd N:8.87] RRD RUNTIME[0.0011s] RRD STDOUT[OK u:0.00 s:0.00 r:0.09] RRD_STATUS[TRUE]  SQL[SELECT * FROM `device_graphs` WHERE `device_id` = '2810'] SQL RUNTIME[0.00009012s]  o Modules Enabled  unix-agent, system, os including: includes/polling/unix-agent.inc.php ##### Module Start: unix-agent #####  Observium UNIX Agent: SQL[SELECT `attrib_value` FROM `entity_attribs` WHERE `entity_type` = 'device' AND `entity_id` = '2810' AND `attrib_type` = 'agent_port'] SQL RUNTIME[0.00005007s] execution time: 128ms RRD CMD[update /data/observium/rrd/lgpu003.lon.compute.pgs.com/agent.rrd N:128] RRD RUNTIME[0.0011s] RRD STDOUT[OK u:0.00 s:0.00 r:0.22] RRD_STATUS[TRUE]  > $agent_data - /data/observium/poller.php:163 /data/observium/includes/polling/unix-agent.inc.php:120 ==================================================== array( [] => string(0) "" [Observium] => string(180) "Version: 1.0.1 AgentOS: linux PluginsDirectory: /usr/lib/observium_agent/plugins LocalDirectory: /usr/lib/observium_agent/local AgentDirectory: /etc/observium OnlyFrom: 0.0.0.0 ::1" [df] => string(881) "/dev/md1 ext4 20010876 4430668 14557028 24% / /dev/md0 ext4 95003 74129 15758 83% /boot /dev/md3 ext2 1068306848 73000 1068233848 1% /scratch /dev/libcfs01 gpfs 1687509762048 1553300455424 134209306624 93% /gpfs/libcfs01 /dev/liccfs01 gpfs 1687509762048 1420015497216 267494264832 85% /gpfs/liccfs01 /dev/lidcfs01 gpfs 2250023780352 1587546140672 662477639680 71% /gpfs/lidcfs01 /dev/lidhqc01 gpfs 281251627008 33490636800 247760990208 12% /gpfs/lidhqc01 /dev/liecfs01 gpfs 2531275407360 2252017315840 279258091520 89% /gpfs/liecfs01 /dev/lhqc01 gpfs 1030878658560 469552205824 561326452736 46% /gpfs/lhqc01 /dev/lifcfs01 gpfs 2061757317120 1766504898560 295252418560 86% /gpfs/lifcfs01 /dev/ligcfs01 gpfs 2061757317120 1027651788800 1034105528320 50% /gpfs/ligcfs01" [mounts] => string(564) "/dev/md1 / ext4 rw,relatime,barrier=1,data=ordered 0 0 /dev/md0 /boot ext4 rw,relatime,barrier=1,data=ordered 0 0 /dev/md3 /scratch ext2 rw,relatime,errors=continue 0 0 /dev/libcfs01 /gpfs/libcfs01 gpfs rw,relatime 0 0 /dev/liccfs01 /gpfs/liccfs01 gpfs rw,relatime 0 0 /dev/lidcfs01 /gpfs/lidcfs01 gpfs rw,relatime 0 0 /dev/lidhqc01 /gpfs/lidhqc01 gpfs rw,relatime 0 0 /dev/liecfs01 /gpfs/liecfs01 gpfs rw,relatime 0 0 /dev/lhqc01 /gpfs/lhqc01 gpfs rw,relatime 0 0 /dev/lifcfs01 /gpfs/lifcfs01 gpfs rw,relatime 0 0 /dev/ligcfs01 /gpfs/ligcfs01 gpfs rw,relatime 0 0" [ps] => string(9933) "(root,31020,1968,0.0) /sbin/init (root,0,0,0.0) [kthreadd] (root,0,0,0.0) [migration/0] (root,0,0,0.0) [ksoftirqd/0] (root,0,0,0.0) [stopper/0] (root,0,0,0.0) [watchdog/0] (root,0,0,0.0) [migration/1] (root,0,0,0.0) [stopper/1] (root,0,0,0.0) [ksoftirqd/1] (root,0,0,0.0) [watchdog/1] (root,0,0,0.0) [migration/2] (root,0,0,0.0) [stopper/2] (root,0,0,0.0) [ksoftirqd/2] (root,0,0,0.0) [watchdog/2] (root,0,0,0.0) [migration/3] (root,0,0,0.0) [stopper/3] (root,0,0,0.0) [ksoftirqd/3] (root,0,0,0.0) [watchdog/3] (root,0,0,0.0) [migration/4] (root,0,0,0.0) [stopper/4] (root,0,0,0.0) [ksoftirqd/4] (root,0,0,0.0) [watchdog/4] (root,0,0,0.0) [migration/5] (root,0,0,0.0) [stopper/5] (root,0,0,0.0) [ksoftirqd/5] (root,0,0,0.0) [watchdog/5] (root,0,0,0.0) [migration/6] (root,0,0,0.0) [stopper/6] (root,0,0,0.0) [ksoftirqd/6] (root,0,0,0.0) [watchdog/6] (root,0,0,0.0) [migration/7] (root,0,0,0.0) [stopper/7] (root,0,0,0.0) [ksoftirqd/7] (root,0,0,0.0) [watchdog/7] (root,0,0,0.0) [events/0] (root,0,0,0.0) [events/1] (root,0,0,0.0) [events/2] (root,0,0,0.0) [events/3] (root,0,0,0.0) [events/4] (root,0,0,0.0) [events/5] (root,0,0,0.0) [events/6] (root,0,0,0.0) [events/7] (root,0,0,0.0) [cgroup] (root,0,0,0.0) [khelper] (root,0,0,0.0) [netns] (root,0,0,0.0) [async/mgr] (root,0,0,0.0) [pm] (root,0,0,0.0) [sync_supers] (root,0,0,0.0) [bdi-default] (root,0,0,0.0) [kintegrityd/0] (root,0,0,0.0) [kintegrityd/1] (root,0,0,0.0) [kintegrityd/2] (root,0,0,0.0) [kintegrityd/3] (root,0,0,0.0) [kintegrityd/4] (root,0,0,0.0) [kintegrityd/5] (root,0,0,0.0) [kintegrityd/6] (root,0,0,0.0) [kintegrityd/7] (root,0,0,0.0) [kblockd/0] (root,0,0,0.0) [kblockd/1] (root,0,0,0.0) [kblockd/2] (root,0,0,0.0) [kblockd/3] (root,0,0,0.0) [kblockd/4] (root,0,0,0.0) [kblockd/5] (root,0,0,0.0) [kblockd/6] (root,0,0,0.0) [kblockd/7] (root,0,0,0.0) [kacpid] (root,0,0,0.0) [kacpi_notify] (root,0,0,0.0) [kacpi_hotplug] (root,0,0,0.0) [ata_aux] (root,0,0,0.0) [ata_sff/0] (root,0,0,0.0) [ata_sff/1] (root,0,0,0.0) [ata_sff/2] (root,0,0,0.0) [ata_sff/3] (root,0,0,0.0) [ata_sff/4] (root,0,0,0.0) [ata_sff/5] (root,0,0,0.0) [ata_sff/6] (root,0,0,0.0) [ata_sff/7] (root,0,0,0.0) [ksuspend_usbd] (root,0,0,0.0) [khubd] (root,0,0,0.0) [kseriod] (root,0,0,0.0) [md/0] (root,0,0,0.0) [md/1] (root,0,0,0.0) [md/2] (root,0,0,0.0) [md/3] (root,0,0,0.0) [md/4] (root,0,0,0.0) [md/5] (root,0,0,0.0) [md/6] (root,0,0,0.0) [md/7] (root,0,0,0.0) [md_misc/0] (root,0,0,0.0) [md_misc/1] (root,0,0,0.0) [md_misc/2] (root,0,0,0.0) [md_misc/3] (root,0,0,0.0) [md_misc/4] (root,0,0,0.0) [md_misc/5] (root,0,0,0.0) [md_misc/6] (root,0,0,0.0) [md_misc/7] (root,0,0,0.0) [linkwatch] (root,0,0,0.0) [khungtaskd] (root,0,0,0.0) [kswapd0] (root,0,0,0.0) [ksmd] (root,0,0,0.0) [khugepaged] (root,0,0,0.0) [aio/0] (root,0,0,0.0) [aio/1] (root,0,0,0.0) [aio/2] (root,0,0,0.0) [aio/3] (root,0,0,0.0) [aio/4] (root,0,0,0.0) [aio/5] (root,0,0,0.0) [aio/6] (root,0,0,0.0) [aio/7] (root,0,0,0.0) [crypto/0] (root,0,0,0.0) [crypto/1] (root,0,0,0.0) [crypto/2] (root,0,0,0.0) [crypto/3] (root,0,0,0.0) [crypto/4] (root,0,0,0.0) [crypto/5] (root,0,0,0.0) [crypto/6] (root,0,0,0.0) [crypto/7] (root,0,0,0.0) [kthrotld/0] (root,0,0,0.0) [kthrotld/1] (root,0,0,0.0) [kthrotld/2] (root,0,0,0.0) [kthrotld/3] (root,0,0,0.0) [kthrotld/4] (root,0,0,0.0) [kthrotld/5] (root,0,0,0.0) [kthrotld/6] (root,0,0,0.0) [kthrotld/7] (root,0,0,0.0) [kpsmoused] (root,0,0,0.0) [usbhid_resumer] (root,0,0,0.0) [deferwq] (root,0,0,0.0) [kdmremove] (root,0,0,0.0) [kstriped] (root,0,0,0.0) [scsi_eh_0] (root,0,0,0.0) [scsi_eh_1] (root,0,0,0.0) [scsi_eh_2] (root,0,0,0.0) [scsi_eh_3] (root,0,0,0.0) [scsi_eh_4] (root,0,0,0.0) [scsi_eh_5] (root,0,0,0.0) [scsi_eh_6] (root,0,0,0.0) [scsi_eh_7] (root,0,0,0.0) [scsi_eh_8] (root,0,0,0.0) [scsi_eh_9] (root,0,0,0.0) [scsi_eh_10] (root,0,0,0.0) [fw_event0] (root,0,0,0.0) [poll_0_status] (root,0,0,0.0) [md1_raid1] (root,0,0,0.0) [md2_raid1] (root,0,0,0.0) [jbd2/md1-8] (root,0,0,0.0) [ext4-dio-unwrit] (root,11620,660,0.0) /sbin/udevd -d (root,0,0,0.0) [edac-poller] (root,0,0,0.0) [md0_raid1] (root,0,0,0.0) [kipmi0] (root,0,0,0.2) [md3_raid10] (root,0,0,0.0) [jbd2/md0-8] (root,0,0,0.0) [ext4-dio-unwrit] (root,0,0,0.0) [kauditd] (root,0,0,0.0) [flush-9:1] (root,93172,736,0.0) auditd (root,1836,128,0.0) /var/cfengine/scripts/gl/ipmi/ipmi_placeholder-v1 (root,255020,1688,0.0) /sbin/rsyslogd -i /var/run/syslogd.pid -c 4 (root,9136,572,0.0) irqbalance (rpc,18956,760,0.0) rpcbind -i (rpcuser,27552,1020,0.0) rpc.statd (root,4668,488,0.0) mdadm --monitor --scan -f --pid-file=/var/run/mdadm/mdadm.pid (root,1992460,2184,0.0) /opt/pbis/sbin/lwsmd --start-as-daemon (root,1835284,4604,0.0) lw-container lwreg (root,1178792,4156,0.0) lw-container eventlog (root,1570808,5624,0.0) lw-container netlogon (root,1904732,8740,0.0) lw-container lwio (root,3693612,41092,0.0) lw-container lsass (root,1716240,13508,0.0) lw-container gpagent (root,1257784,7124,0.0) lw-container reapsysl (root,0,0,0.0) [rpciod/0] (root,0,0,0.0) [rpciod/1] (root,0,0,0.0) [rpciod/2] (root,0,0,0.0) [rpciod/3] (root,0,0,0.0) [rpciod/4] (root,0,0,0.0) [rpciod/5] (root,0,0,0.0) [rpciod/6] (root,0,0,0.0) [rpciod/7] (root,27464,804,0.0) rpc.idmapd (dbus,31256,1892,0.0) dbus-daemon --system (root,189112,2884,0.0) cupsd -C /etc/cups/cupsd.conf (root,4064,624,0.0) /usr/sbin/acpid (68,37632,4972,0.0) hald (root,18092,1160,0.0) hald-runner (root,20236,1256,0.0) hald-addon-input: Listening on /dev/input/event5 /dev/input/event0 /dev/input/event1 (68,17792,1020,0.0) hald-addon-acpi: listening on acpid socket /var/run/acpid.socket (root,383512,1536,0.0) automount --pid-file /var/run/autofs.pid (root,181776,3752,0.0) /usr/local/eshmd/bin/eshmd -i 5 -s 8000 (root,107292,280,0.0) rpc.rquotad (root,0,0,0.0) [lockd] (root,0,0,0.0) [nfsd4] (root,0,0,0.0) [nfsd4_callbacks] (root,0,0,0.0) [nfsd] (root,0,0,0.0) [nfsd] (root,0,0,0.0) [nfsd] (root,0,0,0.0) [nfsd] (root,0,0,0.0) [nfsd] (root,0,0,0.0) [nfsd] (root,0,0,0.0) [nfsd] (root,0,0,0.0) [nfsd] (root,21476,964,0.0) rpc.mountd (root,6748,284,0.0) /usr/sbin/mcelog --daemon (root,64308,1092,0.0) /usr/sbin/sshd (root,22076,980,0.0) xinetd -stayalive -pidfile /var/run/xinetd.pid (root,30420,1544,0.0) ntpd -x -p /var/run/ntpd.pid (root,52364,784,0.0) /usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf (root,102868,4120,0.0) sendmail: accepting connections (smmsp,76360,2072,0.0) sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue (root,118772,1180,0.0) /usr/sbin/abrtd (root,9148,748,0.0) abrt-dump-oops -d /var/spool/abrt -rwx /var/log/messages (qpidd,795356,6448,0.0) /usr/sbin/qpidd --data-dir /var/lib/qpidd --daemon (root,421128,1432,0.0) /usr/sbin/collectd -C /etc/collectd.conf (root,117184,1244,0.0) crond (root,21424,468,0.0) /usr/sbin/atd (root,29436,1060,0.0) cfservd (root,29576,1700,0.0) cfenvd (root,100996,1984,0.0) cfexecd (root,132868,2784,0.0) /usr/bin/perl /var/cfengine/scripts/gl/bin/eshmd_mon-v1.pl (root,132868,2808,0.0) /usr/bin/perl /var/cfengine/scripts/gl/mon/gl.lw_mon.v5.pl (root,18564,556,0.0) /usr/sbin/smartd -q never (root,0,0,0.0) [kslowd000] (root,0,0,0.0) [kslowd001] (root,0,0,0.0) [nfsiod] (root,0,0,0.0) [kpanfs_dispatch] (root,0,0,0.0) [kpanfs_trace] (root,0,0,0.0) [kpanfs_thpool] (root,0,0,0.0) [kpanfs_thpool] (root,0,0,0.0) [kpanfs_thpool] (root,0,0,0.0) [kpanfs_thpool] (root,0,0,0.0) [kpanfs_thpool] (root,0,0,0.0) [kpanfs_thpool] (root,0,0,0.0) [kpanfs_thpool] (root,0,0,0.0) [kpanfs_thpool] (root,0,0,0.0) [kpanfs_timeout] (root,0,0,0.0) [kpanfs_cbreclai] (root,0,0,0.0) [kpanfs_cbb_all] (root,0,0,0.0) [kpanfs_cbbreak] (root,0,0,0.0) [kpanfs_cbbreak] (root,0,0,0.0) [kpanfs_cbbreak] (root,0,0,0.0) [kpanfs_cbbreak] (root,0,0,0.0) [kpanfs_cbbreak] (root,0,0,0.0) [kpanfs_cbbreak] (root,0,0,0.0) [kpanfs_cbbreak] (root,0,0,0.0) [kpanfs_cbbreak] (root,0,0,0.0) [kpanfs_statahea] (sgeadm,68408,6656,0.0) /opt/sge/bin/lx-amd64/sge_execd (cube,19272,1172,0.0) /cube/var/NID/Linux_x86_64/bin/nid (root,4048,544,0.0) /sbin/mingetty /dev/tty1 (cube,19220,780,0.0) /cube/var/NID/Linux_x86_64/bin/nidexe (root,4048,540,0.0) /sbin/mingetty /dev/tty2 (root,4048,544,0.0) /sbin/mingetty /dev/tty3 (root,4048,540,0.0) /sbin/mingetty /dev/tty4 (root,4048,540,0.0) /sbin/mingetty /dev/tty5 (root,4048,544,0.0) /sbin/mingetty /dev/tty6 (cube,17012,968,0.0) /cube/var/NID/Linux_x86_64/bin/nidcron (cube,128212,2624,0.0) perl -S /cube/var/NID/bin/nidMonitor -b /cube/var/NID/Linux_x86_64/bin (root,87512,1448,0.0) /usr/sbin/lldpd -x (_lldpd,88096,3436,0.0) /usr/sbin/lldpd -x (root,191876,5912,0.0) //usr/sbin/snmpd (nobody,14456,1148,0.0) /usr/bin/nmon -f -t -m /var/log/nmon (root,113012,7876,0.0) /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/runmmfs (root,0,0,0.0) [gpfs_aio/0] (root,0,0,0.0) [gpfs_aio/1] (root,0,0,0.0) [gpfs_aio/2] (root,0,0,0.0) [gpfs_aio/3] (root,0,0,0.0) [gpfs_aio/4] (root,0,0,0.0) [gpfs_aio/5] (root,0,0,0.0) [gpfs_aio/6] (root,0,0,0.0) [gpfs_aio/7] (root,0,0,0.0) [gpfsSwapdKproc] (root,4763892,408716,0.0) /usr/lpp/mmfs/bin/mmfsd (root,0,0,0.0) [mmkproc] (root,0,0,0.0) [mmkproc] (root,0,0,0.0) [mmkproc] (root,0,0,0.0) [nfsWatchKproc] (root,1965984,8840,0.0) /usr/lib64/icinga2/sbin/icinga2 --no-stack-rlimit daemon -c /etc/icinga2/icinga2.conf -d -e /var/log/icinga2/error.log (root,11616,620,0.0) /sbin/udevd -d (root,11616,564,0.0) /sbin/udevd -d (root,108792,4984,0.0) sshd: root@pts/0 (root,108456,1908,0.0) -bash (root,29684,2440,0.0) /var/cfengine/bin/cfagent -Dfrom_cfexecd:scheduled_run (root,72408,12888,0.1) ./gpu_burn 300 (root,180566688,157284,74.7) ./gpu_burn 300 (root,180566688,157252,74.5) ./gpu_burn 300 (root,180566688,157264,74.5) ./gpu_burn 300 (root,180566688,157252,75.0) ./gpu_burn 300 (root,13848,1220,0.0) nvidia-smi -l 5 -q -d temperature (root,110252,1416,0.0) /bin/bash /usr/bin/observium_agent (root,13360,984,0.0) ps ax -o user,vsz,rss,pcpu,command --columns 10000 (root,8376,788,0.0) sed -e 1d -e s/ *\([^ ]*\) *\([^ ]*\) *\([^ ]*\) *\([^ ]*\) */(\1,\2,\3,\4) /" [mem] => string(1197) "MemTotal: 65865212 kB MemFree: 51775056 kB Buffers: 5334476 kB Cached: 5184664 kB SwapCached: 2268 kB Active: 9008448 kB Inactive: 1982424 kB Active(anon): 308492 kB Inactive(anon): 163956 kB Active(file): 8699956 kB Inactive(file): 1818468 kB Unevictable: 262176 kB Mlocked: 544 kB SwapTotal: 66027516 kB SwapFree: 66008268 kB Dirty: 120 kB Writeback: 0 kB AnonPages: 732076 kB Mapped: 393896 kB Shmem: 716 kB Slab: 1965544 kB SReclaimable: 1838404 kB SUnreclaim: 127140 kB KernelStack: 9256 kB PageTables: 13824 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 98960120 kB Committed_AS: 7115356 kB VmallocTotal: 34359738367 kB VmallocUsed: 340592 kB VmallocChunk: 34359392248 kB HardwareCorrupted: 0 kB AnonHugePages: 331776 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 3672064 kB DirectMap2M: 61274112 kB DirectMap1G: 2097152 kB" [cpu] => string(29) "0.43 0.20 0.07 5/1152 29452 8" [uptime] => string(22) "7174024.10 56177544.51" [netif] => string(59) "eth0 1000Mb/s Full on yes eth1 Unknown! Unknown!(255) on no" [lnx_if:sep(58)] => string(578) "lo: 342472063 3009674 0 0 0 0 0 0 342472063 3009674 0 0 0 0 0 0 eth0: 5147806920800 3855905679 0 0 0 0 0 247377 1757147938294 1527667409 0 0 0 0 0 0 eth1: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 [lo] Link detected: yes [eth0] Speed: 1000Mb/s Duplex: Full Auto-negotiation: on Link detected: yes [eth1] Speed: Unknown! Duplex: Unknown! (255) Auto-negotiation: on Link detected: no" [tcp_conn_stats] => string(28) "CLOSE_WAIT 6 ESTABLISHED 106" [md] => string(489) "Personalities : [raid1] [raid10] md3 : active raid10 sda4[0] sdb4[1] 1085337600 blocks super 1.1 2 near-copies [2/2] [UU] bitmap: 0/9 pages [0KB], 65536KB chunk md0 : active raid1 sda3[0] sdb3[1] 102336 blocks super 1.0 [2/2] [UU] md2 : active raid1 sdb1[1] sda1[0] 66027520 blocks super 1.1 [2/2] [UU] md1 : active raid1 sdb2[1] sda2[0] 20463616 blocks super 1.1 [2/2] [UU] bitmap: 1/1 pages [4KB], 65536KB chunk unused devices: " [diskstat] => string(250) "1462456827 8 0 sda 126511548 98481296 32502526842 1010197874 27008597 4073633 11292053413 490477661 0 212859864 1500659594 8 16 sdb 127961490 97142684 32894719870 503022460 26950572 4131652 11292053413 452300842 0 211292006 955308735" [kernel] => string(4643) "1462456827 nr_free_pages 12943659 nr_inactive_anon 40989 nr_active_anon 77204 nr_inactive_file 454617 nr_active_file 2174989 nr_unevictable 65544 nr_mlock 136 nr_anon_pages 99954 nr_mapped 98474 nr_file_pages 2630352 nr_dirty 30 nr_writeback 0 nr_slab_reclaimable 459601 nr_slab_unreclaimable 31785 nr_page_table_pages 3456 nr_kernel_stack 1157 nr_unstable 0 nr_bounce 0 nr_vmscan_write 5210 nr_writeback_temp 0 nr_isolated_anon 0 nr_isolated_file 0 nr_shmem 179 numa_hit 12830774781 numa_miss 0 numa_foreign 0 numa_interleave 42294 numa_local 12830774781 numa_other 0 nr_anon_transparent_hugepages 162 pgpgin 4572279484 pgpgout 5648332436 pswpin 490 pswpout 5210 pgalloc_dma 0 pgalloc_dma32 30344802 pgalloc_normal 13600683936 pgalloc_movable 0 pgfree 13688548283 pgactivate 93122717 pgdeactivate 11138079 pgfault 3832008716 pgmajfault 43129 pgrefill_dma 0 pgrefill_dma32 464111 pgrefill_normal 3533254 pgrefill_movable 0 pgsteal_dma 0 pgsteal_dma32 21455977 pgsteal_normal 1984007089 pgsteal_movable 0 pgscan_kswapd_dma 0 pgscan_kswapd_dma32 21574272 pgscan_kswapd_normal 1970755168 pgscan_kswapd_movable 0 pgscan_direct_dma 0 pgscan_direct_dma32 9952 pgscan_direct_normal 13287360 pgscan_direct_movable 0 zone_reclaim_failed 0 pginodesteal 921 slabs_scanned 2033664 kswapd_steal 1992166864 kswapd_inodesteal 1293448 kswapd_low_wmark_hit_quickly 162839 kswapd_high_wmark_hit_quickly 191131 kswapd_skip_congestion_wait 0 pageoutrun 1139358 allocstall 5426 pgrotated 132962 compact_blocks_moved 811296 compact_pages_moved 22235472 compact_pagemigrate_failed 0 compact_stall 13885 compact_fail 7157 compact_success 6728 htlb_buddy_alloc_success 0 htlb_buddy_alloc_fail 0 unevictable_pgs_culled 815 unevictable_pgs_scanned 0 unevictable_pgs_rescued 206 unevictable_pgs_mlocked 1794 unevictable_pgs_munlocked 1658 unevictable_pgs_cleared 0 unevictable_pgs_stranded 0 unevictable_pgs_mlockfreed 0 thp_fault_alloc 1207601 thp_fault_fallback 54980 thp_collapse_alloc 23144 thp_collapse_alloc_failed 1818 thp_split 12098 cpu 90656192 34077 17905656 5610394913 15933358 2648 284314 0 0 cpu0 13412578 5580 2392110 699200196 1844725 31 12745 0 0 cpu1 11533405 2822 1818383 702237306 1455197 361 36622 0 0 cpu2 11884847 2263 1786234 701995423 1249475 373 34713 0 0 cpu3 11132922 2435 1573309 702969550 1235478 388 34729 0 0 cpu4 9033510 4784 1847186 703451990 2592295 32 20120 0 0 cpu5 9743631 4805 2092360 702388655 2483793 430 50100 0 0 cpu6 10697174 5224 3884657 699875312 2313299 489 45965 0 0 cpu7 13218122 6160 2511414 698276478 2759092 542 49317 0 0 intr 3854894341 148 0 0 0 1 0 0 0 1 31943323 0 0 0 0 0 0 0 0 0 173 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 23916 0 0 0 0 0 0 19313680 17468806 16566985 20493506 55397549 58230115 54563485 74509831 1 160371581 212474380 158828081 143800289 117605290 79992559 199759784 108199418 0 0 0 0 0 0 0 0 0 42 58 48 37 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ctxt 2229601897 btime 1455282803 processes 8567798 procs_running 5 procs_blocked 0 softirq 5110944526 0 1508710334 1524740 1183624230 311219963 0 47796635 357660445 547370 1699860809" [netctr] => string(399) "1462456827 lo 342472063 3009674 0 0 0 0 0 0 342472063 3009674 0 0 0 0 0 0 eth0 5147806922252 3855905701 0 0 0 0 0 247377 1757147945825 1527667431 0 0 0 0 0 0 eth1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0" [ipmi_sensors] => string(0) "" [ntp] => string(0) "" [cups_queues] => string(3) "---" [postfix_mailq] => string(46) "/var/spool/mqueue is empty Total requests: 0" [local] => string(64) "Could not open /proc/drbd for reading: No such file or directory" [lmsensors] => string(121) "power_meter-acpi-0 Adapter: ACPI interface power1: 571.00 W (interval = 1.00 s) The MySQL extension is not loaded" [nvidia] => array( [smi] => string(328) "index, name, temperature.gpu, fan.speed [%], power.draw [W], utilization.gpu [%], utilization.memory [%] 0, Tesla K80, 40, [Not Supported], 134.90 W, 99 %, 17 % 1, Tesla K80, 39, [Not Supported], 145.87 W, 99 %, 17 % 2, Tesla K80, 41, [Not Supported], 139.74 W, 99 %, 17 % 3, Tesla K80, 40, [Not Supported], 146.86 W, 99 %, 18 %" ) ) No include: No include: Observium No include: df No include: mounts No include: ps No include: mem No include: cpu No include: uptime No include: netif No include: lnx_if:sep(58) No include: tcp_conn_stats No include: md Including: unix-agent/diskstat.inc.php diskstats RRD CMD[update /data/observium/rrd/lgpu003.lon.compute.pgs.com/diskstat-sda.rrd N:126511548:98481296:32502526842:1010197874:27008597:4073633:11292053413:490477661:0:212859864:1500659594] RRD RUNTIME[0.0011s] RRD STDOUT[OK u:0.00 s:0.00 r:0.22] RRD_STATUS[TRUE]  RRD CMD[update /data/observium/rrd/lgpu003.lon.compute.pgs.com/diskstat-sdb.rrd N:127961490:97142684:32894719870:503022460:26950572:4131652:11292053413:452300842:0:211292006:955308735] RRD RUNTIME[0.0011s] RRD STDOUT[OK u:0.00 s:0.00 r:0.23] RRD_STATUS[TRUE]  No include: kernel No include: netctr No include: ipmi_sensors No include: ntp No include: cups_queues No include: postfix_mailq No include: local Including: unix-agent/lmsensors.inc.php Including: unix-agent/nvidia.inc.php nvidia-smi: Discover sensor: temperature, lgpu003.lon.compute.pgs.com, , 0, nvidia-smi, Nvidia Card 1: Tesla K80, SCALE: 1, LIMITS: (, , , 100), CURRENT: 40, agent, , SQL[SELECT COUNT(`sensor_id`) FROM `sensors`  WHERE `poller_type`= 'agent' AND `sensor_class` = 'temperature' AND `device_id` = '2810' AND `sensor_type` = 'nvidia-smi' AND `sensor_index` = '0'] SQL RUNTIME[0.00006509s] SQL[SELECT * FROM `sensors` WHERE `sensor_class` = 'temperature' AND `device_id` = '2810' AND `sensor_type` = 'nvidia-smi' AND `sensor_index` = '0'] SQL RUNTIME[0.00007296s] .Discover sensor: power, lgpu003.lon.compute.pgs.com, , 0, nvidia-smi, Nvidia Card 1: Tesla K80, SCALE: 1, LIMITS: (, , , ), CURRENT: 134.9, agent, , SQL[SELECT COUNT(`sensor_id`) FROM `sensors`  WHERE `poller_type`= 'agent' AND `sensor_class` = 'power' AND `device_id` = '2810' AND `sensor_type` = 'nvidia-smi' AND `sensor_index` = '0'] SQL RUNTIME[0.00004005s] SQL[SELECT * FROM `sensors` WHERE `sensor_class` = 'power' AND `device_id` = '2810' AND `sensor_type` = 'nvidia-smi' AND `sensor_index` = '0'] SQL RUNTIME[0.00005889s] LCurrent sensor value: "134.9", scale: "1" limit_high: "88.635" -> "202.35" limit_high_warn: "" -> "" limit_low: "" -> "" limit_low_warn: "" -> "" SQL[INSERT INTO `eventlog` (`device_id`,`entity_id`,`entity_type`,`timestamp`,`severity`,`message`) VALUES ('2810','11545','sensor',NOW(),'6','Sensor updated (limits): limit_high -> \"202.35\"')] SQL RUNTIME[0.00132990s] SQL[UPDATE `sensors` set `sensor_limit` ='202.35' WHERE `sensor_id` = '11545'] SQL RUNTIME[3.90010285s] .Discover sensor: temperature, lgpu003.lon.compute.pgs.com, , 1, nvidia-smi, Nvidia Card 2: Tesla K80, SCALE: 1, LIMITS: (, , , 100), CURRENT: 39, agent, , SQL[SELECT COUNT(`sensor_id`) FROM `sensors`  WHERE `poller_type`= 'agent' AND `sensor_class` = 'temperature' AND `device_id` = '2810' AND `sensor_type` = 'nvidia-smi' AND `sensor_index` = '1'] SQL RUNTIME[0.00647306s] SQL[SELECT * FROM `sensors` WHERE `sensor_class` = 'temperature' AND `device_id` = '2810' AND `sensor_type` = 'nvidia-smi' AND `sensor_index` = '1'] SQL RUNTIME[0.00645590s] .Discover sensor: power, lgpu003.lon.compute.pgs.com, , 1, nvidia-smi, Nvidia Card 2: Tesla K80, SCALE: 1, LIMITS: (, , , ), CURRENT: 145.87, agent, , SQL[SELECT COUNT(`sensor_id`) FROM `sensors`  WHERE `poller_type`= 'agent' AND `sensor_class` = 'power' AND `device_id` = '2810' AND `sensor_type` = 'nvidia-smi' AND `sensor_index` = '1'] SQL RUNTIME[0.00475097s] SQL[SELECT * FROM `sensors` WHERE `sensor_class` = 'power' AND `device_id` = '2810' AND `sensor_type` = 'nvidia-smi' AND `sensor_index` = '1'] SQL RUNTIME[0.00253797s] LCurrent sensor value: "145.87", scale: "1" limit_high: "110.505" -> "218.805" limit_high_warn: "" -> "" limit_low: "" -> "" limit_low_warn: "" -> "" SQL[INSERT INTO `eventlog` (`device_id`,`entity_id`,`entity_type`,`timestamp`,`severity`,`message`) VALUES ('2810','11549','sensor',NOW(),'6','Sensor updated (limits): limit_high -> \"218.805\"')] SQL RUNTIME[0.00050592s] SQL[UPDATE `sensors` set `sensor_limit` ='218.805' WHERE `sensor_id` = '11549'] SQL RUNTIME[0.01669812s] .Discover sensor: temperature, lgpu003.lon.compute.pgs.com, , 2, nvidia-smi, Nvidia Card 3: Tesla K80, SCALE: 1, LIMITS: (, , , 100), CURRENT: 41, agent, , SQL[SELECT COUNT(`sensor_id`) FROM `sensors`  WHERE `poller_type`= 'agent' AND `sensor_class` = 'temperature' AND `device_id` = '2810' AND `sensor_type` = 'nvidia-smi' AND `sensor_index` = '2'] SQL RUNTIME[0.00214219s] SQL[SELECT * FROM `sensors` WHERE `sensor_class` = 'temperature' AND `device_id` = '2810' AND `sensor_type` = 'nvidia-smi' AND `sensor_index` = '2'] SQL RUNTIME[0.00228095s] .Discover sensor: power, lgpu003.lon.compute.pgs.com, , 2, nvidia-smi, Nvidia Card 3: Tesla K80, SCALE: 1, LIMITS: (, , , ), CURRENT: 139.74, agent, , SQL[SELECT COUNT(`sensor_id`) FROM `sensors`  WHERE `poller_type`= 'agent' AND `sensor_class` = 'power' AND `device_id` = '2810' AND `sensor_type` = 'nvidia-smi' AND `sensor_index` = '2'] SQL RUNTIME[0.00094914s] SQL[SELECT * FROM `sensors` WHERE `sensor_class` = 'power' AND `device_id` = '2810' AND `sensor_type` = 'nvidia-smi' AND `sensor_index` = '2'] SQL RUNTIME[0.00304294s] LCurrent sensor value: "139.74", scale: "1" limit_high: "88.71" -> "209.61" limit_high_warn: "" -> "" limit_low: "" -> "" limit_low_warn: "" -> "" SQL[INSERT INTO `eventlog` (`device_id`,`entity_id`,`entity_type`,`timestamp`,`severity`,`message`) VALUES ('2810','11553','sensor',NOW(),'6','Sensor updated (limits): limit_high -> \"209.61\"')] SQL RUNTIME[0.00136685s] SQL[UPDATE `sensors` set `sensor_limit` ='209.61' WHERE `sensor_id` = '11553'] SQL RUNTIME[0.01222897s] .Discover sensor: temperature, lgpu003.lon.compute.pgs.com, , 3, nvidia-smi, Nvidia Card 4: Tesla K80, SCALE: 1, LIMITS: (, , , 100), CURRENT: 40, agent, , SQL[SELECT COUNT(`sensor_id`) FROM `sensors`  WHERE `poller_type`= 'agent' AND `sensor_class` = 'temperature' AND `device_id` = '2810' AND `sensor_type` = 'nvidia-smi' AND `sensor_index` = '3'] SQL RUNTIME[0.00332594s] SQL[SELECT * FROM `sensors` WHERE `sensor_class` = 'temperature' AND `device_id` = '2810' AND `sensor_type` = 'nvidia-smi' AND `sensor_index` = '3'] SQL RUNTIME[0.00253105s] .Discover sensor: power, lgpu003.lon.compute.pgs.com, , 3, nvidia-smi, Nvidia Card 4: Tesla K80, SCALE: 1, LIMITS: (, , , ), CURRENT: 146.86, agent, , SQL[SELECT COUNT(`sensor_id`) FROM `sensors`  WHERE `poller_type`= 'agent' AND `sensor_class` = 'power' AND `device_id` = '2810' AND `sensor_type` = 'nvidia-smi' AND `sensor_index` = '3'] SQL RUNTIME[0.00365710s] SQL[SELECT * FROM `sensors` WHERE `sensor_class` = 'power' AND `device_id` = '2810' AND `sensor_type` = 'nvidia-smi' AND `sensor_index` = '3'] SQL RUNTIME[0.00153804s] LCurrent sensor value: "146.86", scale: "1" limit_high: "76.455" -> "220.29" limit_high_warn: "" -> "" limit_low: "" -> "" limit_low_warn: "" -> "" SQL[INSERT INTO `eventlog` (`device_id`,`entity_id`,`entity_type`,`timestamp`,`severity`,`message`) VALUES ('2810','11555','sensor',NOW(),'6','Sensor updated (limits): limit_high -> \"220.29\"')] SQL RUNTIME[0.00073099s] SQL[UPDATE `sensors` set `sensor_limit` ='220.29' WHERE `sensor_id` = '11555'] SQL RUNTIME[0.01149607s] . Processes: Sensors: SQL[SELECT * FROM `sensors` WHERE `device_id` = '2810' AND `sensor_class` = 'temperature' AND `poller_type` = 'agent'] SQL RUNTIME[0.00117397s] SQL[SELECT * FROM `sensors` WHERE `device_id` = '2810' AND `sensor_class` = 'humidity' AND `poller_type` = 'agent'] SQL RUNTIME[0.00066495s] SQL[SELECT * FROM `sensors` WHERE `device_id` = '2810' AND `sensor_class` = 'fanspeed' AND `poller_type` = 'agent'] SQL RUNTIME[0.00056005s] SQL[SELECT * FROM `sensors` WHERE `device_id` = '2810' AND `sensor_class` = 'airflow' AND `poller_type` = 'agent'] SQL RUNTIME[0.00160098s] SQL[SELECT * FROM `sensors` WHERE `device_id` = '2810' AND `sensor_class` = 'voltage' AND `poller_type` = 'agent'] SQL RUNTIME[0.00061083s] SQL[SELECT * FROM `sensors` WHERE `device_id` = '2810' AND `sensor_class` = 'current' AND `poller_type` = 'agent'] SQL RUNTIME[0.00151587s] SQL[SELECT * FROM `sensors` WHERE `device_id` = '2810' AND `sensor_class` = 'power' AND `poller_type` = 'agent'] SQL RUNTIME[0.00054407s] SQL[SELECT * FROM `sensors` WHERE `device_id` = '2810' AND `sensor_class` = 'apower' AND `poller_type` = 'agent'] SQL RUNTIME[0.00049996s] SQL[SELECT * FROM `sensors` WHERE `device_id` = '2810' AND `sensor_class` = 'rpower' AND `poller_type` = 'agent'] SQL RUNTIME[0.00070190s] SQL[SELECT * FROM `sensors` WHERE `device_id` = '2810' AND `sensor_class` = 'impedance' AND `poller_type` = 'agent'] SQL RUNTIME[0.00050402s] SQL[SELECT * FROM `sensors` WHERE `device_id` = '2810' AND `sensor_class` = 'resistance' AND `poller_type` = 'agent'] SQL RUNTIME[0.00021696s] SQL[SELECT * FROM `sensors` WHERE `device_id` = '2810' AND `sensor_class` = 'frequency' AND `poller_type` = 'agent'] SQL RUNTIME[0.00049186s] SQL[SELECT * FROM `sensors` WHERE `device_id` = '2810' AND `sensor_class` = 'dbm' AND `poller_type` = 'agent'] SQL RUNTIME[0.00106478s] SQL[SELECT * FROM `sensors` WHERE `device_id` = '2810' AND `sensor_class` = 'snr' AND `poller_type` = 'agent'] SQL RUNTIME[0.00042820s] SQL[SELECT * FROM `sensors` WHERE `device_id` = '2810' AND `sensor_class` = 'capacity' AND `poller_type` = 'agent'] SQL RUNTIME[0.00106883s] SQL[SELECT * FROM `sensors` WHERE `device_id` = '2810' AND `sensor_class` = 'load' AND `poller_type` = 'agent'] SQL RUNTIME[0.00024819s] SQL[SELECT * FROM `sensors` WHERE `device_id` = '2810' AND `sensor_class` = 'runtime' AND `poller_type` = 'agent'] SQL RUNTIME[0.00081110s] SQL[SELECT * FROM `sensors` WHERE `device_id` = '2810' AND `sensor_class` = 'state' AND `poller_type` = 'agent'] SQL RUNTIME[0.00040889s] SQL[SELECT * FROM `sensors` WHERE `device_id` = '2810' AND `sensor_class` = 'waterflow' AND `poller_type` = 'agent'] SQL RUNTIME[0.00221491s] SQL[SELECT * FROM `sensors` WHERE `device_id` = '2810' AND `sensor_class` = 'counter' AND `poller_type` = 'agent'] SQL RUNTIME[0.00037885s] Virtual machines: SQL[SELECT * FROM `vminfo` WHERE `device_id` = '2810' AND `vm_source` = 'agent'] SQL RUNTIME[0.00056815s]  o Module time  4.1478s including: includes/polling/system.inc.php ##### Module Start: system #####  CMD[/usr/bin/snmpget -v2c -c *** -Pu -OQUs -m SNMPv2-MIB -M /data/observium/mibs/rfc:/data/observium/mibs/net-snmp 'udp':'lgpu003.lon.compute.pgs.com':'161' sysUpTime.0 sysLocation.0 sysContact.0 sysName.0]  CMD EXITCODE[0] CMD RUNTIME[0.0112s] CMD EXITDELAY[10ms] STDOUT[ sysUpTime.0 = 17:4:48:24.03 sysLocation.0 = Megacenter, Weybridge, United-Kingdom sysContact.0 = GCR sysName.0 = lgpu003 ] SNMP STATUS[TRUE] CMD[/usr/bin/snmpget -v2c -c *** -Pu -Oqv -m SNMPv2-MIB -M /data/observium/mibs/rfc:/data/observium/mibs/net-snmp 'udp':'lgpu003.lon.compute.pgs.com':'161' sysDescr.0]  CMD EXITCODE[0] CMD RUNTIME[0.0092s] STDOUT[ Linux lgpu003 2.6.32-504.16.2.el6.x86_64 #1 SMP Wed Apr 22 06:48:29 UTC 2015 x86_64 ] SNMP STATUS[TRUE] CMD[/usr/bin/snmpget -v2c -c *** -Pu -Oqvn -m SNMPv2-MIB -M /data/observium/mibs/rfc:/data/observium/mibs/net-snmp 'udp':'lgpu003.lon.compute.pgs.com':'161' sysObjectID.0]  CMD EXITCODE[0] CMD RUNTIME[0.0099s] CMD EXITDELAY[10ms] STDOUT[ .1.3.6.1.4.1.8072.3.2.10 ] SNMP STATUS[TRUE] CMD[/usr/bin/snmpget -v2c -c *** -Pu -Ovqn -m SNMP-FRAMEWORK-MIB -M /data/observium/mibs/rfc:/data/observium/mibs/net-snmp 'udp':'lgpu003.lon.compute.pgs.com':'161' snmpEngineID.0]  CMD EXITCODE[0] CMD RUNTIME[0.0101s] CMD EXITDELAY[10ms] STDOUT[ "80 00 1F 88 03 0C C4 7A A8 CD 12 " ] SNMP STATUS[TRUE] Using UNIX Agent Uptime (7174024 sec. => 83 days, 47m 4s) RRD CMD[update /data/observium/rrd/lgpu003.lon.compute.pgs.com/uptime.rrd N:7174024] RRD RUNTIME[0.0011s] RRD STDOUT[OK u:0.00 s:0.00 r:4.31] RRD_STATUS[TRUE]   o Uptime  83 days, 47m 4s > $uptimes - /data/observium/poller.php:163 /data/observium/includes/polling/system.inc.php:191 ================================================ array( [sysUpTime] => int(1486104) [unix-agent] => double(7174024) [use] => string(10) "unix-agent" [message] => string(23) "Using UNIX Agent Uptime" [uptime] => double(7174024) [formatted] => string(15) "83 days, 47m 4s" [previous] => string(7) "7173938" [diff] => double(-86) [rebooted] => int(0) )  o sysObjectID  .1.3.6.1.4.1.8072.3.2.10  o snmpEngineID  80001F88030CC47AA8CD12  o sysDescr  Linux lgpu003 2.6.32-504.16.2.el6.x86_64 #1 SMP Wed Apr 22 06:48:29 UTC 2015 x86_64  o sysName  lgpu003  o Location  Megacenter, Weybridge, United-Kingdom SQL[SELECT * FROM `devices_locations` WHERE `device_id` = '2810'] SQL RUNTIME[0.00060511s]  o Module time  0.0803s including: includes/polling/os.inc.php ##### Module Start: os #####  SQL[SELECT * FROM `observium_attribs`] SQL RUNTIME[0.00020194s] SQL[SELECT * FROM `entPhysical` WHERE `device_id` = '2810' AND `entPhysicalContainedIn` = '0'] SQL RUNTIME[0.00019503s]  o OS Poller  Group CMD[/usr/bin/snmpget -v2c -c *** -Pu -Oqv -m MIB-Dell-10892 -M /data/observium/mibs/rfc:/data/observium/mibs/net-snmp:/data/observium/mibs/dell 'udp':'lgpu003.lon.compute.pgs.com':'161' chassisModelName.1]  CMD EXITCODE[0] CMD RUNTIME[0.0236s] CMD EXITDELAY[10ms] STDOUT[ No Such Object available on this agent at this OID ] SNMP STATUS[FALSE] SNMP ERROR[#1000 - Failed response] SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '2810' AND `error_code` = '1000' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-Oqv' AND `mib` = 'MIB-Dell-10892' AND `oid` = 'chassisModelName.1';] SQL RUNTIME[0.03924203s] SQL[UPDATE `snmp_errors` set `error_count` ='3',`updated` ='1462456825' WHERE `error_id` = '1484'] SQL RUNTIME[0.00094509s] CMD[/usr/bin/snmpget -v2c -c *** -Pu -Oqv -m CPQSINFO-MIB -M /data/observium/mibs/rfc:/data/observium/mibs/net-snmp:/data/observium/mibs/hp 'udp':'lgpu003.lon.compute.pgs.com':'161' cpqSiProductName.0]  CMD EXITCODE[0] CMD RUNTIME[0.0199s] STDOUT[ No Such Object available on this agent at this OID ] SNMP STATUS[FALSE] SNMP ERROR[#1000 - Failed response] SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '2810' AND `error_code` = '1000' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-Oqv' AND `mib` = 'CPQSINFO-MIB' AND `oid` = 'cpqSiProductName.0';] SQL RUNTIME[0.01143789s] SQL[UPDATE `snmp_errors` set `error_count` ='3',`updated` ='1462456825' WHERE `error_id` = '1486'] SQL RUNTIME[0.00175500s] CMD[/usr/bin/snmpget -v2c -c *** -Pu -Oqv -m UCD-SNMP-MIB -M /data/observium/mibs/rfc:/data/observium/mibs/net-snmp 'udp':'lgpu003.lon.compute.pgs.com':'161' .1.3.6.1.4.1.2021.7890.1.3.1.1.6.100.105.115.116.114.111]  CMD EXITCODE[0] CMD RUNTIME[0.0187s] STDOUT[ "CentOS 6.2" ] SNMP STATUS[TRUE]  o Hardware  Generic x86 [64bit]  o Version  2.6.32-504.16.2.el6.x86_64  o Features  CentOS 6.2  o Serial    o Asset    o Module time  0.1343s ##### lgpu003.lon.compute.pgs.com [2810] completed poller modules at 2016-05-05 15:00:25 #####   o Graphs [checked]  ping, ping_snmp, agent, uptime  o Poller time  4.3844 seconds Updating lgpu003.lon.compute.pgs.com - > $update_array ======== array( [uptime] => double(7174024) [last_polled] => array( [0] => string(5) "NOW()" ) [last_polled_timetaken] => double(4.3844) [device_state] => string(128) "a:1:{s:15:"poller_mod_perf";a:3:{s:10:"unix-agent";d:4.1478000000000002;s:6:"system";d:0.080299999999999996;s:2:"os";d:0.1343;}}" ┗ serialized ~ array( [poller_mod_perf] => array( [unix-agent] => double(4.1478) [system] => double(0.0803) [os] => double(0.1343) ) ) ) SQL[UPDATE `devices` set `uptime` ='7174024',`last_polled` =NOW(),`last_polled_timetaken` ='4.3844',`device_state` ='a:1:{s:15:\"poller_mod_perf\";a:3:{s:10:\"unix-agent\";d:4.1478000000000002;s:6:\"system\";d:0.080299999999999996;s:2:\"os\";d:0.1343;}}' WHERE `device_id` = '2810'] SQL RUNTIME[0.00133896s]  o Updated Data  uptime, last_polled, last_polled_timetaken, device_state > $data - /data/observium/poller.php:163 /data/observium/includes/alerts.inc.php:57 ========================================== array( [device_status] => string(1) "1" [device_status_type] => string(0) "" [device_ping] => string(4) "0.06" [device_snmp] => string(4) "8.87" [device_uptime] => string(7) "7173938" [device_rebooted] => int(0) [device_duration_poll] => string(4) "3.68" ) SQL[UPDATE `alert_table` set `state` ='{\"metrics\":{\"device_status\":\"1\"}}',`count` ='0',`alert_status` ='1',`last_message` ='Checks OK',`last_checked` ='1462456825',`last_ok` ='1462456825' WHERE `alert_table_id` = '43475'] SQL RUNTIME[0.00077200s] RRD CMD[update /data/observium/rrd/lgpu003.lon.compute.pgs.com/alert-43475.rrd N:1:1] RRD RUNTIME[0.0018s] RRD STDOUT[OK u:0.00 s:0.00 r:4.46] RRD_STATUS[TRUE]  SQL[DELETE FROM `observium_processes` WHERE `process_pid` = '6528' AND `process_name` = 'poller.php' AND `device_id` = '2810'] SQL RUNTIME[0.00124598s] SQL[INSERT INTO `perf_times` (`type`,`doing`,`start`,`duration`,`devices`) VALUES ('poll','lgpu003*','1462456820.6557','4.566','1')] SQL RUNTIME[0.00315809s] ./poller.php: lgpu003* - 1 devices polled in 4.566 secs ##### Completed polling run at 2016-05-05 15:00:25 #####  NOTE, $config['snmp']['hide_auth'] is set to TRUE, snmp community and snmp v3 auth hidden from debug output.  o Devices Polled  1  o Poller Time  4.566 secs  o Memory usage  8MB (peak: 8MB)  o MySQL Usage  Cell[11/0.023s] Row[19/0.077s] Rows[31/0.026s] Column[0/0s] Update[8/3.947s] Insert[6/0.009s] Delete[1/0.001s]  o RRDTool Usage  update[8/0.009s]