False system uptime alarm / DB device uptime is set to 0

Hi Team,

We have a weird issue related to the system's uptime since some days ago.

Mentioned device is Mellanox MLNX-OS switch with uptime more than 1000 days: Uptime: 1034d 15h 11m 40.032s

So please find attached debug information:

./poller.php -d -m system -h c2-ibsw1
DEBUG!
Observium 0.15.3.6350
Poller

Starting polling run:

SQL[SELECT `device_id` FROM `devices` WHERE `disabled` = 0 AND `hostname` LIKE 'c2-ibsw1' ORDER BY `device_id` ASC]
RUNTIME[0.00060105s]

SQL[SELECT * FROM `devices` WHERE `device_id` = '132']
RUNTIME[0.00011492s]

SQL[SELECT * FROM `devices_attribs` WHERE `device_id` = '132']
RUNTIME[0.00006580s]

SQL[SELECT * FROM `alert_tests` WHERE 1]
RUNTIME[0.00008512s]
Cached 9 alert rules.

SQL[SELECT *,`alert_table`.`alert_table_id` AS `alert_table_id` FROM `alert_table` LEFT JOIN `alert_table-state` ON `alert_table`.`alert_table_id` = `alert_table-state`.`alert_table_id` WHERE `device_id` = '132']
RUNTIME[0.00033712s]
c2-ibsw1 132 mlnx-os (mellanox)

CMD[/sbin/fping -t 500 -c 1 -q x.x.x.x 2>&1]
EXITCODE[0]
RUNTIME[0.0053s]
STDOUT[
x.x.x.x : xmt/rcv/%loss = 1/1/0%, min/avg/max = 0.70/0.70/0.70
]

SQL[SELECT `version` FROM `dbSchema`;]
RUNTIME[0.00015402s]

CMD[/usr/bin/snmpget -v2c -c *** -Pu -OQUst -m SNMPv2-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'c2-ibsw1':'161' sysObjectID.0 sysUpTime.0]
EXITCODE[0]
RUNTIME[0.0137s]
STDOUT[
sysObjectID.0 = enterprises.33049.2.1
sysUpTime.0 = 1546241901
]
SNMP_STATUS[TRUE]
RRD /opt/observium/rrd/c2-ibsw1/status.rrd already exists - no need to create.
RRD[cmd[update /opt/observium/rrd/c2-ibsw1/status.rrd N:1 --daemon unix:/var/run/rrdcached.sock]
stdout[OK u:0.00 s:0.00 r:0.02]
stderr[]]
RRD /opt/observium/rrd/c2-ibsw1/ping.rrd already exists - no need to create.
RRD[cmd[update /opt/observium/rrd/c2-ibsw1/ping.rrd N:0.70 --daemon unix:/var/run/rrdcached.sock]
stdout[OK u:0.00 s:0.00 r:0.02]
stderr[]]
RRD /opt/observium/rrd/c2-ibsw1/ping_snmp.rrd already exists - no need to create.
RRD[cmd[update /opt/observium/rrd/c2-ibsw1/ping_snmp.rrd N:17.19 --daemon unix:/var/run/rrdcached.sock]
stdout[OK u:0.00 s:0.00 r:0.03]
stderr[]]

SQL[SELECT * FROM `device_graphs` WHERE `device_id` = '132']
RUNTIME[0.00013113s]

including: includes/polling/system.inc.php

CMD[/usr/bin/snmpget -v2c -c *** -Pu -OQUs -m SNMPv2-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'c2-ibsw1':'161' sysUpTime.0 sysLocation.0 sysContact.0 sysName.0]
EXITCODE[0]
RUNTIME[0.0139s]
STDOUT[
sysUpTime.0 = 178:23:06:59.03
sysLocation.0 = XXX
sysContact.0 = XXX
sysName.0 = c2-ibsw1
]
SNMP_STATUS[TRUE]

CMD[/usr/bin/snmpget -v2c -c *** -Pu -Oqv -m SNMPv2-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'c2-ibsw1':'161' sysDescr.0]
EXITCODE[0]
RUNTIME[0.0137s]
STDOUT[
Linux c2-ibsw1 2.6.27-MELLANOXuni-m460ex EFM_PPC_M460EX EFM_1.1.2500 #1 2011-02-22 15:51:54 ppc
]
SNMP_STATUS[TRUE]

CMD[/usr/bin/snmpget -v2c -c *** -Pu -Oqvn -m SNMPv2-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'c2-ibsw1':'161' sysObjectID.0]
EXITCODE[0]
RUNTIME[0.0133s]
STDOUT[
.1.3.6.1.4.1.33049.2.1
]
SNMP_STATUS[TRUE]

CMD[/usr/bin/snmpget -v2c -c *** -Pu -Ovqn -m SNMP-FRAMEWORK-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'c2-ibsw1':'161' snmpEngineID.0]
EXITCODE[0]
RUNTIME[0.0126s]
STDOUT[
"80 00 81 19 04 30 30 30 32 43 39 36 32 34 42 36
30 "
]
SNMP_STATUS[TRUE]

CMD[/usr/bin/snmpget -v2c -c *** -Pu -Oqv -m HOST-RESOURCES-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'c2-ibsw1':'161' hrSystemUptime.0]
EXITCODE[0]
RUNTIME[0.017s]
STDOUT[
0:0:00:00.00
]
SNMP_STATUS[TRUE]
Using SNMP Agent hrSystemUptime (0 sec. => 0s)

SQL[INSERT INTO `eventlog` (`device_id`,`entity_id`,`entity_type`,`timestamp`,`severity`,`message`) VALUES ('132','132','device',NOW(),'4','Device rebooted: after 0s')]
RUNTIME[0.00038815s]
RRD /opt/observium/rrd/c2-ibsw1/uptime.rrd already exists - no need to create.
RRD[cmd[update /opt/observium/rrd/c2-ibsw1/uptime.rrd N:0 --daemon unix:/var/run/rrdcached.sock]
stdout[OK u:0.00 s:0.00 r:0.12]
stderr[]]
Uptime: 0s

SQL[SELECT * FROM `devices_locations` WHERE `device_id` = '132']
RUNTIME[0.00010180s]
Module [ system ] time: 0.0897s

including: includes/polling/os.inc.php
I'm Generic :'(

Hardware: Version: Features: Serial: Asset:
Module [ os ] time: 0.0004s
Graphs [checked]: ping, ping_snmp, uptime

Polled in 0.122 seconds
Updating c2-ibsw1 - Array
(
[uptime] => 0
[last_polled] => Array
(
[0] => NOW()
)

[last_polled_timetaken] => 0.122
[device_state] => a:1:{s:15:"poller_mod_perf";a:2:{s:6:"system";s:6:"0.0897";s:2:"os";s:6:"0.0004";}}
)

SQL[UPDATE `devices` set `uptime` ='0',`last_polled` =NOW(),`last_polled_timetaken` ='0.122',`device_state` ='a:1:{s:15:\"poller_mod_perf\";a:2:{s:6:\"system\";s:6:\"0.0897\";s:2:\"os\";s:6:\"0.0004\";}}' WHERE `device_id` = '132']
RUNTIME[0.00152802s]
UPDATED!

Checking alerts
Array
(
[device_status] => 1
[device_status_type] =>
[device_ping] => 0.70
[device_snmp] => 17.19
[device_uptime] => 0
[device_rebooted] => 1
[device_duration_poll] => 6.92
)

SQL[INSERT INTO `perf_times` (`type`,`doing`,`start`,`duration`,`devices`) VALUES ('poll','c2-ibsw1','1427718776.5666','0.131','1')]
RUNTIME[0.00013614s]
./poller.php: c2-ibsw1 - 1 devices polled in 0.131 secs
NOTE, $config['snmp']['hide_auth'] sets as TRUE, snmp community and snmp v3 auth hidden from debug output.
Memory usage: 4.25MB (peak: 4.25MB)
MySQL: Cell[1/0s] Row[2/0s] Rows[3/0s] Column[0/0s] Update[1/0s] Insert[2/0s] Delete[0/0s]

And a query directly from the MySQL DB:

mysql> select hostname,uptime from devices where hostname='c2-ibsw1';
--------------------+

--------------------+

--------------------+
1 row in set (0.00 sec)

Thank you,
Plamen Stoev