RRD update error with big Counter64 numbers

TL;DR; PHP integers are not suitable for SNMP Counter64 numbers and floats are not precise nor always working for RRD DERIVE and COUNTER DSTs.

This is how it looks in log:

[2016/03/10 06:06:26 -0600] poller.php(31697): RRD ERROR: /opt/observium/rrd/router.example.com/port-1.rrd: not a simple signed integer: '1.844674407371E+19', CMD: update /opt/observium/rrd/router.example.com/port-1.rrd N:21763770421:13548302322:0:0:81705353:83379237:1.844674407371E+19:0:0:0:52:9223372036854775807:0:2884:0

Don't ask me how can these numbers be so big. This comes from Cisco CSR 1000v.

Here is the analysis of the problem and root cause:

Data from snmpwalk_cache_oid() is string type (snmpwalk ouptut splited by "="). This means that any number is fine.

All variables that go directly to rrdtool_update() remain strings until rrdupdate command.

However, there are variables that pass through some calculations like these:

$this_port['ifInNUcastPkts']  = $this_port['ifInBroadcastPkts']  + $this_port['ifInMulticastPkts'];

$this_port['ifOutNUcastPkts'] = $this_port['ifOutBroadcastPkts'] + $this_port['ifOutMulticastPkts'];

or these:

$stat_oids = array_merge($stat_oids_ifEntry, $stat_oids_ifXEntry);

For any mathematical operations string variables will be converted to numbers.

First ones are 32 bit numbers and work fine but latter one adds together 64 bit unsigned numbers. PHP integer is 64 bit unsigned ie. 63 bit signed. If the number in string is bigger than PHP_INT_MAX (64 bit unsigned number = 9223372036854775807), then it will be evaluated as a float: [http://php.net/manual/en/language.types.string.php#language.types.string.conversion
]

In rrdtool_update() floats are converted back to strings by implode(). In this case they can be converted using exponential notation: http://php.net/manual/en/language.types.string.php#language.types.string.casting

rrdtool can handle floating point values but only since 1.5.0 (look at https://github.com/oetiker/rrdtool-1.x/commit/06a486e). New data type DDERIVE (and DCOUNTER) must be used for that.

But even then the precision will be lost and calculated change could be wrong.

Numbers for RRD are not limited with 64 bits signed integers like in PHP but with 30 digits (look at https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=543631#39 ).

The right way to handle Counter64 numbers is to use gmp_add() from GMP (or similar) function that can work with arbitrary-length numbers.

The same bug is already reported in http://jira.observium.org/browse/OBSERVIUM-1522