Log inSkip to main contentSkip to sidebar
  • Observium
  • Fisheye
Observium
  • More
    DashboardsProjectsIssues
  • Give feedback to Atlassian
  • Help
    • Jira Core help
    • Keyboard Shortcuts
    • About Jira
    • Jira Credits
  • Log In

Open issues

  • All issues
  • Open issues
  • Done issues
  • Viewed recently
  • Created recently
  • Resolved recently
  • Updated recently
View all issues and filters
Order by Priority
Order by Priority
  1. Bug
    OBS-4984192 PB of data in one day on multiple customers
  2. Improvement
    OBS-3024Extend API for Billing
  3. Paid Support
    OBS-4470Cosnsolidation path
  4. Improvement
    OBS-5054Ports with links HALF duplex
  5. Bug
    OBS-5052Observium alerting interface errors when there are none
  6. Help
    OBS-4983Scheduled Maintenance not impacting Widgets
  7. Help
    OBS-4981How to make configuration per device group
  8. Help
    OBS-4962Alert Checkers - not working
  9. Help
    OBS-4441Cisco WLAN
  10. New Feature
    OBS-3792Receives alot of alerts
  11. Help
    OBS-4535Change Mailaddress of account itsupportliwest-6767211
  12. New Feature
    OBS-4331Combine LDAP and local authentication
  13. Help
    OBS-4321Do I need to go through the rrd tarring when migrating to a new server?
  14. Bug
    OBS-4295Automated Installation
  15. Help
    OBS-4222alert issue
  16. New Feature
    OBS-4128routing monitoring and alert management
  17. Help
    OBS-4123Pages are loading slow
  18. Improvement
    OBS-4054Single device with multiple IPMI targets
  19. Bug
    OBS-3997F5 stuck on discovery
  20. Help
    OBS-3933Brocade ICX6610 crashing while pooled
  21. Bug
    OBS-3900Getting unknown in alert status
  22. Vendor Bug
    OBS-3287Adding Collectd graphs to dashboards
  23. Bug
    OBS-5063No ESSID / Client information
  24. Improvement
    OBS-5001Add "ignore until <date>" on the list "Status Warnings and Notifications"
  25. Add New Device / OS
    OBS-5027Adding Neptune/Ribbon Optical
  26. Bug
    OBS-5050cannot login to observium.org Subscription Management
  27. Add New Device / OS
    OBS-4985Add support NGFW mib
  28. New Feature
    OBS-5039Allow remote auth user management
  29. Help
    OBS-4335Auth_mecanism remote not working as expected
  30. Vendor Bug
    OBS-5026huawei s2326/s2328 ddm
  31. Help
    OBS-5024Observium 501(c)(3) Non-Profit Charity Professional Subscription Request
  32. Improvement
    OBS-5021Cisco NCS PSU info
  33. Add New Device / OS
    OBS-5002Missing support for Schneider chiller BCEF0401A
  34. Bug
    OBS-5011Citrix ADC (NetScaler) GSLB vservers & services
  35. Improvement
    OBS-4457Allow static mapping of sysLocation strings to coordinates.
  36. Add New Device / OS
    OBS-4997Mtlink-BDCOM
  37. Help
    OBS-4996Creazione allert
  38. Improvement
    OBS-4995Add more user action events to eventlog
  39. Help
    OBS-4992OID non trovato
  40. Add New Device / OS
    OBS-4757Fortinet VPN Tunnel Improvements
  41. Help
    OBS-4909OLT ZTE C610 some components not discover and not visible www
  42. Add New Device / OS
    OBS-4973agregar MIB ZTE Observium
  43. Add New Device / OS
    OBS-4777Add ZTE C320
  44. New Feature
    OBS-4822Add VPRN, VPLS and EPIPE Nokia like VRFs Huawei and OSPF Cisco
  45. Bug
    OBS-3599Observium cannot detect aruba-controller
  46. Improvement
    OBS-4346aruba mibs / devices - snmpwalk attached. would really like to see APs/clients/etc
  47. Help
    OBS-4975Missing Gauge Alert option
  48. New Feature
    OBS-4971Radius Authentication using EAP protocol
  49. New Feature
    OBS-4616please add the ability to export device lists and info
  50. Help
    OBS-4944Alert on event in event log like "Storage removed" or "Memory pool removed"
Refresh results
1 2 3 4 5Next >>
Uploaded image for project: 'Observium'
  1. Observium
  2. OBS-4984

192 PB of data in one day on multiple customers

Log In
Pending Response
Export
undefined
    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Blocker
    • None
    • None
    • Ports
      • Monitoring
      • billing
      • device
      • mysql
      • poller

    Description

      There is an issue in Observium where some customers have 192+ PetaBytes of data usage tracked on their bill. This data occurs in one day (different per customer) and returns back to normal the next day.

      What would be causing these spikes in data usage?

      Attachments

        Activity

          • All
          • Comments
          • History
          • Activity
          adama Adam Armstrong added a comment - 2025/03/06 02:03 AM

          Hi,

          The only thing that could really cause this would be incorrect data being returned by the device, with the delta between two polled datapoints being very large, 192PB is probably the maximum size of the database field signed, or something there abouts.

          This should not really happen though. 

          How distant are the devices being polled from the polling system? What hardware are they?

          You should be able to see the deltas in the database, but we don't actually record the actual number returned by the polled device.

          Do you get similar odd behaviour in the RRD-based port graphs? RRD probably clips the number off because of the maximum value on the DS, though.

          Our logic to generate the in/out delta is quite simple: 

                      if (is_numeric($period) && is_numeric($entity['bill_port_counter_in']) && is_numeric($entity['bill_port_counter_out']) && $data['in'] >= $entity['bill_port_counter_in'] && $data['out'] >= $entity['bill_port_counter_out']) {
                          // Counters are higher or equal to before, seems legit.
                          $in_delta  = int_sub($data['in'], $entity['bill_port_counter_in']);
                          $out_delta = int_sub($data['out'], $entity['bill_port_counter_out']);
                          echo("Counters valid, delta generated.\n");
                      } elseif (is_numeric($period) && is_numeric($entity['bill_port_counter_in']) && is_numeric($entity['bill_port_counter_out'])) {
                          // Counters are lower, we must have wrapped. We'll take the measurement as the amount for this period.
                          $in_delta  = $data['in'];
                          $out_delta = $data['out'];
                          echo("Counters wrapped, delta fudged.\n");
                      } else {
                          // First update. delta is zero, only insert counters.
                          echo("No existing counters.\n");
                          $in_delta  = 0;
                          $out_delta = 0;
                      }}}{{            if ($in_delta == $data['in']) {
                          // Deltas are equal to counters. Clearly fail.
                          echo("In Deltas equal counters. Resetting.");
                          $in_delta  = 0;
                      }}}{{            if ($out_delta == $data['out']) {
                          // Deltas are equal to counters. Clearly fail.
                          echo("Out Deltas equal counters. Resetting.");
                          $out_delta  = 0;
          {{            }}}

          The last two if statements were added to try to prevent these sorts of spikes. We could potentially add another filter to zero any spike which is greater than some number, potentially not the device-reported port speed though, because that's quite unreliable.

          I suspect what's happening is that this device is returning a maximum value, overflowing the database field, i don't think it'll be from returning zeros, because we filter out the negative spike from zeroing the counters, then ignore the spike caused by the counters returning to normal.

          If you know what you're doing you could add some debugging output to dump a lot of the data to a text file to try to spot what exactly the device is returning.

          Also you could look in the database at the times the spike is occurring to see what exactly the deltas being recorded are.

          adam.

          adama Adam Armstrong added a comment - 2025/03/06 02:03 AM Hi, The only thing that could really cause this would be incorrect data being returned by the device, with the delta between two polled datapoints being very large, 192PB is probably the maximum size of the database field signed, or something there abouts. This should not really happen though.  How distant are the devices being polled from the polling system? What hardware are they? You should be able to see the deltas in the database, but we don't actually record the actual number returned by the polled device. Do you get similar odd behaviour in the RRD-based port graphs? RRD probably clips the number off because of the maximum value on the DS, though. Our logic to generate the in/out delta is quite simple:              if (is_numeric($period) && is_numeric($entity ['bill_port_counter_in'] ) && is_numeric($entity ['bill_port_counter_out'] ) && $data ['in'] >= $entity ['bill_port_counter_in'] && $data ['out'] >= $entity ['bill_port_counter_out'] ) {                 // Counters are higher or equal to before, seems legit.                 $in_delta  = int_sub($data ['in'] , $entity ['bill_port_counter_in'] );                 $out_delta = int_sub($data ['out'] , $entity ['bill_port_counter_out'] );                 echo("Counters valid, delta generated.\n");             } elseif (is_numeric($period) && is_numeric($entity ['bill_port_counter_in'] ) && is_numeric($entity ['bill_port_counter_out'] )) {                 // Counters are lower, we must have wrapped. We'll take the measurement as the amount for this period.                 $in_delta  = $data ['in'] ;                 $out_delta = $data ['out'] ;                 echo("Counters wrapped, delta fudged.\n");             } else {                 // First update. delta is zero, only insert counters.                 echo("No existing counters.\n");                 $in_delta  = 0;                 $out_delta = 0;             }}}{{            if ($in_delta == $data ['in'] ) {                 // Deltas are equal to counters. Clearly fail.                 echo("In Deltas equal counters. Resetting.");                 $in_delta  = 0;             }}}{{            if ($out_delta == $data ['out'] ) {                 // Deltas are equal to counters. Clearly fail.                 echo("Out Deltas equal counters. Resetting.");                 $out_delta  = 0; {{            }}} The last two if statements were added to try to prevent these sorts of spikes. We could potentially add another filter to zero any spike which is greater than some number, potentially not the device-reported port speed though, because that's quite unreliable. I suspect what's happening is that this device is returning a maximum value, overflowing the database field, i don't think it'll be from returning zeros, because we filter out the negative spike from zeroing the counters, then ignore the spike caused by the counters returning to normal. If you know what you're doing you could add some debugging output to dump a lot of the data to a text file to try to spot what exactly the device is returning. Also you could look in the database at the times the spike is occurring to see what exactly the deltas being recorded are. adam.

          People

            adama Adam Armstrong
            jmajor John Major
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              2025/02/13 05:10 PM
              Updated:
              2025/03/06 02:03 AM
              1 of 927
              • Atlassian Jira Project Management Software
              • About Jira
              • Report a problem

              Powered by a free Atlassian Jira open source license for Observium. Try Jira - bug tracking software for your team.

              Atlassian