Uploaded image for project: 'Observium'
  1. Observium
  2. OBS-471

Preliminary support for Proxmox VE guest monitoring

Details

    Description

      This patch against r4386 adds preliminary support for
      monitoring guests running on Proxmox VE platform. Both
      OpenVZ containers and KVM virtual machines are supported.

      Poller connects to Proxmox node via Proxmox's PVE2 API
      and polls stats from all guests on all hosts in a cluster.
      Stores RRD files under proxmox subdir for each device.

      Included PVE2 API client code used by poller is taken from
      https://github.com/CpuID/pve2-api-php-client.

      It also adds "Proxmox VE Guests" tab on the device page
      that displays CPU, Memory, Disk Usage, Disk I/O and Network
      Traffic statistics for each guest.

      Attachments

        Issue Links

          Activity

            [OBS-471] Preliminary support for Proxmox VE guest monitoring
            landy Mike Stupalov made changes -
            Workflow Original: classic default workflow [ 10942 ] New: Observium workflow [ 13799 ]
            anthonysomerset Anthony Somerset made changes -
            Link New: This issue is cloned by OBSERVIUM-494 [ OBSERVIUM-494 ]
            adama Adam Armstrong made changes -
            Status Original: Resolved [ 5 ] New: Closed [ 6 ]

            You are only considering the overall cost on the Observium side. If, for example, MySQL is running on another host, cost of forking rrdtool and database load are two completely separate things. While I totally agree about impact of fork()ing, your way still creates unnecessary load on the database server itself, which is even worse when both Observium and MySQL share the same limited pool of IO/s on a resource constrained (virtual) machine. With many concurrent Observium sessions it can sum up to quite a bit, especially if you are monitoring hundreds of devices and hit the moment when both discovery and polling processes run in the background. Even worse if you are using the syslog feature as well (should be moved from MySQL to ElasticSearch, or lean on Graylog2/Logstash, btw).

            I also agree that Observium is not CPU bound, but it is IO bound, which is a lot worse. I do not agree, however, that relaxing a few simple rules makes code unmaintainable. I'm the first person to impose rules and procedures on others and I expect my employees to comply at all times, but rules aren't written in stone and can evolve over time to better fit new features, new requirements.

            But, hey ... It's your project. I sincerely just wanted to help.

            marko Marko Dinic added a comment - You are only considering the overall cost on the Observium side. If, for example, MySQL is running on another host, cost of forking rrdtool and database load are two completely separate things. While I totally agree about impact of fork()ing, your way still creates unnecessary load on the database server itself, which is even worse when both Observium and MySQL share the same limited pool of IO/s on a resource constrained (virtual) machine. With many concurrent Observium sessions it can sum up to quite a bit, especially if you are monitoring hundreds of devices and hit the moment when both discovery and polling processes run in the background. Even worse if you are using the syslog feature as well (should be moved from MySQL to ElasticSearch, or lean on Graylog2/Logstash, btw). I also agree that Observium is not CPU bound, but it is IO bound, which is a lot worse. I do not agree, however, that relaxing a few simple rules makes code unmaintainable. I'm the first person to impose rules and procedures on others and I expect my employees to comply at all times, but rules aren't written in stone and can evolve over time to better fit new features, new requirements. But, hey ... It's your project. I sincerely just wanted to help.
            adama Adam Armstrong made changes -
            Resolution New: Incomplete [ 4 ]
            Status Original: Open [ 1 ] New: Resolved [ 5 ]

            We don't accept code which doesn't conform to our standard conventions for doing things because it unnecessarily decreases the maintainability of the project.

            Contributing code to an open source project is like contributing a baby to an orphanage, the future maintenance and updating of the code usually takes more time than it took to write the code in the first place.

            We have standard ways of doing things because it makes it quick and easy for us to maintain the code, and for people to debug problems.

            You're also gravely mistaken on the performance hit of MySQL queries, particularly this kind of query, and particularly in the context of graph.php.

            The whole purpose of graph.php is to fork rrdtool, load one or more RRDs and build a graph, all of which costs at least two orders of magnitude more than the single MySQL lookup to fetch the entity data from the database.

            Maintaining a large body of code which is not generally CPU performance bound almost always veers in favour of writing maintainable code rather than sacrificing maintainability in favour of performance.

            adama Adam Armstrong added a comment - We don't accept code which doesn't conform to our standard conventions for doing things because it unnecessarily decreases the maintainability of the project. Contributing code to an open source project is like contributing a baby to an orphanage, the future maintenance and updating of the code usually takes more time than it took to write the code in the first place. We have standard ways of doing things because it makes it quick and easy for us to maintain the code, and for people to debug problems. You're also gravely mistaken on the performance hit of MySQL queries, particularly this kind of query, and particularly in the context of graph.php. The whole purpose of graph.php is to fork rrdtool, load one or more RRDs and build a graph, all of which costs at least two orders of magnitude more than the single MySQL lookup to fetch the entity data from the database. Maintaining a large body of code which is not generally CPU performance bound almost always veers in favour of writing maintainable code rather than sacrificing maintainability in favour of performance.

            Marko - i found the patch useful, i also the +1 for keeping queries down

            now i just need to find a way to continue to make sure the patch holds with updates until adam releases it

            although judging by the removal of patches this doesnt seem likely?

            anthonysomerset Anthony Somerset added a comment - Marko - i found the patch useful, i also the +1 for keeping queries down now i just need to find a way to continue to make sure the patch holds with updates until adam releases it although judging by the removal of patches this doesnt seem likely?
            adama Adam Armstrong made changes -
            Attachment Original: proxmox_ve_v2.patch [ 11003 ]
            adama Adam Armstrong made changes -
            Attachment Original: proxmox_ve_fix.patch [ 11004 ]
            adama Adam Armstrong made changes -
            Attachment Original: proxmox_ve.sql [ 11001 ]

            People

              adama Adam Armstrong
              marko Marko Dinic
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: