Uploaded image for project: 'Observium'
  1. Observium
  2. OBS-3187

IPMI chassis en power supply support

Details

    • Bug
    • Resolution: Fixed
    • Major
    • None
    • Professional Edition
    • Unix Agent

    Description

      The function parse_ipmitool_sensor() is not working correctly due to the empty discrete value in includes/definitions/entities/sensors.inc.php.

      The following statements in parse_ipmitool_sensor() (includes/entities/sensor.inc.php) returns true if the $unit is discrete:

      if (isset($config['ipmi_unit'][$unit])) {
      

      and the code below (from OBS-3028) will never run:

      } elseif ($unit == 'discrete') {
      

      because an empty value is set in the array, so isset() returns true.

      The attached patch will remove empty value from the array. Another option could be checking is there is an actual value in the unit-array:

      if (isset($config['ipmi_unit'][$unit]) && $config['ipmi_unit'][$unit]) {
      

      Attachments

        1. ipmi_discrete.diff
          0.5 kB
        2. ipmi_patch_bernard.diff
          4 kB
        3. ipmi_patch_rework.diff
          4 kB
        4. ipmi_sensors.diff
          3 kB
        5. ipmi_sensors.diff
          3 kB
        6. ipmitool-sdr-r730.txt
          69 kB
        7. ipmitool-sensors-r730.txt
          21 kB
        8. poller_patched.log
          115 kB
        9. poller.log
          111 kB
        10. poller-r730.txt
          329 kB

        Issue Links

          Activity

            [OBS-3187] IPMI chassis en power supply support

            Sure. I've attached the poller output, along with the output from calling 'ipmitool sensor' and 'ipmitool -v sdr' directly. The latter file shows that (for example) the sensor named 'PS1 PG Fail' is of the 'Voltage' type, not 'Power Supply', so its assertions don't match what the existing entPhysicalClass 'powersupply' matching does.

            ipmitool-sdr-r730.txt

            andrewbonney Andrew Bonney added a comment - Sure. I've attached the poller output, along with the output from calling 'ipmitool sensor' and 'ipmitool -v sdr' directly. The latter file shows that (for example) the sensor named 'PS1 PG Fail' is of the 'Voltage' type, not 'Power Supply', so its assertions don't match what the existing entPhysicalClass 'powersupply' matching does. ipmitool-sdr-r730.txt

            IPMI sensors polled without changes.

            landy Mike Stupalov added a comment - IPMI sensors polled without changes.

            Thanks Mike. Would the sensor output still be useful?

            andrewbonney Andrew Bonney added a comment - Thanks Mike. Would the sensor output still be useful?

            Note, in r10396 discrete statuses disabled by default, while support unstable and many false positives.
            For enable, go to Global settings -> Entities -> check "Enable polling IPMI discrete sensors".

            landy Mike Stupalov added a comment - Note, in r10396 discrete statuses disabled by default, while support unstable and many false positives. For enable, go to Global settings -> Entities -> check "Enable polling IPMI discrete sensors".

            Ok, now should be fixed in r10395.

            landy Mike Stupalov added a comment - Ok, now should be fixed in r10395.

            andrewbonney Please also attach debug for your devices (with fail description and dell hardware), I need to see more ipmi output examples:

            ./poller.php -d -m unix-agent,sensors,status -h <device> 

            landy Mike Stupalov added a comment - andrewbonney Please also attach debug for your devices (with fail description and dell hardware), I need to see more ipmi output examples: ./poller.php -d -m unix-agent,sensors,status -h <device>
            andrewbonney Andrew Bonney added a comment - - edited

            The attached patch removes the state inversions from the previous version. Looking at the IPMI spec and the live sensor output I have access to I'm not sure it will be possible to improve the various discrete sensors without calling some additional IPMI commands. It's only when you know the sensor type behind each description that you can identify what their assertions really mean. These may be a good thing (power supply is present) or a bad thing (chassis intrusion), but can be represented by the same value.

            ipmi_sensors.diff

            andrewbonney Andrew Bonney added a comment - - edited The attached patch removes the state inversions from the previous version. Looking at the IPMI spec and the live sensor output I have access to I'm not sure it will be possible to improve the various discrete sensors without calling some additional IPMI commands. It's only when you know the sensor type behind each description that you can identify what their assertions really mean. These may be a good thing (power supply is present) or a bad thing (chassis intrusion), but can be represented by the same value. ipmi_sensors.diff
            andrewbonney Andrew Bonney added a comment - - edited

            We're having the same issue, although I believe there's an additional bug which presents itself if you're using ipmitool remotely rather than in conjunction with the unix-agent. If this is the case then even with the suggested patch the values don't get processed due to a lookup for "$config['*ipmi*']['states']['*unix-agent-state'*]" in includes/definitions/entities/status.inc.php.

            The attached patch resolves this for us, although note that this includes two additional changes to value handling. We have some sensors described as 'PS1 PG Fail' for example, which need the power supply status flag inverting. It also appears that the sensors classed as 'other' are inverted (noted on a Dell R730), but I don't have enough data to say if this will be consistent for every IPMI implementation.

            Edit: After a further look I'd certainly exclude the state inversions. I'm looking into whether this can be done any more consistently across implementations.

            ipmi_sensors.diff

            andrewbonney Andrew Bonney added a comment - - edited We're having the same issue, although I believe there's an additional bug which presents itself if you're using ipmitool remotely rather than in conjunction with the unix-agent. If this is the case then even with the suggested patch the values don't get processed due to a lookup for "$config ['*ipmi*'] ['states'] ['*unix-agent-state'*] " in includes/definitions/entities/status.inc.php. The attached patch resolves this for us, although note that this includes two additional changes to value handling. We have some sensors described as 'PS1 PG Fail' for example, which need the power supply status flag inverting. It also appears that the sensors classed as 'other' are inverted (noted on a Dell R730), but I don't have enough data to say if this will be consistent for every IPMI implementation. Edit: After a further look I'd certainly exclude the state inversions. I'm looking into whether this can be done any more consistently across implementations. ipmi_sensors.diff

            I attached the logs

            veldenb Bernard van der Velden added a comment - I attached the logs

            Without the patch data from the section '##### Module Start: status #####' is missing.
            With the patch the sensors are recognized:

            +---------------+------------------+-------+--------+-------+--------+---------------------+
            | Descr         | Type             | Index | Origin | Value | Status | Last Changed        |
            +---------------+------------------+-------+--------+-------+--------+---------------------+
            | Chassis Intru | unix-agent-state | 41    | agent  | ok    | ok     | 2020-01-02 15:35:43 |
            | PS1 Status    | unix-agent-state | 42    | agent  | ok    | ok     | 2019-01-02 13:17:13 |
            | PS2 Status    | unix-agent-state | 43    | agent  | ok    | ok     | 2019-01-02 13:17:13 |
            +---------------+------------------+-------+--------+-------+--------+---------------------+
            

            The logs from the unix-agent which also reads ipmi data, the issue is not in the ipmi module:

            ./poller.php -d -m unix-agent,status -h storage2 
            

            poller.log poller_patched.log

            veldenb Bernard van der Velden added a comment - Without the patch data from the section '##### Module Start: status #####' is missing. With the patch the sensors are recognized: +---------------+------------------+-------+--------+-------+--------+---------------------+ | Descr | Type | Index | Origin | Value | Status | Last Changed | +---------------+------------------+-------+--------+-------+--------+---------------------+ | Chassis Intru | unix-agent-state | 41 | agent | ok | ok | 2020-01-02 15:35:43 | | PS1 Status | unix-agent-state | 42 | agent | ok | ok | 2019-01-02 13:17:13 | | PS2 Status | unix-agent-state | 43 | agent | ok | ok | 2019-01-02 13:17:13 | +---------------+------------------+-------+--------+-------+--------+---------------------+ The logs from the unix-agent which also reads ipmi data, the issue is not in the ipmi module: ./poller.php -d -m unix-agent,status -h storage2 poller.log poller_patched.log
            landy Mike Stupalov added a comment - - edited

            Hrm, not fully understand trouble, please attach debug polling output for:

            ./poller.php -d -m ipmi -h <device>
            

            for see real ipmitool output from your server.

            landy Mike Stupalov added a comment - - edited Hrm, not fully understand trouble, please attach debug polling output for: ./poller.php -d -m ipmi -h <device> for see real ipmitool output from your server.

            People

              landy Mike Stupalov
              veldenb Bernard van der Velden
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: