Uploaded image for project: 'Observium'
  1. Observium
  2. OBS-3371

Webhook alerts stopped working

Details

    • Help
    • Resolution: Not A Bug
    • Major
    • None
    • Professional Edition
    • Alerting

    Description

      Hi,

      since june 10th we don't receive alerts on our webhooks anymore. a curl POST on the respective webhooks still works normally (source: Observium Server).

      the network between our Observium server and the target server of the webhooks is unchanged.
      On june 10th we only did an svn update and configured more alert checkers.

      Currently we have 22 alert checkers and 15 contacts (not all of which are used) configured.

      The alert checks themselves seem to work normally as they still alert us by mail (mails are only our backup).
      In the attachment you will find a document with the results of the test push and the curl post.

      Can you please help us with this problem? Without these notifications our on-call service is currently much more difficult.

      Thanks in advance.

       

      Best Regards
      Luca Protopapa
      Leucom

       

      Attachments

        Activity

          [OBS-3371] Webhook alerts stopped working
          Leucom Tobler Christoph added a comment -

          Hi Mike

          Sorry for the late reply and thank you for adding this option. It really looks like we have a problem with ipv6 on our Observium server.

          I completely forgot about v6. I thought there would be a fallback in case a transfer via ipv6 did not work.

          Thank you very much.

          Best regards
          Luca Protopapa

          Leucom Tobler Christoph added a comment - Hi Mike Sorry for the late reply and thank you for adding this option. It really looks like we have a problem with ipv6 on our Observium server. I completely forgot about v6. I thought there would be a fallback in case a transfer via ipv6 did not work. Thank you very much. Best regards Luca Protopapa
          landy Mike Stupalov added a comment - - edited

          Anyway, I added option for force http(s) requests with specified IP version.
          See in Global Settings -> Network -> HTTP(S) IP version.
          But note, this option global for all http requests (for all notifications).

          Added in rolling version 20.6.10491

          landy Mike Stupalov added a comment - - edited Anyway, I added option for force http(s) requests with specified IP version. See in Global Settings -> Network -> HTTP(S) IP version. But note, this option global for all http requests (for all notifications). Added in rolling version 20.6.10491
          landy Mike Stupalov added a comment -

          seems as at june 10th was broken IPv6 on your service.

          landy Mike Stupalov added a comment - seems as at june 10th was broken IPv6 on your service.
          landy Mike Stupalov added a comment - - edited

          Ohh, I found why main trouble. Service not responds by IPv6. (Curl by default use IPv4, but in observium by default used IPv6). Try your curl command with argument -6, you will get timeout error.

          landy Mike Stupalov added a comment - - edited Ohh, I found why main trouble. Service not responds by IPv6. (Curl by default use IPv4, but in observium by default used IPv6). Try your curl command with argument -6, you will get timeout error.
          landy Mike Stupalov added a comment -

          Hi, there is not was changes in notifications code.
          Trouble on used webhook service (https://matt.leucom.ch), with more long json data it's return errors:

          $ curl -i -X POST -H 'Content-Type: application/json; charset=utf-8' -d '{"ALERT_STATE":"ALERT","ALERT_URL":"https:\/\/observium.example.com\/device\/device=101\/tab=alert\/alert_entry=5665\/","ALERT_UNIXTIME":1512748845,"ALERT_TIMESTAMP":"2017-12-08 19:00:45 +03:00","ALERT_TIMESTAMP_RFC2822":"Fri, 08 Dec 2017 19:00:45 +0300","ALERT_TIMESTAMP_RFC3339":"2017-12-08T19:00:45+03:00","ALERT_ID":"5665","ALERT_MESSAGE":"SLA changed to FAILURE","CONDITIONS":"rtt_event ne ok (warning)","METRICS":"rtt_event = warning","DURATION":"4m 14s (2017-12-08 18:56:31)","ENTITY_LINK":"<a href=\"https:\/\/observium.example.com\/device\/device=101\/tab=slas\/id=500\/\" class=\"entity-popup \" data-eid=\"500\" data-etype=\"sla\">SLA #22 (10.0.1.25)<\/a>","ENTITY_NAME":"SLA #22 (10.0.1.25)","ENTITY_ID":"500","ENTITY_TYPE":"sla","ENTITY_DESCRIPTION":null,"DEVICE_HOSTNAME":"device.hostname","DEVICE_SYSNAME":"device.sysname","DEVICE_ID":"101","DEVICE_LINK":"<a href=\"https:\/\/observium.example.com\/device\/device=101\/\" class=\"entity-popup \" data-eid=\"101\" data-etype=\"device\">device.hostname<\/a>","DEVICE_HARDWARE":"CISCO2901\/K9","DEVICE_OS":"Cisco IOS 15.4(3)M8 (UNIVERSALK9)","DEVICE_LOCATION":"North Pole, 101 Santa Claus Lane","DEVICE_UPTIME":"7 days, 23h 44m 44s","DEVICE_REBOOTED":"2017-11-30 19:15:51","TITLE":"ALERT: [device.hostname] [sla] [SLA #22 (10.0.1.25)] SLA changed to FAILURE"}' https://matt.leucom.ch/hooks/1i63c1bcz7gi8x1be1acg8k3xo ; echo
          		 
          HTTP/1.1 100 Continue
          		 
           
          		 
          HTTP/1.1 400 Bad Request
          		 
          Content-Type: application/json
          		 
          X-Request-Id: e4ykuq1rzfnbmjmtocneou4kkh
          		 
          X-Version-Id: 5.23.0.5.23.1.39287ef7e12a22a189490da4349d086e.false
          		 
          Date: Fri, 12 Jun 2020 11:01:40 GMT
          		 
          Content-Length: 155
          		 
           
          		 
          {"id":"web.incoming_webhook.text.app_error","message":"No text specified.","detailed_error":"","request_id":"e4ykuq1rzfnbmjmtocneou4kkh","status_code":400}

          Need more detailed information, which service is used here and API description (seems as not valid Webhook api).

          landy Mike Stupalov added a comment - Hi, there is not was changes in notifications code. Trouble on used webhook service ( https://matt.leucom.ch ), with more long json data it's return errors: $ curl -i -X POST -H 'Content-Type: application/json; charset=utf-8' -d '{"ALERT_STATE":"ALERT","ALERT_URL":"https:\/\/observium.example.com\/device\/device=101\/tab=alert\/alert_entry=5665\/","ALERT_UNIXTIME":1512748845,"ALERT_TIMESTAMP":"2017-12-08 19:00:45 +03:00","ALERT_TIMESTAMP_RFC2822":"Fri, 08 Dec 2017 19:00:45 +0300","ALERT_TIMESTAMP_RFC3339":"2017-12-08T19:00:45+03:00","ALERT_ID":"5665","ALERT_MESSAGE":"SLA changed to FAILURE","CONDITIONS":"rtt_event ne ok (warning)","METRICS":"rtt_event = warning","DURATION":"4m 14s (2017-12-08 18:56:31)","ENTITY_LINK":"<a href=\"https:\/\/observium.example.com\/device\/device=101\/tab=slas\/id=500\/\" class=\"entity-popup \" data-eid=\"500\" data-etype=\"sla\">SLA #22 (10.0.1.25)<\/a>","ENTITY_NAME":"SLA #22 (10.0.1.25)","ENTITY_ID":"500","ENTITY_TYPE":"sla","ENTITY_DESCRIPTION":null,"DEVICE_HOSTNAME":"device.hostname","DEVICE_SYSNAME":"device.sysname","DEVICE_ID":"101","DEVICE_LINK":"<a href=\"https:\/\/observium.example.com\/device\/device=101\/\" class=\"entity-popup \" data-eid=\"101\" data-etype=\"device\">device.hostname<\/a>","DEVICE_HARDWARE":"CISCO2901\/K9","DEVICE_OS":"Cisco IOS 15.4(3)M8 (UNIVERSALK9)","DEVICE_LOCATION":"North Pole, 101 Santa Claus Lane","DEVICE_UPTIME":"7 days, 23h 44m 44s","DEVICE_REBOOTED":"2017-11-30 19:15:51","TITLE":"ALERT: [device.hostname] [sla] [SLA #22 (10.0.1.25)] SLA changed to FAILURE"}' https://matt.leucom.ch/hooks/1i63c1bcz7gi8x1be1acg8k3xo ; echo HTTP/1.1 100 Continue   HTTP/1.1 400 Bad Request Content-Type: application/json X-Request-Id: e4ykuq1rzfnbmjmtocneou4kkh X-Version-Id: 5.23.0.5.23.1.39287ef7e12a22a189490da4349d086e.false Date: Fri, 12 Jun 2020 11:01:40 GMT Content-Length: 155   {"id":"web.incoming_webhook.text.app_error","message":"No text specified.","detailed_error":"","request_id":"e4ykuq1rzfnbmjmtocneou4kkh","status_code":400} Need more detailed information, which service is used here and API description (seems as not valid Webhook api).

          People

            landy Mike Stupalov
            Leucom Tobler Christoph
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: