Details
-
Improvement
-
Resolution: Fixed
-
Minor
-
None
-
Professional Edition
-
CentOS 7, Python 3.4.5, PHP 7.2 from remirepo
Description
When the log file is a named fifo pipe or stdout (for injecting logs directly into journald) and the poller wrapper is running on Python 3, it fails with an "OSError: [Errno 29] Illegal seek" on the `f = open(log_path,'a')` line.
This is a known but unsolved python 3 bug as can be seen here https://bugs.python.org/issue27805 and https://bugs.python.org/issue20074.
The solution we used for now is to overwrite the open function to the open mode if appropriate, we did not want to modify the poller-wrapper directly to avoid issues with upgrades.
`python3 poller-wrapper-wrapper.py poller-wrapper.py -w 10`
poller-wrapper-wrapper.py is attached.
Attachments
Activity
| Status | Original: Resolved [ 5 ] | New: Closed [ 6 ] |
| Resolution | New: Fixed [ 1 ] | |
| Status | Original: In Progress [ 3 ] | New: Resolved [ 5 ] |
We need all logs to go to the systemd journal, and not to file. We accomplish that by using the following systemd files to create and then listen on special FIFO files:
|
observium-logs@.socket |
[Unit]
|
Description=Plain-Old-Stupid-Log-File to Journal Transfer Assistant (%I.log)
|
|
|
[Socket]
|
SocketUser=observium
|
SocketGroup=apache
|
SocketMode=0660
|
ExecStartPre=/usr/bin/rm -f /var/www/observium/logs/%I.log
|
ListenFIFO=/var/www/observium/logs/%I.log
|
RemoveOnStop=true
|
|
|
[Install]
|
WantedBy=sockets.target
|
|
observium-logs@.service |
[Unit]
|
Description=Observium Logs Proxy (%I.log)
|
Requires=observium-logs@%i.socket
|
|
|
[Service]
|
User=observium
|
ExecStart=/usr/bin/cat /dev/fd/3
|
We then enable, and start those services like this:
|
commands |
systemctl enable --now observium-logs@cleanup.socket
|
systemctl enable --now observium-logs@db.socket
|
systemctl enable --now observium-logs@debug.socket
|
systemctl enable --now observium-logs@debug_auth.socket
|
systemctl enable --now observium-logs@housekeeping.socket
|
systemctl enable --now observium-logs@observium.socket
|
systemctl enable --now observium-logs@poller.socket
|
systemctl enable --now observium-logs@rrd.socket
|
systemctl enable --now observium-logs@update\x2derrors.socket
|
At this point, we're done and ready for logs. We get all log messages into the systemd journal, and are able to view them like so:
|
example |
# journalctl -u observium-logs@db --since='2 minutes ago'
|
-- Logs begin at Sat 2019-12-28 12:52:44 UTC, end at Tue 2020-02-18 18:22:56 UTC. --
|
Feb 18 18:21:08 obs3 cat[696]: [2020/02/18 18:21:08 +0000] poller.php(119863): Failed dbQuery (#1146 - Table 'observium.lsp' doesn't exist), Query: SELECT * FROM `lsp` WHERE `device_id` = '61'
|
Feb 18 18:21:53 obs3 cat[696]: [2020/02/18 18:21:53 +0000] poller.php(119859): Failed dbQuery (#1146 - Table 'observium.lsp' doesn't exist), Query: SELECT * FROM `lsp` WHERE `device_id` = '72'
|
or, a different example:
|
example |
# journalctl -u 'observium-logs@*' --since='2 minutes ago'
|
-- Logs begin at Sat 2019-12-28 12:52:44 UTC, end at Tue 2020-02-18 18:23:44 UTC. --
|
Feb 18 18:22:26 obs3 cat[700]: [2020/02/18 18:22:26 +0000] poller.php(119859): /var/www/observium/poller.php: device.example.com - 1 devices polled in 145.3 secs
|
Feb 18 18:22:26 obs3 cat[700]: [2020/02/18 18:22:26 +0000] poller-wrapper2.py(119848): /var/www/observium/poller-wrapper2.py: processed 15 devices in 146.38 seconds with 10 threads, load average (5min) 0.19
|
Feb 18 18:22:26 obs3 cat[700]: [2020/02/18 18:22:26 +0000] discovery.php(120878): /var/www/observium/discovery.php: new - 0 devices discovered in 0.002 secs
|
| Status | Original: Pending Response [ 10000 ] | New: In Progress [ 3 ] |
| Status | Original: In Review [ 10101 ] | New: Pending Response [ 10000 ] |
Hey, I attached a patch as discussed ... months ago on IRC. It basically fixes the issue our poller wrapper mitigated.
p.s. maybe a good idea to start switching to python3?
| Status | Original: Pending Response [ 10000 ] | New: In Review [ 10101 ] |
| Attachment | New: OBS-2631.patch [ 17077 ] |
Commited (with little difference) in r10295.
Thanks.