Policy discussion: Unix-Agent, scripts, paths, future development

This discussion is related to: BUG#164

Hello guys
I'm new to observium, so excuse me if I'm not familiar with some concepts.
I want to initiate a discussion about future changes in observium and in particular the unix agent scripts. I'm interested in writing more checks (postgresql, dhcp, openvz, etc) but after a discussion in irc yesterday, I want to clear some concepts before I proceed with writing anything.
My goal is to have as much as posible a working configuration out of the box. I'm looking for almost zero configuration. So when a new use comes to observium and copies all scripts to his check_mk directory, he should expect a proper result every time, and should not get broken graphs. The user should not care if he has mysql, dmidecode or memcached installed and configured correct, before he copies the scripts.

Please provide constructive criticism and ideas/problems that I have missed.

1) Script output
a) There is a problem with observium that if the script does not return the proper output, the graph is broken and sometimes can break other graphs too. Some of the scripts right now does not check their output at all, witch leads to problems. A check can also be implemented in the poller, but I'm not really sure how to do it YET and what good does it do. So I propose that from now on, there will be no "Configuration file missing", "Data fetch failure" or anything like that in the scripts.

The script should succeed or return nothing at all.

b) There should be proper checks if the needed binaries and configuration files are installed BEFORE they are executed/included OR you should check the result of the execute/include if it was successful. There was a discussion yesterday about using full paths in scripts or relaying on $PATH to search for your binary.

• If I use full path (/usr/bin/program), the script is not very portable out of the box, because the "program" may be in different location in different OSes. But setting the full path can give me the ability to check if the binary exists before I execute it. A nice waring // EDIT LINE HERE // DO NOT EDIT BENEATH can be put at the begining of every script. - This option is what I prefer.
• Another option is to try to discover the binary in $PATH, or `whereis`, or `type`. This is not very portable because some systems return different output
• Third option is to rely on $PATH. This is very convenient way to work out of the box for every OS, but there is no way to check if the binary exists before trying to execute it. For example when I put $ntpq="ntpq"; the program is executed without checking if "ntpq" exists, So we should be checking the output of the command if it is valid. This option is again a bit harder to implement because of system differences (sh: ntpq does not exists // ntpq program not found // -bash: ntpq: command not found// etc).

2) Versions
Some times, the output of a command is changed between program versions. For example old versions of NFS have fewer stats than the newer versions, witch leads to different rrd files and different graphs.. So clearly there should be a way to check the version in the script. Here is an example.
<<<myprogram>>>
version1
stats:534
stats2:31
<<<myprogram>>>
version2
stats:235
stats2:523
stats3:28345

Another way is to create different scripts for different versions like:
<<<myprogram-v1>>>

<<<myprogram-v2>>>

I have no clear opinion on both options.

3) Perl/PHP/Python Modules
Sometimes your scripts require modules to get the needed stats. They should be documented in the script, a check if they exist should be included. Modules are system independent but not all systems have them installed.

I'm willing to write/edit the existing documentation when we decide how this stuff should work. And of my new additions a full documentation is going to be provided.