Download the PHP package geoffroy-aubry/supervisor without Composer

Supervisor

 Estimated code coverage: 87% (553 of 638 lines).

Oversee script execution, recording stdout, stderr and exit code with timestamping, and ensure email notifications will be sent (on startup, success, warning or error)… plus many other features.

Technologies : Supervisor is in Bash, only unit tests are in PHP.

Supported OS : Debian/Ubuntu Linux, FreeBSD/OS X.

Table of Contents

• Features
  • General
  • Notifications
  • Ergonomics
  • Code quality
• Requirements
• Usage
  • Help on command prompt
  • Exit status
  • Successful execution
  • Errors
  • Tags
  • Locks
  • Customized mails
  • Supervisor supervised
  • Archiving
  • Monitoring
  • Summary
• Installation
• Copyrights & licensing
• Change log
• Continuous integration
• Git branching model

Features

General

• Oversee script execution, recording stdout, stderr and exit code
• Prefix each line of stdout with timestamps with hundredths of seconds.
• An execution ID uniquely identify each script execution. Used in: supervisor.info.log, supervisor.error.log, <script>_<exec_id>.info.log and <script>_<exec_id>.error.log.
• Possibility to block parallel script execution (only on Debian/Ubuntu Linux). Useful if an execution takes too long…
• You can specify a configuration file to load in addition to the default one (overloading).
• Entirely configurable tag system allowing executed scripts to dynamically specify via stdout warning, mailto, mail attachment, instigator…
• Automatic log archiving mechanism.
• Display a summary of supervisor's activity during last days, including final status per day and per supervised script.
• Handle CSV output, with definition of field separator, field enclosure and number of the field to watch in configuration file.
• Add proper signal handling and interrupts: SIGHUP, SIGINT, SIGQUIT and SIGTERM now transmitted to supervised script.

Notifications

• Ensure email notifications will be sent on following events (configurable): startup, success, warning or error. Catch exit code, fatal error, exceptions…
• All mail contains content of stdout and stderr of executed script in Gzip attachment.
• Mails are fully customizable. Just provide your file on command line.
• Possibility to inject multiple external parameters on command line into customized emails.
• Supervisor itself is monitoring by another process, sending critical email notifications using an exponential backoff algorithm in minute increments.

Ergonomics

• During supervision and in addition of all log files, Supervisor displays in real time outputs, warning and errors of overseen script, adding timestamp and respecting both colors and indentation. Name of all log files are printed too.

Code quality

• Bash code executed with following directives:
  • set -o nounset: treat unset variables and parameters other than the special parameters @ or * as an error when performing parameter expansion. An error message will be written to the standard error, and a non-interactive shell will exit.
  • set -o pipefail: The return value of a pipeline is the value of the last (rightmost) command to exit with a non-zero status, or zero if all commands in the pipeline exit successfully.
• A lot of unit tests (via PHPUnit through a PHP wrapper) covering almost the whole application.

Requirements

• Bash v4 (2009) and above
• mutt, to send email notifications (apt-get install mutt, or brew install mutt)
• Unit tests require PHP >= 5.3.3 and Xdebug
• On FreeBSD/OS X only, you need to install these GNU tools:

Tested on Debian/Ubuntu Linux and FreeBSD/OS X.

Usage

Help on command prompt

Displayed by:

Text version

Description
    Oversee script execution, recording stdout, stderr and exit code with timestamping,
    and ensure email notifications will be sent (on start, success, warning or error).

Usage
    supervisor [OPTION]… <script-path> [<script-parameters>]
    supervisor [-c <conf-file>] --archive=<min-days>
    supervisor [-c <conf-file>] --monitor
    supervisor [-c <conf-file>] --summarize

Options
    --archive=<min-days>
        Archive in Gzip supervisor's logs older than <min-days>.

    -c <conf-file>, --conf=<conf-file>
        Specify a configuration file to load in addition to the default one,
        namely '/conf/supervisor-dist.sh'.

    --customized-mails=<file>
        Path to a Bash script customizing sent mails by redefining some of
        the sendMailOn[Init|Success|Warning|Error]() functions.
        See --param option.

    --exec-id=<string>
        Allow to force execution id, used in mails and name of logs.
        By default: YYYYMMDDHHIISS_XXXXX, where X are random digits.

    --extra-param-mode=[only-value|with-name|none]
        Two extra parameters are added to the end of <script-parameters>: a unique
        execution ID and name of file recording script's stderr. with-name mode prefixes
        values with '--exec-id=' and '--error-log-file=' respectively.
        By default: only-value.

    -h, --help
        Display this help.

    --mail-instigator=<email>
        Specify who executed the supervisor.

    --mail-to=<email>
        Add a new recipient's email address. Multiple --mail-to options may be used.

    --monitor
        Check whether supervisor's error log file is empty. If not, then send critical
        email notifications using an exponential backoff algorithm in minute increments.
        Typically called every minute with a cron job:
            * * * * * <user> /path/to/supervisor.sh --conf=<conf-file> --monitor

    -p <key>=<value>, --param=<key>=<value>
        Allow to inject multiple external parameters into customized emails.
        Assign the value <value> to the Bash variable $EXT_<key>.
        See --customized-mails option.

    --summarize=<max-nb-days>
        Display a summary of supervisor's activity during last <max-nb-days> days,
        including final status per day and per supervised script.
        Also send this summary by email.

    <script-path>
        Executable script to oversee.

    <script-parameters>
        Optional oversaw script's parameters.

Exit status
      0 if and only if no error
     65 Missing script name!
     66 Script '…' not found!
     67 Script '…' is not executable!
     68 Exit code changed from 0 to 68 due to errors.
     69 Another instance of '…' is still running with supervisor!
     71 Customized mails file not found: '…'
     72 Invalid Mutt command: '…'
    xxx Any code not null returned by user script

Exit status

List of exit status:

• 0 If and only if no error
• 65 Missing script name!
• 66 Script '…' not found!
• 67 Script '…' is not executable!
• 68 Exit code changed from 0 to 68 due to errors.
• 69 Another instance of '…' is still running with supervisor!
• 71 Customized mails file not found: '…'
• 72 Invalid Mutt command: '…'
• Any code not null returned by user script

Successful execution

An execution is successful if and only if stderr is empty, exit status is 0 and there are no warnings.

A short example, here in Bash but no matter the language:

Output:

If SUPERVISOR_MAIL_SEND_ON_STARTUP is set to 1 in configuration file:

Note that Supervisor add two parameters in addition of script parameters:

• a unique execution ID
• name of file recording script's stderr

If SUPERVISOR_MAIL_SEND_ON_SUCCESS is set to 1 in configuration file:

Content of supervisor.info.log.20131006223705_01765.gz attachment:

Content of bash_colored_simple.sh.20131006223705_01765.info.log.gz attachment:

Note that colors are stripped but indentation is kept.

Errors

An execution end in error status if either stderr is not empty or if exit status is different from 0.

PHP fatal error

We will oversee a PHP script throwing a fatal error:

Output:

If SUPERVISOR_MAIL_SEND_ON_ERROR is set to 1 in configuration file:

Content of supervisor.info.log.20131006223703_01618.gz attachment:

Content of php_fatal_error.php.20131006223703_01618.info.log.gz attachment:

Content of php_fatal_error.php.20131006223703_01618.error.log.gz attachment:

Other errors

Similarly, scripts provided below end with an error status.

It should be noted that when stderr is not-empty while exit status is 0, then Supervisor change exit status from 0 to 68 (see Exit status).

PHP notice

Content of tests/resources/php_notice.php:

Result in php_notice.php.[…].error.log.gz attachment:

PHP exception

Content of tests/resources/php_exception.php:

Result in php_exception.php.[…].error.log.gz attachment:

PHP stderr

Content of tests/resources/php_stderr.php:

Result in php_stderr.php.[…].error.log.gz attachment:

Exit not null

Content of tests/resources/bash_exit_not_null.sh:

Result in bash_exit_not_null.sh.[…].error.log.gz attachment:

Tags

Tags allow user scripts to send some information to supervisor.

Tags are strings between square brackets —for example: [warning]— used in output of supervised scripts. Tags must be at the beginning of a line or only preceded by tabulations —defined by SUPERVISOR_LOG_TABULATION— or spaces. A tag concerns all the line.

Warnings

If a successful executed script contains warning tags in its output, then its status will be changed from Success to Warning.

Default configuration: SUPERVISOR_WARNING_TAG='[WARNING]'.

If SUPERVISOR_MAIL_SEND_ON_WARNING is set to 1 in configuration file:

Please note that warnings are listed with 2 lines of context.

Debug traces

By default, messages preceded by a debug tag are hidden in Supervisor's output (but still present in <script>_<exec_id>.info.log). Set SUPERVISOR_SHOW_DEBUG_MSG to 1 to display them.

Default configuration:

E-mail related tags

Default configuration:

TO DOCUMENT

Locks

Sometimes there's the need to ensure that a script is only executed one time. In order to achieve this, set SUPERVISOR_LOCK_SCRIPT to 1 in your configuration file. Only available on Debian/Ubuntu Linux, otherwise leave 0.

Example of output:

Example of sent mail:

Customized mails

TO DOCUMENT

Supervisor supervised

TO DOCUMENT

Archiving

Archive in Gzip format all supervisor's logs older than <min-days>.

Monitoring

Check whether supervisor's error log file is empty. If not, then send critical email notifications using an exponential backoff algorithm in minute increments. Files supervisor.info.log and supervisor.error.log will be attached in Gzip format.

Typically called every minute with a cron job:

Mail generated through unit tests:

Summary

Display a summary of supervisor's activity during last <max-nb-days> days, including final status per day and per supervised script. Also send this summary by email to SUPERVISOR_MAIL_TO list.

Installation

1. Move to the directory where you wish to store the source and clone the repository:

2. You should be on stable branch. If not, switch your clone to that branch:

3. Create your own configuration file and adapt it:

   Note: file conf/supervisor-dist.sh is always loaded. Then optional configuration file specified with option --conf is loaded (overwriting). But if no option --conf is found, then conf/supervisor.sh is loaded if exists.

4. If you have to supervise scripts producing CSV output, then install Awk CSV parser component:

   • Manually:

     • See Installation section of Awk CSV parser's README.md.

     • Then update SUPERVISOR_CSV_PARSER constant in /conf/supervisor.sh:

   • Via composer (require PHP >= 5.3.3):

     See http://getcomposer.org/doc/00-intro.md#installation-nix for more information to install composer.

5. You can create a symlink to supervisor.sh:

It's ready for use:

Copyrights & licensing

Licensed under the GNU Lesser General Public License v3 (LGPL version 3). See LICENSE file for details.

Change log

See CHANGELOG file for details.

Continuous integration

 Estimated code coverage: 87% (553 of 638 lines).

Require PHP >= 5.3.3 and Xdebug.

Create your own configuration file and adapt it:

Launch unit tests with PHPUnit:

On FreeBSD/OS X you must exclude tests about exclusive execution (SUPERVISOR_LOCK_SCRIPT):

Git branching model

The git branching model used for development is the one described and assisted by twgit tool: https://github.com/Twenga/twgit.