diff options
| -rw-r--r-- | nightly/README.txt | 237 |
1 files changed, 194 insertions, 43 deletions
diff --git a/nightly/README.txt b/nightly/README.txt index 6b4d79d6..464842d3 100644 --- a/nightly/README.txt +++ b/nightly/README.txt @@ -1,44 +1,195 @@ - +INTRO +----- This directory (nightly/) contains a simple, automatic build-and-test -system for Valgrind, intended to be run by cron. - -Note (importantly) it doesn't test the sources in the tree of which -this directory is a part (viz, nightly/..). Instead it checks out -a complete new tree, builds and tests that independently of the -existing tree. - -To use, choose a tag, probably a machine name, and run - - bin/nightly /path/to/valgrind/nightly/ <tag> - -and supply the following two config files: - -- conf/<tag>.conf: this is sourced by the 'nightly' script, and can define - any or all of the following environment variables: - - ABT_DETAILS: describes the machine in more detail, eg. the OS. The default - is empty. - ABT_CONFIGURE_OPTIONS: gives extra configure options. The default is empty. - ABT_EVAL: if provided, it must be the name of a shell script that executes - the shell command $1 with arguments $2 .. ${$#}. Allows to compile and - run the Valgrind regression tests on another system than the system the - 'nightly' script runs on. It is assumed that the remote system shares the - local filesystem tree through e.g. NFS. It is the responsibility of the - shell script to set the remote working directory such that it matches the - local current directory ($PWD). - ABT_RUN_REGTEST: if provided, it must be the name of an argumentless shell - function (also specified in the tag.conf file) that will be used to run - the tests. If not specified, the usual "perl tests/vg_regtest --all" - will be used. - ABT_JOBS: allows parallel builds -- it's passed as the argument to "make - -j" when building Valgrind and the tests. The default is 1. - [XXX: the .NOTPARALLEL that currently resides in Makefile.all.am foils - this!] - -- conf/<tag>.sendmail: this should be a script that sends an email to the - desired recipient (eg. the valgrind-developers list). It takes three - command line arguments. The first is the email subject line, the second - is the name of the file containing the email's body (showing the tests - that failed, and the difference between now and 24 hours ago), the third - is the name of the file containing all the diffs (which can be made into - an attachment, for example). +system for Valgrind, intended to be run nightly by cron or a similar +program. + + +BASIC OPERATIONS +---------------- +When run, the system checks out two trees: the SVN trunk from 24 hours ago +and the SVN trunk from now. ("24 hours ago" and "now" are determined when +the script starts running, so if any commits happen while the tests are +running they will not be tested.) + +If the two trees are different (i.e. there have been commits in the past 24 +hours) it builds ("make"), installs ("make install") and runs the regression +tests ("make regtest") in both, and compares the results. Note that the +"make install" isn't necessary in order to run the tests because the +regression tests use the code built (with "make") within the tree, but it's +worth doing because it tests that "make install" isn't totally broken. +After checking both trees, it emails a summary of the results to a +recipient. All this typically takes something like 30 minutes. + +If the two trees are identical, the tests are not run and no results are +emailed. This avoids spamming people with uninteresting results emails when +no commits have happened recently. + + +SETTING UP +---------- +To set up nightly testing for a machine, do the following. + +(1) Check out just this directory from the repository, eg: + + svn co svn://svn.valgrind.org/valgrind/trunk/nightly $DIR + + where $DIR is the name of the directory you want it to be in. + + Note that this doesn't check out the whole Valgrind tree, just the + directory containing the nightly testing stuff. This is possible + because the testing script doesn't check the code in the tree it belongs + to; rather it checks out new trees (within $DIR) and tests them + independently. + +(2) Choose a tag that identifies the test results. This is usually the + machine name. We'll call it $TAG in what follows. + +(3) Create a configuration file $DIR/conf/$TAG.conf. It is sourced by the + 'nightly' script, and can define any or all of the following environment + variables. (In most cases, only ABT_DETAILS is needed.) + + - ABT_DETAILS: describes the machine in more detail, eg. the OS. The + default is empty, but you should define it. An example: + + export ABT_DETAILS="Ubuntu 9.04, Intel x86-64" + + You could also use some invocation of 'uname' or something similar + to generate this string. Eg. on Ubuntu Linux this works nicely: + + export ABT_DETAILS="`cat /etc/issue.net`, `uname -m`" + + And on Mac OS X this works nicely: + + export ABT_DETAILS=`uname -mrs` + + The advantage of doing it like this is that if you update the OS on + the test machine you won't have to update ABT_DETAILS manually. + + - ABT_CONFIGURE_OPTIONS: gives extra configure options. The default is + empty. + + - ABT_EVAL: if provided, it must be the name of a shell script that + executes the shell command $1 with arguments $2 .. ${$#}. Allows to + compile and run the Valgrind regression tests on another system than + the system the 'nightly' script runs on. It is assumed that the remote + system shares the local filesystem tree through e.g. NFS. It is the + responsibility of the shell script to set the remote working directory + such that it matches the local current directory ($PWD). + + - ABT_RUN_REGTEST: if provided, it must be the name of an argumentless + shell function (also specified in the $TAG.conf file) that will be used + to run the tests. If not specified, the usual "make regtest" will be + used. + + - ABT_JOBS: allows parallel builds -- it's passed as the argument to + "make -j" when building Valgrind and the tests. The default is 1. + [XXX: the .NOTPARALLEL that currently resides in Makefile.all.am foils + this!] + + Note that the appropriate syntax to use in this file will depend on the + shell from which the $DIR/bin/nightly script is run (which in turn may + depend on what shell is used by cron or any similar program). + +(4) Create a mailer script $DIR/conf/$TAG.sendmail. It must be executable. + It's used to send email results to the desired recipient (e.g. + valgrind-developers@lists.sourceforge.net) It must handle three command + line arguments. + + - The first argument is the email subject line. It contains + $ABT_DETAILS plus some other stuff. + + - The second argument is the name of the file containing the email's + body (which shows the tests that failed, and the differences between now + and 24 hours ago). + + - The third is the name of the file containing all the diffs from + failing tests. Depending on the test results you get, you could + inline this file into the email body, or attach it, or compress and + attach it, or even omit it. The right choice depends on how many + failures you typically get -- if you get few failures, inlining the + results make them easier to read; if you get many failures, + compressing might be a good idea to minimise the size of the emails. + + The best way to do this depends on how mail is set up on your machine. + You might be able to use /usr/bin/mail, or you might need something more + elaborate like using Mutt to send mail via an external account. + +(5) To run the tests, execute: + + $DIR/bin/nightly $DIR $TAG + + You probably want to put this command into a cron file or equivalent + so it is run regularly (preferably every night). + + +OUTPUT FILES +------------ +If the tests are run, the following files are produced: + +- $DIR/old.verbose and $DIR/new.verbose contain full output of the whole + process for each of the two trees. + +- $DIR/old.short and $DIR/new.short contain summary output of the process + for each of the two trees. The diff between these two files goes in + $DIR/diff.short. + +- $DIR/final contains the overall summary, constructed from $DIR/old.short, + $DIR/new.short, $DIR/diff.short and some other bits and pieces. (The name + of this file is what's passed as the second argument to + $DIR/conf/$TAG.sendmail.) + +- $DIR/diffs holds the diffs from all the failing tests in the newer tree, + concatenated together; the diff from each failure is truncated at 100 + lines to minimise possible size blow-outs. (The name of this file is + what's passed as the third argument to $DIR/conf/$TAG.sendmail.) + +- $DIR/sendmail.log contains the output (stdout and stderr) from + $DIR/conf/$TAG.sendmail goes in $DIR/sendmail.log. + +- $DIR/valgrind/ contains the tested tree (and $DIR/valgrind/Inst/ contains + the installed code). It is used for both trees and so after the tests + have completed it will contain the newer tree. + +If the tests aren't run, the following file is produced: + +- $DIR/unchanged.log is created only if no tests were run because the two + trees were identical. It will contain a short explanatory message. + +Each time the tests are run, all files from previous runs are deleted. + + +TROUBLESHOOTING +--------------- +If something goes wrong, looking at the output files can be useful. For +example, if no email was sent but you expected one, check sendmail.log to +see if the mailer script had a problem. Or check if unchanged.log exists. + +Occasionally the SVN server isn't available when the tests runs, for either +or both trees. When this happens the email will be sent but it won't be +very informative. Usually it's just a temporary server problem and it'll +run fine the next time without you having to do anything. + +Note that the test suite is imperfect: +- There are very few machines where all tests pass; that's why the old/new + diff is produced. Some of the tests may not be as portable as intended. +- Some tests are non-deterministic, and so may pass one day and fail the + next. + +Improving the test suite to avoid these problems is a long-term goal but it +isn't easy. + + +MAINTENANCE +----------- +The scripts in the nightly/ directory occasionally get updated. If that +happens, you can just "svn update" within $DIR to get the updated versions, +which will then be used the next time the tests run. (It's possible that +the scripts will be changed in a way that requires changes to the files in +$DIR/conf/, but we try to avoid this.) + +If you want such updates to happen automatically, you could write a script +that does all the steps in SETTING UP above, and instead run that script +from cron. + + |