==============================================================================
CFEngine acceptance testsuite
==============================================================================

CFEngine has an extensive testsuite which covers lot of functionality which can
be tested as a series of cf-agent runs.

You are encouraged to run this testsuite on any new software/hardware
configuration, in order to
 * verify that CFEngine functions correctly
 * provide developers with reproducible way to fix any problems encountered in
   new configurations / environment

In case you find a bug you are encouraged to create tests in format of testsuite
which demonstrate bug found, so the test could be added to this testsuite and
checked for in the future.

------------------------------------------------------------------------------
Preparing for running tests
------------------------------------------------------------------------------

* Compile CFEngine.
  - It is advised to use Tokyo Cabinet as it gives much better performance in
    test suite over Berkeley DB.

* Install fakeroot(1). If this tool is not available for your operating system,
  you may any other "fake root" environment or even sudo(1). Alternative tools
  are specified by --gainroot option of `testall' script.

------------------------------------------------------------------------------
Running testsuite
------------------------------------------------------------------------------

All tests ought only to create files and directories in /tmp, and ought not to
modify other files.

Run

  ./testall --agent=$workdir/bin/cf-agent

e.g.

  ./testall --agent=/var/cfengine/bin/cf-agent

Testing will start. For every test case the name and result (failed / passed)
will be produced. At the end testing summary will be provided.

Test runner creates the following log files:

 * test.log contains detailed information about each test case (name, standard
   output/error contents, return code, and test status).
 * summary.log contains summary information, like the one displayed during
   testsuite run.

Also a directory .succeeded will be created, containing stamps for each passed
test case, so test cases which passed before and failing in subsequent testsuite
run will be additionally marked in output as "(UNEXPECTED FAILURE)".

You might run a subset of tests by passing either filenames:

  ./testall --agent=$workdir/bin/cf-agent 01_vars/01_basic/001.cf 01_vars/01_basic/002.x.cf

or directories to 'testall':

  ./testall --agent=$workdir/bin/cf-agent 01_vars

------------------------------------------------------------------------------
Creating/editing test cases
------------------------------------------------------------------------------

Each test should be 100% standalone, and must contain at least 3 main bundles:
	init		setup, create initial and hoped-for final states
	test		the actual test code
	check		the comparison of expected and actual results

Look in default.cf for some standard check bundles (for example, to compare
files when testing file edits, or for cleaning up temporary files).

Tests should be named with only digits (e.g., "001.cf") unless they are expected
to crash (that is, if they contan syntax errors or other faults), in which case
the filename should have an 'x' suffix (e.g., "001.x.cf").

Tests which are not expected to pass yet (e.g. there is a bug in code which
prevents tests from passing) should be placed in 'staging' subdirectory in the
test directory where they belong. Such test cases will be only run if --staging
argument to ./testall is passed.

Tests which need network connectivity should be placed to 'network'
subdirectories. Those tests may be disabled by passing --no-network option to
'testall'.

NOTE: Since the class 'ok' is used in most tests, never create a persistent
class called 'ok' in any test. Persistent classes are cleaned up between test
runs, but better safe than sorry.

------------------------------------------------------------------------------
Glossary
------------------------------------------------------------------------------

For purposes of testing, here is what our terms mean:

Pass: the test did what we expected (whether that was setting a variable,
editing a file, killing or starting a process, or correctly failing to do
these actions in the light of existing conditions or attributes).  Note that
in the case of tests that end in an 'x', a Pass is generated when the test
abnormally terminates and we wanted it to do that.

FAIL: not doing what we wanted: either test finished and returned "FAIL" from
check bundle, or something went wrong - cf-agent might have dropped core,
cf-promises may have denied execution of the promises, etc.

FAILed to crash: test was expected to crash, but did not. This is another kind
of failure, split into separate kind due to low impact.

Skipped: test is skipped due to be either explicitly disabled or being
Nova-specific and being run on Community cf-agent.
