CI Autotest Metadata: Difference between revisions
No edit summary |
No edit summary |
||
| Line 9: | Line 9: | ||
These may be added to CONFIG in an autotest's .pro file. Some of these are documented in "the qmake manual":http://doc-snapshot.qt.io/5.0/qmake-common-projects.html#building-a-testcase . The values which aren't documented are subject to change. | These may be added to CONFIG in an autotest's .pro file. Some of these are documented in "the qmake manual":http://doc-snapshot.qt.io/5.0/qmake-common-projects.html#building-a-testcase . The values which aren't documented are subject to change. | ||
{background:#009900}. |''. Value |''. Supported by "make check" |''. Supported by CI |''. Description | | {background:#009900}. |''. Value |''. Supported by "make check" |''. Supported by CI |''. Description | | ||
| testcase | yes | yes | This project is a testcase. | |||
Mandatory for anything intended to be run by "make check". | | |||
{background:#ddd}. | insignificant_test | yes | yes | This testcase's exit code should be ignored. | |||
This may be used to mark unstable tests so that they don't block CI. When used in this manner, it is best to raise a bug report for the unstable test, to increase the likelihood of the test being fixed. | | |||
| parallel_test | no | yes | This test may be run concurrently with other tests on a single host. | |||
Autotests which don't require exclusive access to some shared resource (e.g. keyboard/mouse focus) should be marked with CONFIG+=parallel_test to increase the CI throughput. | |||
However, if parallel_test is set on tests which aren't parallel-safe, the stability of CI will suffer. | | |||
{background:#ddd}. | testcase.timeout | no | yes | Maximum permitted runtime for this test, in seconds. The test will be terminated if it does not complete within this time. | | |||
== Custom "check" target == | == Custom "check" target == | ||
| Line 16: | Line 24: | ||
* set ''check.commands'' to the desired test command(s) in the .pro file | * set ''check.commands'' to the desired test command(s) in the .pro file | ||
* do ''QMAKE_EXTRA_TARGETS*=check''. ('''= is recommended over += because there may be a default no-op "check" target.) | * do ''QMAKE_EXTRA_TARGETS*=check''. ('''= is recommended over += because there may be a default no-op "check" target.) | ||
''' insignificant_test should be supported with: _insignificant_test:check.commands = -$${check.commands}_ | |||
* the ''$(TESTRUNNER)'' make variable '''must''' be supported, in order to wrap your test command(s) with the testrunner script which implements log capturing, killing hanging tests etc. '''This is very important!''' | |||
* the ''$(TESTARGS)'' make variable may be supported, if your test command supports testlib-compatible arguments. Otherwise, it should be omitted. | |||
* if the .pro file exists solely for the "check" target, and doesn't need to compile anything, ''TEMPLATE=subdirs'' is recommended. | |||
# make sure qmake puts the "check" target into the makefile | Putting the above together, we can construct the following example, which allows "make check" to run some perl-based autotests: | ||
<code> | |||
# "subdirs" project, nothing to compile. | |||
TEMPLATE=subdirs | |||
# set the command to be run by "make check". | |||
# We support $(TESTRUNNER) so that the CI system can inject its test runner script | |||
# to capture the logs from our test command, kill the test if it hangs, etc. | |||
# "prove —recurse <somedir>" runs all perl autotests found under a given directory tree. | |||
check.commands = $(TESTRUNNER) prove —recurse $$PWD | |||
# support insignificant_test to ignore the exit code when necessary | |||
insignificant_test:check.commands =-$${check.commands} | |||
# make sure qmake puts the "check" target into the makefile | |||
QMAKE_EXTRA_TARGETS '''= check | |||
</code> | |||
Examples of foreign tests integrated into the Qt5 CI include: | |||
''' '''qtbase/tests/auto/cmake''' => runs some tests via CMake's "ctest" command | |||
Revision as of 11:47, 25 February 2015
CI Autotest Metadata
This page describes certain metadata which may be associated with a test, possibly affecting how the test is run by the Qt Project CI system.
CONFIG values
These may be added to CONFIG in an autotest's .pro file. Some of these are documented in "the qmake manual":http://doc-snapshot.qt.io/5.0/qmake-common-projects.html#building-a-testcase . The values which aren't documented are subject to change.
{background:#009900}. |. Value |. Supported by "make check" |. Supported by CI |. Description | | testcase | yes | yes | This project is a testcase. Mandatory for anything intended to be run by "make check". | {background:#ddd}. | insignificant_test | yes | yes | This testcase's exit code should be ignored. This may be used to mark unstable tests so that they don't block CI. When used in this manner, it is best to raise a bug report for the unstable test, to increase the likelihood of the test being fixed. | | parallel_test | no | yes | This test may be run concurrently with other tests on a single host. Autotests which don't require exclusive access to some shared resource (e.g. keyboard/mouse focus) should be marked with CONFIG+=parallel_test to increase the CI throughput. However, if parallel_test is set on tests which aren't parallel-safe, the stability of CI will suffer. | {background:#ddd}. | testcase.timeout | no | yes | Maximum permitted runtime for this test, in seconds. The test will be terminated if it does not complete within this time. |
Custom "check" target
It is possible to provide a custom "check" target in order to integrate foreign (non-QTestLib) tests into the test procedure, with some minor caveats.
- set check.commands to the desired test command(s) in the .pro file
- do QMAKE_EXTRA_TARGETS*=check. (= is recommended over += because there may be a default no-op "check" target.)
insignificant_test should be supported with: _insignificant_test:check.commands = -$${check.commands}_
- the $(TESTRUNNER) make variable must be supported, in order to wrap your test command(s) with the testrunner script which implements log capturing, killing hanging tests etc. This is very important!
- the $(TESTARGS) make variable may be supported, if your test command supports testlib-compatible arguments. Otherwise, it should be omitted.
- if the .pro file exists solely for the "check" target, and doesn't need to compile anything, TEMPLATE=subdirs is recommended.
Putting the above together, we can construct the following example, which allows "make check" to run some perl-based autotests:
# "subdirs" project, nothing to compile.
TEMPLATE=subdirs
# set the command to be run by "make check".
# We support $(TESTRUNNER) so that the CI system can inject its test runner script
# to capture the logs from our test command, kill the test if it hangs, etc.
# "prove —recurse <somedir>" runs all perl autotests found under a given directory tree.
check.commands = $(TESTRUNNER) prove —recurse $$PWD
# support insignificant_test to ignore the exit code when necessary
insignificant_test:check.commands =-$${check.commands}
# make sure qmake puts the "check" target into the makefile
QMAKE_EXTRA_TARGETS '''= check
Examples of foreign tests integrated into the Qt5 CI include:
qtbase/tests/auto/cmake => runs some tests via CMake's "ctest" command