CI Autotest Metadata: Difference between revisions
Jump to navigation
Jump to search
No edit summary |
(Add to Coin category) |
||
| (7 intermediate revisions by 5 users not shown) | |||
| Line 1: | Line 1: | ||
[[Category:Developing_Qt::QA]] | [[Category:Developing_Qt::QA]] | ||
[[Category:Coin]] | |||
This page describes certain metadata which may be associated with a test, possibly affecting how the test is run by the [[CI_Overview|Qt Project CI system]]. | |||
This page describes certain metadata which may be associated with a test, possibly affecting how the test is run by the Qt Project CI system. | |||
== CONFIG values == | == CONFIG values == | ||
These may be added to CONFIG in an autotest's .pro file. Some of these are documented in | These may be added to CONFIG in an autotest's .pro file. Some of these are documented in [http://doc.qt.io/qt-5/qmake-common-projects.html#building-a-testcase the qmake manual] . The values which aren't documented are subject to change. | ||
{ | {|- class="wikitable" | ||
! Value | |||
! Supported by "make check" | |||
! Supported by CI | |||
! Description | |||
|- | |||
| testcase | |||
| yes | |||
| yes | |||
| This project is a testcase. Mandatory for anything intended to be run by "make check". | |||
|- | |||
| insignificant_test | |||
| yes | |||
| yes | |||
| This testcase's exit code should be ignored. This may be used to mark unstable tests so that they don't block CI. When used in this manner, it is best to raise a bug report for the unstable test, to increase the likelihood of the test being fixed. | |||
|- | |||
| parallel_test | |||
| no | |||
| yes | |||
| This test may be run concurrently with other tests on a single host. Autotests which don't require exclusive access to some shared resource (e.g. keyboard/mouse focus) should be marked with CONFIG+=parallel_test to increase the CI throughput. However, if parallel_test is set on tests which aren't parallel-safe, the stability of CI will suffer. | |||
|- | |||
| testcase.timeout | |||
| no | |||
| yes | |||
| Maximum permitted runtime for this test, in seconds. The test will be terminated if it does not complete within this time. | |||
|} | |||
== Custom | == Custom "check" target == | ||
It is possible to provide a custom | It is possible to provide a custom "check" target in order to integrate foreign (non-QTestLib) tests into the test procedure, with some minor caveats. | ||
* set ''check.commands'' to the desired test command(s) in the .pro file | * set ''check.commands'' to the desired test command(s) in the .pro file | ||
* do ''QMAKE_EXTRA_TARGETS*=check''. ( | * do ''QMAKE_EXTRA_TARGETS*=check''. (*= is recommended over += because there may be a default no-op "check" target.) | ||
* insignificant_test should be supported with: ''insignificant_test:check.commands = -$${check.commands}'' | |||
* the ''$(TESTRUNNER)'' make variable '''must''' be supported, in order to wrap your test command(s) with the testrunner script which implements log capturing, killing hanging tests etc. '''This is very important!''' | |||
* the ''$(TESTARGS)'' make variable may be supported, if your test command supports testlib-compatible arguments. Otherwise, it should be omitted. | |||
* if the .pro file exists solely for the "check" target, and doesn't need to compile anything, ''TEMPLATE=subdirs'' is recommended. | |||
Putting the above together, we can construct the following example, which allows "make check" to run some perl-based autotests: | |||
<code lang="make"> | |||
# "subdirs" project, nothing to compile. | |||
TEMPLATE=subdirs | |||
# set the command to be run by "make check". | |||
# We support $(TESTRUNNER) so that the CI system can inject its test runner script | |||
# to capture the logs from our test command, kill the test if it hangs, etc. | |||
# "prove —recurse <somedir>" runs all perl autotests found under a given directory tree. | |||
check.commands = $(TESTRUNNER) prove —recurse $$PWD | |||
# support insignificant_test to ignore the exit code when necessary | |||
insignificant_test:check.commands =-$${check.commands} | |||
# make sure qmake puts the "check" target into the makefile | |||
QMAKE_EXTRA_TARGETS *= check | |||
</code> | |||
Examples of foreign tests integrated into the Qt5 CI include: | |||
* '''qtbase/tests/auto/cmake''' => runs some tests via CMake's "ctest" command | |||
* '''qtqa/scripts''' => selftests of the Qt5 CI system, using a custom script | |||
Latest revision as of 13:52, 25 July 2018
This page describes certain metadata which may be associated with a test, possibly affecting how the test is run by the Qt Project CI system.
CONFIG values
These may be added to CONFIG in an autotest's .pro file. Some of these are documented in the qmake manual . The values which aren't documented are subject to change.
| Value | Supported by "make check" | Supported by CI | Description |
|---|---|---|---|
| testcase | yes | yes | This project is a testcase. Mandatory for anything intended to be run by "make check". |
| insignificant_test | yes | yes | This testcase's exit code should be ignored. This may be used to mark unstable tests so that they don't block CI. When used in this manner, it is best to raise a bug report for the unstable test, to increase the likelihood of the test being fixed. |
| parallel_test | no | yes | This test may be run concurrently with other tests on a single host. Autotests which don't require exclusive access to some shared resource (e.g. keyboard/mouse focus) should be marked with CONFIG+=parallel_test to increase the CI throughput. However, if parallel_test is set on tests which aren't parallel-safe, the stability of CI will suffer. |
| testcase.timeout | no | yes | Maximum permitted runtime for this test, in seconds. The test will be terminated if it does not complete within this time. |
Custom "check" target
It is possible to provide a custom "check" target in order to integrate foreign (non-QTestLib) tests into the test procedure, with some minor caveats.
- set check.commands to the desired test command(s) in the .pro file
- do QMAKE_EXTRA_TARGETS*=check. (*= is recommended over += because there may be a default no-op "check" target.)
- insignificant_test should be supported with: insignificant_test:check.commands = -$${check.commands}
- the $(TESTRUNNER) make variable must be supported, in order to wrap your test command(s) with the testrunner script which implements log capturing, killing hanging tests etc. This is very important!
- the $(TESTARGS) make variable may be supported, if your test command supports testlib-compatible arguments. Otherwise, it should be omitted.
- if the .pro file exists solely for the "check" target, and doesn't need to compile anything, TEMPLATE=subdirs is recommended.
Putting the above together, we can construct the following example, which allows "make check" to run some perl-based autotests:
# "subdirs" project, nothing to compile.
TEMPLATE=subdirs
# set the command to be run by "make check".
# We support $(TESTRUNNER) so that the CI system can inject its test runner script
# to capture the logs from our test command, kill the test if it hangs, etc.
# "prove —recurse <somedir>" runs all perl autotests found under a given directory tree.
check.commands = $(TESTRUNNER) prove —recurse $$PWD
# support insignificant_test to ignore the exit code when necessary
insignificant_test:check.commands =-$${check.commands}
# make sure qmake puts the "check" target into the makefile
QMAKE_EXTRA_TARGETS *= check
Examples of foreign tests integrated into the Qt5 CI include:
- qtbase/tests/auto/cmake => runs some tests via CMake's "ctest" command
- qtqa/scripts => selftests of the Qt5 CI system, using a custom script