CI Autotest Metadata: Difference between revisions

From Qt Wiki
Jump to navigation Jump to search
(Convert ExpressionEngine links)
(Add to Coin category)
 
(3 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{{Cleanup | reason=Auto-imported from ExpressionEngine.}}
[[Category:Developing_Qt::QA]]
[[Category:Developing_Qt::QA]]
[[Category:Coin]]


= CI Autotest Metadata =
This page describes certain metadata which may be associated with a test, possibly affecting how the test is run by the [[CI_Overview|Qt Project CI system]].
 
This page describes certain metadata which may be associated with a test, possibly affecting how the test is run by the Qt Project CI system.


== CONFIG values ==
== CONFIG values ==


These may be added to CONFIG in an autotest's .pro file. Some of these are documented in [http://doc-snapshot.qt.io/5.0/qmake-common-projects.html#building-a-testcase the qmake manual] . The values which aren't documented are subject to change.
These may be added to CONFIG in an autotest's .pro file. Some of these are documented in [http://doc.qt.io/qt-5/qmake-common-projects.html#building-a-testcase the qmake manual] . The values which aren't documented are subject to change.


{background:#009900}. |''. Value |''. Supported by "make check" |''. Supported by CI |''. Description |
{|- class="wikitable"
| testcase | yes | yes | This project is a testcase.
! Value  
Mandatory for anything intended to be run by "make check". |
! Supported by "make check"
{background:#ddd}. | insignificant_test | yes | yes | This testcase's exit code should be ignored.
! Supported by CI  
This may be used to mark unstable tests so that they don't block CI. When used in this manner, it is best to raise a bug report for the unstable test, to increase the likelihood of the test being fixed. |
! Description  
| parallel_test | no | yes | This test may be run concurrently with other tests on a single host.
|-
Autotests which don't require exclusive access to some shared resource (e.g. keyboard/mouse focus) should be marked with CONFIG+=parallel_test to increase the CI throughput.
| testcase  
However, if parallel_test is set on tests which aren't parallel-safe, the stability of CI will suffer. |
| yes  
{background:#ddd}. | testcase.timeout | no | yes | Maximum permitted runtime for this test, in seconds. The test will be terminated if it does not complete within this time. |
| yes  
| This project is a testcase. Mandatory for anything intended to be run by "make check".  
|-
| insignificant_test  
| yes  
| yes  
| This testcase's exit code should be ignored. This may be used to mark unstable tests so that they don't block CI. When used in this manner, it is best to raise a bug report for the unstable test, to increase the likelihood of the test being fixed.  
|-
| parallel_test  
| no  
| yes  
| This test may be run concurrently with other tests on a single host. Autotests which don't require exclusive access to some shared resource (e.g. keyboard/mouse focus) should be marked with CONFIG+=parallel_test to increase the CI throughput. However, if parallel_test is set on tests which aren't parallel-safe, the stability of CI will suffer.  
|-
| testcase.timeout  
| no  
| yes  
| Maximum permitted runtime for this test, in seconds. The test will be terminated if it does not complete within this time.  
|}


== Custom "check" target ==
== Custom "check" target ==
Line 26: Line 40:


* set ''check.commands'' to the desired test command(s) in the .pro file
* set ''check.commands'' to the desired test command(s) in the .pro file
* do ''QMAKE_EXTRA_TARGETS*=check''. ('''= is recommended over += because there may be a default no-op "check" target.)
* do ''QMAKE_EXTRA_TARGETS*=check''. (*= is recommended over += because there may be a default no-op "check" target.)
''' insignificant_test should be supported with: _insignificant_test:check.commands = -$${check.commands}_
* insignificant_test should be supported with: ''insignificant_test:check.commands = -$${check.commands}''
* the ''$(TESTRUNNER)'' make variable '''must''' be supported, in order to wrap your test command(s) with the testrunner script which implements log capturing, killing hanging tests etc. '''This is very important!'''
* the ''$(TESTRUNNER)'' make variable '''must''' be supported, in order to wrap your test command(s) with the testrunner script which implements log capturing, killing hanging tests etc. '''This is very important!'''
* the ''$(TESTARGS)'' make variable may be supported, if your test command supports testlib-compatible arguments. Otherwise, it should be omitted.
* the ''$(TESTARGS)'' make variable may be supported, if your test command supports testlib-compatible arguments. Otherwise, it should be omitted.
Line 34: Line 48:
Putting the above together, we can construct the following example, which allows "make check" to run some perl-based autotests:
Putting the above together, we can construct the following example, which allows "make check" to run some perl-based autotests:


<code>
<code lang="make">
# "subdirs" project, nothing to compile.
# "subdirs" project, nothing to compile.
TEMPLATE=subdirs
TEMPLATE=subdirs
Line 48: Line 62:


# make sure qmake puts the "check" target into the makefile
# make sure qmake puts the "check" target into the makefile
QMAKE_EXTRA_TARGETS '''= check
QMAKE_EXTRA_TARGETS *= check
</code>
</code>


Examples of foreign tests integrated into the Qt5 CI include:
Examples of foreign tests integrated into the Qt5 CI include:


''' '''qtbase/tests/auto/cmake''' => runs some tests via CMake's "ctest" command
* '''qtbase/tests/auto/cmake''' => runs some tests via CMake's "ctest" command
* '''qtqa/scripts''' => selftests of the Qt5 CI system, using a custom script

Latest revision as of 13:52, 25 July 2018


This page describes certain metadata which may be associated with a test, possibly affecting how the test is run by the Qt Project CI system.

CONFIG values

These may be added to CONFIG in an autotest's .pro file. Some of these are documented in the qmake manual . The values which aren't documented are subject to change.

Value Supported by "make check" Supported by CI Description
testcase yes yes This project is a testcase. Mandatory for anything intended to be run by "make check".
insignificant_test yes yes This testcase's exit code should be ignored. This may be used to mark unstable tests so that they don't block CI. When used in this manner, it is best to raise a bug report for the unstable test, to increase the likelihood of the test being fixed.
parallel_test no yes This test may be run concurrently with other tests on a single host. Autotests which don't require exclusive access to some shared resource (e.g. keyboard/mouse focus) should be marked with CONFIG+=parallel_test to increase the CI throughput. However, if parallel_test is set on tests which aren't parallel-safe, the stability of CI will suffer.
testcase.timeout no yes Maximum permitted runtime for this test, in seconds. The test will be terminated if it does not complete within this time.

Custom "check" target

It is possible to provide a custom "check" target in order to integrate foreign (non-QTestLib) tests into the test procedure, with some minor caveats.

  • set check.commands to the desired test command(s) in the .pro file
  • do QMAKE_EXTRA_TARGETS*=check. (*= is recommended over += because there may be a default no-op "check" target.)
  • insignificant_test should be supported with: insignificant_test:check.commands = -$${check.commands}
  • the $(TESTRUNNER) make variable must be supported, in order to wrap your test command(s) with the testrunner script which implements log capturing, killing hanging tests etc. This is very important!
  • the $(TESTARGS) make variable may be supported, if your test command supports testlib-compatible arguments. Otherwise, it should be omitted.
  • if the .pro file exists solely for the "check" target, and doesn't need to compile anything, TEMPLATE=subdirs is recommended.

Putting the above together, we can construct the following example, which allows "make check" to run some perl-based autotests:

# "subdirs" project, nothing to compile.
TEMPLATE=subdirs

# set the command to be run by "make check".
# We support $(TESTRUNNER) so that the CI system can inject its test runner script
# to capture the logs from our test command, kill the test if it hangs, etc.
# "prove —recurse <somedir>" runs all perl autotests found under a given directory tree.
check.commands = $(TESTRUNNER) prove —recurse $$PWD

# support insignificant_test to ignore the exit code when necessary
insignificant_test:check.commands =-$${check.commands}

# make sure qmake puts the "check" target into the makefile
QMAKE_EXTRA_TARGETS *= check

Examples of foreign tests integrated into the Qt5 CI include:

  • qtbase/tests/auto/cmake => runs some tests via CMake's "ctest" command
  • qtqa/scripts => selftests of the Qt5 CI system, using a custom script