CI Autotest Metadata: Difference between revisions

Latest revision as of 13:52, 25 July 2018

This page describes certain metadata which may be associated with a test, possibly affecting how the test is run by the Qt Project CI system.

CONFIG values

These may be added to CONFIG in an autotest's .pro file. Some of these are documented in the qmake manual . The values which aren't documented are subject to change.

Value	Supported by "make check"	Supported by CI	Description
testcase	yes	yes	This project is a testcase. Mandatory for anything intended to be run by "make check".
insignificant_test	yes	yes	This testcase's exit code should be ignored. This may be used to mark unstable tests so that they don't block CI. When used in this manner, it is best to raise a bug report for the unstable test, to increase the likelihood of the test being fixed.
parallel_test	no	yes	This test may be run concurrently with other tests on a single host. Autotests which don't require exclusive access to some shared resource (e.g. keyboard/mouse focus) should be marked with CONFIG+=parallel_test to increase the CI throughput. However, if parallel_test is set on tests which aren't parallel-safe, the stability of CI will suffer.
testcase.timeout	no	yes	Maximum permitted runtime for this test, in seconds. The test will be terminated if it does not complete within this time.

Custom "check" target

It is possible to provide a custom "check" target in order to integrate foreign (non-QTestLib) tests into the test procedure, with some minor caveats.

set check.commands to the desired test command(s) in the .pro file
do QMAKE_EXTRA_TARGETS*=check. (*= is recommended over += because there may be a default no-op "check" target.)
insignificant_test should be supported with: insignificant_test:check.commands = -$${check.commands}
the $(TESTRUNNER) make variable must be supported, in order to wrap your test command(s) with the testrunner script which implements log capturing, killing hanging tests etc. This is very important!
the $(TESTARGS) make variable may be supported, if your test command supports testlib-compatible arguments. Otherwise, it should be omitted.
if the .pro file exists solely for the "check" target, and doesn't need to compile anything, TEMPLATE=subdirs is recommended.

Putting the above together, we can construct the following example, which allows "make check" to run some perl-based autotests:

# "subdirs" project, nothing to compile.
TEMPLATE=subdirs

# set the command to be run by "make check".
# We support $(TESTRUNNER) so that the CI system can inject its test runner script
# to capture the logs from our test command, kill the test if it hangs, etc.
# "prove —recurse <somedir>" runs all perl autotests found under a given directory tree.
check.commands = $(TESTRUNNER) prove —recurse $$PWD

# support insignificant_test to ignore the exit code when necessary
insignificant_test:check.commands =-$${check.commands}

# make sure qmake puts the "check" target into the makefile
QMAKE_EXTRA_TARGETS *= check

Examples of foreign tests integrated into the Qt5 CI include:

qtbase/tests/auto/cmake => runs some tests via CMake's "ctest" command
qtqa/scripts => selftests of the Qt5 CI system, using a custom script

@@ Line 1: / Line 1: @@
-=CI Autotest Metadata=
+[[Category:Developing_Qt::QA]]
+[[Category:Coin]]
-This page describes certain metadata which may be associated with a test, possibly affecting how the test is run by the Qt Project CI system.
+This page describes certain metadata which may be associated with a test, possibly affecting how the test is run by the [[CI_Overview|Qt Project CI system]].
-==<span class="caps">CONFIG</span> values==
+== CONFIG values ==
-These may be added to <span class="caps">CONFIG</span> in an autotest’s .pro file. Some of these are documented in [http://doc-snapshot.qt.io/5.0/qmake-common-projects.html#building-a-testcase the qmake manual] ''[doc-snapshot.qt.io]'' . The values which aren’t documented are subject to change.
+These may be added to CONFIG in an autotest's .pro file. Some of these are documented in [http://doc.qt.io/qt-5/qmake-common-projects.html#building-a-testcase the qmake manual] . The values which aren't documented are subject to change.
-{| class="infotable line"
+{|- class="wikitable"
-|- style="background: #009900"
+! Value
-! Value
+! Supported by "make check"
-! Supported by “make check”
+! Supported by CI
-! Supported by CI
+! Description
-! Description
 |-
 | testcase
 | yes
 | yes
-| This project is a testcase.<br /> Mandatory for anything intended to be run by “make check”.
+| This project is a testcase. Mandatory for anything intended to be run by "make check".
-|- style="background: #ddd"
-| insignificant_test
-| yes
-| yes
-| This testcase’s exit code should be ignored.
 |-
-| parallel_test
+| insignificant_test
-| no
+| yes
 | yes
-| This test may be run concurrently with other tests on a single host.<br /> Autotests which don’t require exclusive access to some shared resource (e.g. keyboard/mouse focus) should be marked with <span class="caps">CONFIG</span>+=parallel_test to increase the CI throughput.<br /> However, if parallel_test is set on tests which aren’t parallel-safe, the stability of CI will suffer.
+| This testcase's exit code should be ignored. This may be used to mark unstable tests so that they don't block CI. When used in this manner, it is best to raise a bug report for the unstable test, to increase the likelihood of the test being fixed.
-|- style="background: #ddd"
+|-
-| testcase.timeout
+| parallel_test
 | no
 | yes
-| Maximum permitted runtime for this test, in seconds. The test will be terminated if it does not complete within this time.
+| This test may be run concurrently with other tests on a single host. Autotests which don't require exclusive access to some shared resource (e.g. keyboard/mouse focus) should be marked with CONFIG+=parallel_test to increase the CI throughput. However, if parallel_test is set on tests which aren't parallel-safe, the stability of CI will suffer.
+|-
+| testcase.timeout
+| no
+| yes
+| Maximum permitted runtime for this test, in seconds. The test will be terminated if it does not complete within this time.
 |}
-==Custom “check” target==
+== Custom "check" target ==
-It is possible to provide a custom “check” target in order to integrate foreign (non-QTestLib) tests into the test procedure, with some minor caveats.
+It is possible to provide a custom "check" target in order to integrate foreign (non-QTestLib) tests into the test procedure, with some minor caveats.
 * set ''check.commands'' to the desired test command(s) in the .pro file
-* do ''<span class="caps">QMAKE</span>_EXTRA_TARGETS*=check''. (*= is recommended over += because there may be a default no-op “check” target.)
+* do ''QMAKE_EXTRA_TARGETS*=check''. (*= is recommended over += because there may be a default no-op "check" target.)
 * insignificant_test should be supported with: ''insignificant_test:check.commands = -$${check.commands}''
-* the ''$(<span class="caps">TESTRUNNER</span>)'' make variable '''must''' be supported, in order to wrap your test command(s) with the testrunner script which implements log capturing, killing hanging tests etc. '''This is very important!'''
+* the ''$(TESTRUNNER)'' make variable '''must''' be supported, in order to wrap your test command(s) with the testrunner script which implements log capturing, killing hanging tests etc. '''This is very important!'''
-* the ''$(<span class="caps">TESTARGS</span>)'' make variable may be supported, if your test command supports testlib-compatible arguments. Otherwise, it should be omitted.
+* the ''$(TESTARGS)'' make variable may be supported, if your test command supports testlib-compatible arguments. Otherwise, it should be omitted.
-* if the .pro file exists solely for the “check” target, and doesn’t need to compile anything, ''<span class="caps">TEMPLATE</span>=subdirs'' is recommended.
+* if the .pro file exists solely for the "check" target, and doesn't need to compile anything, ''TEMPLATE=subdirs'' is recommended.
+Putting the above together, we can construct the following example, which allows "make check" to run some perl-based autotests:
+<code lang="make">
+# "subdirs" project, nothing to compile.
+TEMPLATE=subdirs
-Putting the above together, we can construct the following example, which allows “make check” to run some perl-based autotests:
+# set the command to be run by "make check".
+# We support $(TESTRUNNER) so that the CI system can inject its test runner script
+# to capture the logs from our test command, kill the test if it hangs, etc.
+# "prove —recurse <somedir>" runs all perl autotests found under a given directory tree.
+check.commands = $(TESTRUNNER) prove —recurse $$PWD
-Examples of foreign tests integrated into the Qt5 CI include:
+# support insignificant_test to ignore the exit code when necessary
+insignificant_test:check.commands =-$${check.commands}
-* '''qtbase/tests/auto/cmake''' =&gt; runs some tests via CMake’s “ctest” command
+# make sure qmake puts the "check" target into the makefile
-* '''qtqa/scripts''' =&gt; selftests of the Qt5 CI system, using a custom script
+QMAKE_EXTRA_TARGETS *= check
+</code>
-===Categories:===
+Examples of foreign tests integrated into the Qt5 CI include:
-* [[:Category:Developing Qt|Developing_Qt]]
+* '''qtbase/tests/auto/cmake''' => runs some tests via CMake's "ctest" command
-** [[:Category:Developing Qt::QA|QA]]
+* '''qtqa/scripts''' => selftests of the Qt5 CI system, using a custom script

CI Autotest Metadata: Difference between revisions

Latest revision as of 13:52, 25 July 2018

CONFIG values

Custom "check" target

Navigation menu

Search