CI Autotest Metadata: Difference between revisions

From Qt Wiki
Jump to navigation Jump to search
No edit summary
 
No edit summary
Line 1: Line 1:
=CI Autotest Metadata=
[[Category:Developing_Qt::QA]]
 
= CI Autotest Metadata =


This page describes certain metadata which may be associated with a test, possibly affecting how the test is run by the Qt Project CI system.
This page describes certain metadata which may be associated with a test, possibly affecting how the test is run by the Qt Project CI system.


==<span class="caps">CONFIG</span> values==
== CONFIG values ==


These may be added to <span class="caps">CONFIG</span> in an autotest’s .pro file. Some of these are documented in [http://doc-snapshot.qt.io/5.0/qmake-common-projects.html#building-a-testcase the qmake manual] ''[doc-snapshot.qt.io]'' . The values which aren’t documented are subject to change.
These may be added to CONFIG in an autotest's .pro file. Some of these are documented in &quot;the qmake manual&amp;quot;:http://doc-snapshot.qt.io/5.0/qmake-common-projects.html#building-a-testcase . The values which aren't documented are subject to change.


{| class="infotable line"
{background:#009900}. |''. Value |''. Supported by &quot;make check&amp;quot; |''. Supported by CI |''. Description |<br />| testcase | yes | yes | This project is a testcase.<br />Mandatory for anything intended to be run by &quot;make check&amp;quot;. |<br />{background:#ddd}. | insignificant_test | yes | yes | This testcase's exit code should be ignored.<br />This may be used to mark unstable tests so that they don't block CI. When used in this manner, it is best to raise a bug report for the unstable test, to increase the likelihood of the test being fixed. |<br />| parallel_test | no | yes | This test may be run concurrently with other tests on a single host.<br />Autotests which don't require exclusive access to some shared resource (e.g. keyboard/mouse focus) should be marked with CONFIG+=parallel_test to increase the CI throughput.<br />However, if parallel_test is set on tests which aren't parallel-safe, the stability of CI will suffer. |<br />{background:#ddd}. | testcase.timeout | no | yes | Maximum permitted runtime for this test, in seconds. The test will be terminated if it does not complete within this time. |
|- style="background: #009900"
! Value
! Supported by “make check”
! Supported by CI
! Description
|-
| testcase
| yes
| yes
| This project is a testcase.<br /> Mandatory for anything intended to be run by “make check”.
|- style="background: #ddd"
| insignificant_test
| yes
| yes
| This testcase’s exit code should be ignored.
|-
| parallel_test
| no
| yes
| This test may be run concurrently with other tests on a single host.<br /> Autotests which don’t require exclusive access to some shared resource (e.g. keyboard/mouse focus) should be marked with <span class="caps">CONFIG</span>+=parallel_test to increase the CI throughput.<br /> However, if parallel_test is set on tests which aren’t parallel-safe, the stability of CI will suffer.
|- style="background: #ddd"
| testcase.timeout
| no
| yes
| Maximum permitted runtime for this test, in seconds. The test will be terminated if it does not complete within this time.
|}


==Custom “check” target==
== Custom &quot;check&amp;quot; target ==


It is possible to provide a custom “check” target in order to integrate foreign (non-QTestLib) tests into the test procedure, with some minor caveats.
It is possible to provide a custom &quot;check&amp;quot; target in order to integrate foreign (non-QTestLib) tests into the test procedure, with some minor caveats.


* set ''check.commands'' to the desired test command(s) in the .pro file
* set ''check.commands'' to the desired test command(s) in the .pro file
* do ''<span class="caps">QMAKE</span>_EXTRA_TARGETS*=check''. (*= is recommended over += because there may be a default no-op “check” target.)
* do ''QMAKE_EXTRA_TARGETS*=check''. ('''= is recommended over += because there may be a default no-op &quot;check&amp;quot; target.)<br />''' insignificant_test should be supported with: _insignificant_test:check.commands = <s>$${check.commands}_<br />* the ''$(TESTRUNNER)'' make variable '''must''' be supported, in order to wrap your test command(s) with the testrunner script which implements log capturing, killing hanging tests etc. '''This is very important!'''<br />* the ''$(TESTARGS)'' make variable may be supported, if your test command supports testlib-compatible arguments. Otherwise, it should be omitted.<br />* if the .pro file exists solely for the &quot;check&amp;quot; target, and doesn't need to compile anything, ''TEMPLATE=subdirs'' is recommended.
* insignificant_test should be supported with: ''insignificant_test:check.commands = -$${check.commands}''
<br />Putting the above together, we can construct the following example, which allows &quot;make check&amp;quot; to run some perl-based autotests:
* the ''$(<span class="caps">TESTRUNNER</span>)'' make variable '''must''' be supported, in order to wrap your test command(s) with the testrunner script which implements log capturing, killing hanging tests etc. '''This is very important!'''
<br /><code><br /># &quot;subdirs&amp;quot; project, nothing to compile.<br />TEMPLATE=subdirs
* the ''$(<span class="caps">TESTARGS</span>)'' make variable may be supported, if your test command supports testlib-compatible arguments. Otherwise, it should be omitted.
<br /># set the command to be run by &quot;make check&amp;quot;.<br /># We support $(TESTRUNNER) so that the CI system can inject its test runner script<br /># to capture the logs from our test command, kill the test if it hangs, etc.<br /># &quot;prove —recurse &lt;somedir&amp;gt;&quot; runs all perl autotests found under a given directory tree.<br />check.commands = $(TESTRUNNER) prove —recurse $$PWD
* if the .pro file exists solely for the “check” target, and doesn’t need to compile anything, ''<span class="caps">TEMPLATE</span>=subdirs'' is recommended.
<br /># support insignificant_test to ignore the exit code when necessary<br />insignificant_test:check.commands =</s>$${check.commands}
 
Putting the above together, we can construct the following example, which allows “make check” to run some perl-based autotests:
 
Examples of foreign tests integrated into the Qt5 CI include:
 
* '''qtbase/tests/auto/cmake''' =&gt; runs some tests via CMake’s “ctest” command
* '''qtqa/scripts''' =&gt; selftests of the Qt5 CI system, using a custom script
 
===Categories:===


* [[:Category:Developing Qt|Developing_Qt]]
# make sure qmake puts the &quot;check&amp;quot; target into the makefile<br />QMAKE_EXTRA_TARGETS '''= check<br /></code>
** [[:Category:Developing Qt::QA|QA]]
<br />Examples of foreign tests integrated into the Qt5 CI include:
<br />''' '''qtbase/tests/auto/cmake''' =&gt; runs some tests via CMake's &quot;ctest&amp;quot; command

Revision as of 11:29, 24 February 2015


CI Autotest Metadata

This page describes certain metadata which may be associated with a test, possibly affecting how the test is run by the Qt Project CI system.

CONFIG values

These may be added to CONFIG in an autotest's .pro file. Some of these are documented in "the qmake manual&quot;:http://doc-snapshot.qt.io/5.0/qmake-common-projects.html#building-a-testcase . The values which aren't documented are subject to change.

{background:#009900}. |. Value |. Supported by "make check&quot; |. Supported by CI |. Description |
| testcase | yes | yes | This project is a testcase.
Mandatory for anything intended to be run by "make check&quot;. |
{background:#ddd}. | insignificant_test | yes | yes | This testcase's exit code should be ignored.
This may be used to mark unstable tests so that they don't block CI. When used in this manner, it is best to raise a bug report for the unstable test, to increase the likelihood of the test being fixed. |
| parallel_test | no | yes | This test may be run concurrently with other tests on a single host.
Autotests which don't require exclusive access to some shared resource (e.g. keyboard/mouse focus) should be marked with CONFIG+=parallel_test to increase the CI throughput.
However, if parallel_test is set on tests which aren't parallel-safe, the stability of CI will suffer. |
{background:#ddd}. | testcase.timeout | no | yes | Maximum permitted runtime for this test, in seconds. The test will be terminated if it does not complete within this time. |

Custom "check&quot; target

It is possible to provide a custom "check&quot; target in order to integrate foreign (non-QTestLib) tests into the test procedure, with some minor caveats.

  • set check.commands to the desired test command(s) in the .pro file
  • do QMAKE_EXTRA_TARGETS*=check. (= is recommended over += because there may be a default no-op "check&quot; target.)
    insignificant_test should be supported with: _insignificant_test:check.commands = $${check.commands}_
    * the $(TESTRUNNER) make variable must be supported, in order to wrap your test command(s) with the testrunner script which implements log capturing, killing hanging tests etc. This is very important!
    * the $(TESTARGS) make variable may be supported, if your test command supports testlib-compatible arguments. Otherwise, it should be omitted.
    * if the .pro file exists solely for the "check&quot; target, and doesn't need to compile anything, TEMPLATE=subdirs is recommended.


Putting the above together, we can construct the following example, which allows "make check&quot; to run some perl-based autotests:


<br /># &quot;subdirs&amp;quot; project, nothing to compile.<br />TEMPLATE=subdirs
<br /># set the command to be run by &quot;make check&amp;quot;.<br /># We support $(TESTRUNNER) so that the CI system can inject its test runner script<br /># to capture the logs from our test command, kill the test if it hangs, etc.<br /># &quot;prove recurse &lt;somedir&amp;gt;&quot; runs all perl autotests found under a given directory tree.<br />check.commands = $(TESTRUNNER) prove recurse $$PWD
<br /># support insignificant_test to ignore the exit code when necessary<br />insignificant_test:check.commands =</s>$${check.commands}

# make sure qmake puts the &quot;check&amp;quot; target into the makefile<br />QMAKE_EXTRA_TARGETS '''= check<br />


Examples of foreign tests integrated into the Qt5 CI include:
qtbase/tests/auto/cmake => runs some tests via CMake's "ctest&quot; command