Difference between revisions of "Writing good tests"

From Qt Wiki
Jump to: navigation, search
 
Line 1: Line 1:
=Writing good tests=
+
h1. Writing good tests
  
==Aspects to consider==
+
== Aspects to consider ==
  
 
* The tests are run on CI machines potentially under heavy load.
 
* The tests are run on CI machines potentially under heavy load.
* Tests might be run in parallel (depending on the switch <span class="caps">CONFIG</span> += parallel_test.
+
* Tests might be run in parallel (depending on the switch CONFIG ''= parallel_test.<br />* Tests might be run under various desktop systems.<br />* The graphics setup is usually the GL enabling layer of some virtual machine.
* Tests might be run under various desktop systems.
+
<br />h2. Recommendations
* The graphics setup is usually the GL enabling layer of some virtual machine.
+
<br />h3. General
 
+
<br />* Do not use hard-coded timeouts to (&quot;qWait&amp;quot;:http://doc.qt.io/qt-5.0/qttestlib/qtest.html#qWait) to wait for some conditions to become true. Consider using &quot;QTRY_VERIFY&amp;quot;:http://doc.qt.io/qt-5.0/qttestlib/qtest.html#QTRY_VERIFY and &quot;QTRY_COMPARE&amp;quot;:http://doc.qt.io/qt-5.0/qttestlib/qtest.html#QTRY_COMPARE .<br />* &quot;QVERIFY2&amp;quot;:http://doc.qt.io/qt-5.0/qttestlib/qtest.html#QVERIFY2 with an error message is preferable over a plain QVERIFY in order to obtain messages when something fails:<br /><code>QVERIFY2(a &lt; 2, (QByteArray::number(a)'' QByteArrayLiteral(&quot; is not less than 2&amp;quot;))<code>
==Recommendations==
 
 
 
===General===
 
 
 
* Do not use hard-coded timeouts to ([http://doc.qt.io/qt-5.0/qttestlib/qtest.html#qWait qWait] ''[qt.io]'') to wait for some conditions to become true. Consider using [http://doc.qt.io/qt-5.0/qttestlib/qtest.html#QTRY_VERIFY <span class="caps">QTRY</span>_VERIFY] ''[qt.io]'' and [http://doc.qt.io/qt-5.0/qttestlib/qtest.html#QTRY_COMPARE <span class="caps">QTRY</span>_COMPARE] ''[qt.io]'' .
 
* [http://doc.qt.io/qt-5.0/qttestlib/qtest.html#QVERIFY2 QVERIFY2] ''[qt.io]'' with an error message is preferable over a plain <span class="caps">QVERIFY</span> in order to obtain messages when something fails:<br />
 
  
 
* Do not re-use instances of the class under test in several tests. Test instances (for example widgets) should not be member variables of the tests), but peferably be instantiated on the stack to ensure proper cleanup even if a test fails and tests do not interfere with each other
 
* Do not re-use instances of the class under test in several tests. Test instances (for example widgets) should not be member variables of the tests), but peferably be instantiated on the stack to ensure proper cleanup even if a test fails and tests do not interfere with each other
* Tests should ensure their resources are cleaned up even if a test fails (consider that a failed <span class="caps">QCOMPARE</span>/QVERIFY executes a return statement). Classes should be instantiated on the stack or use a [http://doc.qt.io/qt-5/qscopedpointer.html QScopedPointer] ''[qt.io]'' or be parented on a QObject whose deletion is guaranteed. It is recommended to check this in the slot cleanup():<br />
+
* Tests should ensure their resources are cleaned up even if a test fails (consider that a failed QCOMPARE/QVERIFY executes a return statement). Classes should be instantiated on the stack or use a &quot;QScopedPointer&amp;quot;:http://doc.qt.io/qt-5/qscopedpointer.html or be parented on a QObject whose deletion is guaranteed. It is recommended to check this in the slot cleanup():<br /></code><br />// This will be called after every test function.<br />void tst_QGraphicsProxyWidget::cleanup()<br />{<br /> QVERIFY (QApplication::topLevelWidgets().isEmpty());<br />}<br /><code>
  
===Files, I/O resources===
+
=== Files, I/O resources ===
  
 +
* Tests should not create files in their build/source directories nor in common folders like home folders. Use &quot;QTemporaryDir&amp;quot;:http://doc.qt.io/qt-5/qtemporarydir.html and &quot;QTemporaryFile&amp;quot;:http://doc.qt.io/qt-5/qtemporaryfile.html and ensure those folders and files are deleted after test execution (that is, all file handles are closed so that automatic deletion works). Always use </code>QVERIFY (temporaryDir,isValid())</code> .
 
* Tests should not create file watchers on commonly used folders like home or temporary folders (remember that tests run in parallel). Use QTemporaryDir for this as well.
 
* Tests should not create file watchers on commonly used folders like home or temporary folders (remember that tests run in parallel). Use QTemporaryDir for this as well.
  
===Widgets and Windows===
+
=== Widgets and Windows ===
  
 
* If not required for testing purposes, use at most one top level window on the screen to prevent focus fights. If several windows are required, position them explicitly beside each other
 
* If not required for testing purposes, use at most one top level window on the screen to prevent focus fights. If several windows are required, position them explicitly beside each other
* Top level windows with decoration should be at least 160×40 (else, a warning will appear on Windows 8).
+
* Preferably center top level windows within <code> QGuiApplication::primaryScreen()-&gt;availableGeometry() <code> . Most importantly, do not position windows at 0, 0 since the CI uses Ubuntu's Unity as well, which has a taskbar on the left.
* If a widget/window needs to be visible on the screen or active for a test to succeed, use:<br />
+
* Top level windows with decoration should be at least 160x40 (else, a warning will appear on Windows 8).
 +
* Beware of interference from the current cursor position. If necessary, move the cursor away from the window under test using &quot;QCursor::setPos()&quot;:http://doc.qt.io/qt-5/qcursor.html . Note that more sophisticated use of QCursor-API needs to be enclosed within </code> #ifndef QT_NO_CURSOR </code> .
 +
* If a widget/window needs to be visible on the screen or active for a test to succeed, use:<br /><code><br /> QVERIFY (QTest::qWaitForWindowExposed(&amp;amp;view));<br /> QVERIFY (QTest::qWaitForWindowActive(&amp;amp;view));<br /></code>
  
===Practical hints===
+
=== Practical hints ===
  
* On Windows, a crashing test will not display a dialog prompting you to attach a debugger (to prevent the CI from getting stuck). This can be activate by passing the command line option -nocrashhandler.
+
* On Windows, a crashing test will not display a dialog prompting you to attach a debugger (to prevent the CI from getting stuck). This can be activate by passing the command line option <s>nocrashhandler.<br />* It is possible to run single tests by passing its name on the command line (see &quot;Documentation&amp;quot;:http://doc.qt.io/qt-5/qtest-overview.html ). If you type a substring, the test will display the matches.<br />* Qt Creator contains a Perl script scripts/test2tasks.pl that converts testlib text output into a &quot;Task file&amp;quot;:http://doc.qt.io/qtcreator-2.6/creator-task-lists.html that is shown the build issues pane and can be used to quickly navigate to test failures .<br />* An overview of flaky/failing tests can be found &quot;here&amp;quot;:http://testresults.qt</s> project.org/qtmetrics/autoteststatus/index.html .
* It is possible to run single tests by passing its name on the command line (see [http://doc.qt.io/qt-5/qtest-overview.html Documentation] ''[qt.io]'' ). If you type a substring, the test will display the matches.
 
* Qt Creator contains a Perl script scripts/test2tasks.pl that converts testlib text output into a [http://doc.qt.io/qtcreator-2.6/creator-task-lists.html Task file] ''[qt.io]'' that is shown the build issues pane and can be used to quickly navigate to test failures .
 
* An overview of flaky/failing tests can be found [http://testresults.qt here] ''[testresults.qt]''- project.org/qtmetrics/autoteststatus/index.html .
 
  
===Hints for analyzing test flakyness (<span class="caps">GUI</span> tests)===
+
=== Hints for analyzing test flakyness (GUI tests) ===
  
 
* Does the test create unrelated windows that overlap and interfere? For example, most existing widget tests still use a member variable test widget shared between the tests. If a single test then instantiates another widget, this can lead to focus issues. As stated above the member variable should be replaced by per-test widget instances. In some cases, windows created by skipped/failed tests leak.
 
* Does the test create unrelated windows that overlap and interfere? For example, most existing widget tests still use a member variable test widget shared between the tests. If a single test then instantiates another widget, this can lead to focus issues. As stated above the member variable should be replaced by per-test widget instances. In some cases, windows created by skipped/failed tests leak.
 
* Does the test create windows at random positions (notably on X11)? Such windows might end up in the taskbar area or interfere with notification windows of the OS. As stated above, Windows should be centered.
 
* Does the test create windows at random positions (notably on X11)? Such windows might end up in the taskbar area or interfere with notification windows of the OS. As stated above, Windows should be centered.
 
* Is the test being influenced by the mouse cursor position (this is particularly an issue on Mac)? If so, the cursor should be moved to a well-defined position.
 
* Is the test being influenced by the mouse cursor position (this is particularly an issue on Mac)? If so, the cursor should be moved to a well-defined position.

Revision as of 11:05, 24 February 2015

h1. Writing good tests

Aspects to consider

  • The tests are run on CI machines potentially under heavy load.
  • Tests might be run in parallel (depending on the switch CONFIG = parallel_test.
    * Tests might be run under various desktop systems.
    * The graphics setup is usually the GL enabling layer of some virtual machine.


h2. Recommendations
h3. General
* Do not use hard-coded timeouts to ("qWait&quot;:http://doc.qt.io/qt-5.0/qttestlib/qtest.html#qWait) to wait for some conditions to become true. Consider using "QTRY_VERIFY&quot;:http://doc.qt.io/qt-5.0/qttestlib/qtest.html#QTRY_VERIFY and "QTRY_COMPARE&quot;:http://doc.qt.io/qt-5.0/qttestlib/qtest.html#QTRY_COMPARE .
* "QVERIFY2&quot;:http://doc.qt.io/qt-5.0/qttestlib/qtest.html#QVERIFY2 with an error message is preferable over a plain QVERIFY in order to obtain messages when something fails:
QVERIFY2(a < 2, (QByteArray::number(a) QByteArrayLiteral(" is not less than 2&quot;))

  • Do not re-use instances of the class under test in several tests. Test instances (for example widgets) should not be member variables of the tests), but peferably be instantiated on the stack to ensure proper cleanup even if a test fails and tests do not interfere with each other
  • Tests should ensure their resources are cleaned up even if a test fails (consider that a failed QCOMPARE/QVERIFY executes a return statement). Classes should be instantiated on the stack or use a "QScopedPointer&quot;:http://doc.qt.io/qt-5/qscopedpointer.html or be parented on a QObject whose deletion is guaranteed. It is recommended to check this in the slot cleanup():

    // This will be called after every test function.
    void tst_QGraphicsProxyWidget::cleanup()
    {
    QVERIFY (QApplication::topLevelWidgets().isEmpty());
    }

Files, I/O resources

  • Tests should not create files in their build/source directories nor in common folders like home folders. Use "QTemporaryDir&quot;:http://doc.qt.io/qt-5/qtemporarydir.html and "QTemporaryFile&quot;:http://doc.qt.io/qt-5/qtemporaryfile.html and ensure those folders and files are deleted after test execution (that is, all file handles are closed so that automatic deletion works). Always use QVERIFY (temporaryDir,isValid()) .
  • Tests should not create file watchers on commonly used folders like home or temporary folders (remember that tests run in parallel). Use QTemporaryDir for this as well.

Widgets and Windows

  • If not required for testing purposes, use at most one top level window on the screen to prevent focus fights. If several windows are required, position them explicitly beside each other
  • Preferably center top level windows within QGuiApplication::primaryScreen()->availableGeometry() . Most importantly, do not position windows at 0, 0 since the CI uses Ubuntu's Unity as well, which has a taskbar on the left.
  • Top level windows with decoration should be at least 160x40 (else, a warning will appear on Windows 8).
  • Beware of interference from the current cursor position. If necessary, move the cursor away from the window under test using "QCursor::setPos()":http://doc.qt.io/qt-5/qcursor.html . Note that more sophisticated use of QCursor-API needs to be enclosed within #ifndef QT_NO_CURSOR .
  • If a widget/window needs to be visible on the screen or active for a test to succeed, use:

    QVERIFY (QTest::qWaitForWindowExposed(&amp;view));
    QVERIFY (QTest::qWaitForWindowActive(&amp;view));

Practical hints

  • On Windows, a crashing test will not display a dialog prompting you to attach a debugger (to prevent the CI from getting stuck). This can be activate by passing the command line option nocrashhandler.
    * It is possible to run single tests by passing its name on the command line (see "Documentation&quot;:http://doc.qt.io/qt-5/qtest-overview.html ). If you type a substring, the test will display the matches.
    * Qt Creator contains a Perl script scripts/test2tasks.pl that converts testlib text output into a "Task file&quot;:http://doc.qt.io/qtcreator-2.6/creator-task-lists.html that is shown the build issues pane and can be used to quickly navigate to test failures .
    * An overview of flaky/failing tests can be found "here&quot;:http://testresults.qt
    project.org/qtmetrics/autoteststatus/index.html .

Hints for analyzing test flakyness (GUI tests)

  • Does the test create unrelated windows that overlap and interfere? For example, most existing widget tests still use a member variable test widget shared between the tests. If a single test then instantiates another widget, this can lead to focus issues. As stated above the member variable should be replaced by per-test widget instances. In some cases, windows created by skipped/failed tests leak.
  • Does the test create windows at random positions (notably on X11)? Such windows might end up in the taskbar area or interfere with notification windows of the OS. As stated above, Windows should be centered.
  • Is the test being influenced by the mouse cursor position (this is particularly an issue on Mac)? If so, the cursor should be moved to a well-defined position.