Lancelot Graphics Testing: Difference between revisions
No edit summary |
No edit summary |
||
| Line 1: | Line 1: | ||
[[Category:Developing_Qt::QA]] | [[Category:Developing_Qt::QA]] | ||
The Lancelot system is an extension of the QTest unit test framework. Its purpose is to help catch regressions in Qt's graphics rendering. It facilitates rendering a suite of test scenes, and comparing the produced output images with the images produced from a previous rendering run. When differences are detected, a simple web-based report is generated. The report shows the produced "before" and "after" images, and highlights the differences. | The Lancelot system is an extension of the [http://doc.qt.io/qt-5/qtest.html QTest] unit test framework. Its purpose is to help catch regressions in Qt's graphics rendering. It facilitates rendering a suite of test scenes, and comparing the produced output images with the images produced from a previous rendering run. When differences are detected, a simple web-based report is generated. The report shows the produced "before" and "after" images, and highlights the differences. | ||
=== Not all differences are regressions! === | |||
The Lancelot system was created on the realization that rendering output is not suited for normal unit testing. The reason is simply that a hard-coded, automated test can not decide if a rendering difference is an improvement or a regression. Hence, such a test would reject a bug-fixing patch set just as a bug-introducing patch set. Hence, manual inspection is needed to make the decision. | |||
For illustration, here is an [https://testresults.qt.io/lancebot/tst_Lancelot/latest_report.html example report]. | |||
There are currently two Lancelot-based tests in Qt, one in qtbase and one in qtdeclarative. | |||
== The QPainter Lancelot test == | |||
Path: <code>qtbase/tests/auto/other/lancelot</code> | |||
This test exercises Qt's 2D graphics engine. The test scenes are specified in Qt Painting Script (QPS), a simple special-purpose script language. QPS scripts map directly over to QPainter API calls. | |||
The test has a number of test functions. Each function paints the suite of test scenes with different color depths/pixel format, in order to try out different painting code paths. Also, two of the functions use the OpenGL paint engine. | |||
To render a test scene locally, use the lance tool. Example: | |||
<code>./qtbase/tests/manual/lance/lance -i qtbase/tests/auto/other/lancelot/scripts/aliasing.qps</code> | |||
== The Qt Quick Lancelot test == | |||
Path: <code>qtdeclarative/tests/manual/scenegraph_lancelot</code> | |||
This test exercises the Quick scene graph rendering. The test scenes are specified as normal qml scripts. For obvious reasons, the scenes must be non-interactive and static. | |||
The test has two test functions. Both renders the suite of test scenes, but one function removes all texts from the scenes. This allows focusing on graphics changes independent of font rendering. | |||
To render a scene locally, simply use the qmlscene tool. Example: | |||
<code>./qtbase/bin/qmlscene qtdeclarative/tests/manual/scenegraph_lancelot/data/borderimages/borderimage.qml</code> | |||
Revision as of 13:55, 8 November 2017
The Lancelot system is an extension of the QTest unit test framework. Its purpose is to help catch regressions in Qt's graphics rendering. It facilitates rendering a suite of test scenes, and comparing the produced output images with the images produced from a previous rendering run. When differences are detected, a simple web-based report is generated. The report shows the produced "before" and "after" images, and highlights the differences.
Not all differences are regressions!
The Lancelot system was created on the realization that rendering output is not suited for normal unit testing. The reason is simply that a hard-coded, automated test can not decide if a rendering difference is an improvement or a regression. Hence, such a test would reject a bug-fixing patch set just as a bug-introducing patch set. Hence, manual inspection is needed to make the decision.
For illustration, here is an example report.
There are currently two Lancelot-based tests in Qt, one in qtbase and one in qtdeclarative.
The QPainter Lancelot test
Path:
qtbase/tests/auto/other/lancelot
This test exercises Qt's 2D graphics engine. The test scenes are specified in Qt Painting Script (QPS), a simple special-purpose script language. QPS scripts map directly over to QPainter API calls. The test has a number of test functions. Each function paints the suite of test scenes with different color depths/pixel format, in order to try out different painting code paths. Also, two of the functions use the OpenGL paint engine.
To render a test scene locally, use the lance tool. Example:
./qtbase/tests/manual/lance/lance -i qtbase/tests/auto/other/lancelot/scripts/aliasing.qps
The Qt Quick Lancelot test
Path:
qtdeclarative/tests/manual/scenegraph_lancelot
This test exercises the Quick scene graph rendering. The test scenes are specified as normal qml scripts. For obvious reasons, the scenes must be non-interactive and static. The test has two test functions. Both renders the suite of test scenes, but one function removes all texts from the scenes. This allows focusing on graphics changes independent of font rendering.
To render a scene locally, simply use the qmlscene tool. Example:
./qtbase/bin/qmlscene qtdeclarative/tests/manual/scenegraph_lancelot/data/borderimages/borderimage.qml