Lancelot Graphics Testing: Difference between revisions

From Qt Wiki
Jump to navigation Jump to search
No edit summary
(Updated to current state)
 
(5 intermediate revisions by 4 users not shown)
Line 1: Line 1:
[[Category:Developing_Qt::QA]]
[[Category:Developing_Qt::QA]]


The Lancelot system is an extension of the [http://doc.qt.io/qt-5/qtest.html QTest] unit test framework. Its purpose is to help catch regressions in Qt's graphics rendering. It facilitates rendering a suite of test scenes, and comparing the produced output images with the images produced from a previous rendering run. When mismatches are detected, a simple web-based report is generated. The report shows the produced "before" and "after" images, and highlights the differences.
The Baseline graphics testing sytem (traditionally, the "Lancelot" system) is an extension of the [http://doc.qt.io/qt-5/qtest.html QTest] unit test framework. Its purpose is to help catch regressions in Qt's graphics rendering. It facilitates rendering a suite of test scenes, and comparing the produced output images with the images produced from a previous rendering run. When mismatches are detected, a simple web-based report is generated. The report shows the produced "before" and "after" images, and highlights the differences.


====Not all mismatches are regressions!====
====Not all mismatches are regressions!====
The Lancelot system was created on the realization that rendering output is not suited for normal unit testing. The reason is simply that a hard-coded, automated test can not decide if a rendering difference is an improvement or a regression. Hence, such a test would reject a bug-fixing code change just as a bug-introducing code change. Hence, manual inspection of the mismatches is needed to make the decision.
The Baseline test system was created on the realization that rendering output is not suited for normal unit testing. The reason is simply that a hard-coded, automated test can not decide if a rendering difference is an improvement or a regression. Hence, such a test would reject a bug-fixing code change just as a bug-introducing code change. Hence, manual inspection of the mismatches is needed to make the decision.


For illustration, here is an [https://testresults.qt.io/lancebot/LanceBot6/tst_Quick3D/auto-reports/Report_Aug20-104336.html example report].
For illustration, here are the [https://testresults.qt.io/lancebot/LanceBot6/tst_QSvgRenderer/auto-reports/ latest reports for commits to the SVG module].


There are currently three Lancelot-based tests in Qt, in qtbase, qtdeclarative, and qtquikc3d; described below.
There are a number of Baseline tests in Qt. By convention, they are placed in the <code>tests/baseline</code> directory in the module's source repo.


The Lancelot system is designed around a networked ''Baseline server'', where the produced images are stored for usage as comparison on later runs. The Baseline server also creates and serves the mismatch reports.
The Baseline system is designed around a networked ''Baseline server'', where the produced images are stored for usage as comparison on later runs. The Baseline server also creates and serves the mismatch reports.


===LanceBot: the early warning system===
===LanceBot: the early warning system===
The LanceBot can run a before-after comparison on a gerrit change. For a given gerrit change (or chain), the bot will run the corresponding lancelot test two times. First, the repo at the change's parent SHA will be compiled and used to generate the lancelot baselines. Then, the SHA including the gerrit change will be compiled, and a comparison lancelot run is performed.
LanceBot is designed to run a before-and-after comparison ("A-B test") on a specific gerrit commit. When the bot is triggered (by a gerrit comment) on a certain commit (or chain of commits) to test, the bot will run the module's default Baseline test 2 times:


If a mismatch is detected, the bot posts a comment to the gerrit change. The comment contains a link to the Lancelot report, so the commit author and reviewers can inspect the rendering differences caused by the patch set. In contrast to the Sanity Bot, LanceBot does not give any -1 votes, since it cannot judge if the mismatch is benign or not. Naturally, reviewers are not expected to approve a commit that causes unacceptable rendering regressions.
# First, the repo at the change's parent SHA will be compiled and the test will be run to generate baseline images.
# Then, the SHA including the gerrit change will be compiled, and a comparison run is performed. In this way, any rendering changes introduced by the commit will be detected.


It may be noted that LanceBot does not post any +1 votes when no mismatches is detected either.Since that is the case for the vast majority of gerrit changes, such votes would be spam.
If a mismatch is detected, LanceBot posts a comment to the gerrit commit. The comment contains a link to the Baseline test report, so the commit author and reviewers can inspect the rendering differences caused by the patch set.
 
In contrast to other bots, LanceBot does not give any -1 votes, since it cannot judge if the mismatch is benign or not. Naturally, reviewers are not expected to approve a commit that causes unacceptable rendering regressions.


====False LanceBot reports====
====False LanceBot reports====
Long experience has shown that false reports are pretty rare. However you may on occasion experience situations like the following:
Long experience has shown that false reports are pretty rare. However one may on occasion experience situations like the following:


*''Your code does not touch graphics at all, but still mismatches are shown''. This may happen if you do changes to e.g. the qml engine. It means your code causes general regressions, and would probably fail in the CI, and they hit on the lancelot test scenes too.
*''Your code does not touch graphics at all, but still mismatches are shown''. This may happen if you do changes to e.g. the qml engine. It means your code causes general regressions, and would probably fail in the CI, and they hit on the lancelot test scenes too.
Line 26: Line 29:


====Contact====
====Contact====
For questions, contact the maintainer on [mailto:eirik.aavitsland@qt.io email].
For questions, contact the Oslo Graphics Team on Teams chat (Qt Company internal), or the maintainer on [mailto:eirik.aavitsland@qt.io email].


==The Lancelot tests==
==Details about some of the Baseline tests==


===The QPainter Lancelot test===
===The QPainter Baseline test===
Path: ''qtbase/tests/auto/other/lancelot''
Path: <code>qtbase/tests/baseline/painting</code>


This test exercises Qt's 2D graphics engine. The test scenes are specified in Qt Painting Script (QPS), a simple special-purpose script language. QPS scripts map directly over to QPainter API calls.
This test exercises Qt's 2D graphics engine. The test scenes are specified in Qt Painting Script (QPS), a simple special-purpose script language. QPS scripts map directly over to QPainter API calls.
Line 39: Line 42:
To render a test scene locally, use the lance tool. Example:
To render a test scene locally, use the lance tool. Example:


<code>./qtbase/tests/manual/lance/lance -i qtbase/tests/auto/other/lancelot/scripts/aliasing.qps</code>
<code>./qtbase/tests/manual/lance/lance -i qtbase/tests/baseline/painting/scripts/aliasing.qps</code>


===The Qt Quick Lancelot test===
===The Qt Quick Baseline test===
Path: ''qtdeclarative/tests/manual/scenegraph_lancelot''
Path: &lt;code&gt;qtdeclarative/tests/baseline/scenegraph&lt;/code&gt;


This test exercises the Quick scene graph rendering. The test scenes are specified as normal qml scripts. For obvious reasons, the scenes must be non-interactive and static.
This test exercises the Quick scene graph rendering. The test scenes are specified as normal qml scripts. For obvious reasons, the scenes must be non-interactive and static.


For best coverage, LanceBot will normally run this test once for each supported graphics stack on the test host machine. For Qt 5.x, that means OpenGL and the 2D rendering backend (''QT_QUICK_BACKEND=software''). For Qt 6.x, it means the supported RHI backends, plus software rendering.
For best coverage, LanceBot will normally run this test once for each supported graphics stack on the test host machine. For Qt 6.x, that means the supported RHI backends, plus software rendering.


To render a scene locally, simply use the qmlscene tool. Example:
To render a scene locally, simply use the qml tool. Example:


<code>/myqtdir/bin/qmlscene qtdeclarative/tests/manual/scenegraph_lancelot/data/borderimages/borderimage.qml</code>
<code>/myqtdir/bin/qml qtdeclarative/tests/baseline/scenegraph/data/borderimages/borderimage.qml</code>


===The Qt Quick3D Lancelot test===
One can also use the qmlscenegrabber tool that is built as part of the test, ref. below.
Path: ''qtquick3d/tests/auto/quick3d_lancelot''


This test exercises the Quick3D rendering. The test scenes are specified as normal qml scripts. For obvious reasons, the scenes must be non-interactive and static.
===The Qt Quick3D Baseline test===
Path: &lt;code&gt;qtquick3d/tests/baseline&lt;/code&gt;
 
This test exercises the Quick3D rendering. The test scenes are specified as normal qml scripts. For obvious reasons, the scenes must be non-interactive and static - i.e. animation, if any, must freeze within a few seconds so that a deterministic window grab can be taken.


LanceBot normally runs this test once with each of the RHI backends supported by the host machine.
LanceBot normally runs this test once with each of the RHI backends supported by the host machine.


To render a scene locally, simply use the qmlscene tool. Example:
To render a scene locally, use the qquick3d_qmlscenegrabber tool that is built as part of the test. Example, for only viewing:
 
<code>qquick3d_scenegrabber -viewonly qtquick3d/tests/auto/quick3d_lancelot/data/camera/DefaultCamera.qml</code>
 
To test the grabbing locally, do for example:
 
<code>qquick3d_/qmlscenegrabber -o out.png qtquick3d/tests/auto/quick3d_lancelot/data/camera/DefaultCamera.qml</code>


<code>/myqtdir/bin/qmlscene qtquick3d/tests/auto/quick3d_lancelot/data/camera/DefaultCamera.qml</code>
That will store the scene grab in ''foo.png.''

Latest revision as of 07:59, 5 June 2024


The Baseline graphics testing sytem (traditionally, the "Lancelot" system) is an extension of the QTest unit test framework. Its purpose is to help catch regressions in Qt's graphics rendering. It facilitates rendering a suite of test scenes, and comparing the produced output images with the images produced from a previous rendering run. When mismatches are detected, a simple web-based report is generated. The report shows the produced "before" and "after" images, and highlights the differences.

Not all mismatches are regressions!

The Baseline test system was created on the realization that rendering output is not suited for normal unit testing. The reason is simply that a hard-coded, automated test can not decide if a rendering difference is an improvement or a regression. Hence, such a test would reject a bug-fixing code change just as a bug-introducing code change. Hence, manual inspection of the mismatches is needed to make the decision.

For illustration, here are the latest reports for commits to the SVG module.

There are a number of Baseline tests in Qt. By convention, they are placed in the <code>tests/baseline</code> directory in the module's source repo.

The Baseline system is designed around a networked Baseline server, where the produced images are stored for usage as comparison on later runs. The Baseline server also creates and serves the mismatch reports.

LanceBot: the early warning system

LanceBot is designed to run a before-and-after comparison ("A-B test") on a specific gerrit commit. When the bot is triggered (by a gerrit comment) on a certain commit (or chain of commits) to test, the bot will run the module's default Baseline test 2 times:

  1. First, the repo at the change's parent SHA will be compiled and the test will be run to generate baseline images.
  2. Then, the SHA including the gerrit change will be compiled, and a comparison run is performed. In this way, any rendering changes introduced by the commit will be detected.

If a mismatch is detected, LanceBot posts a comment to the gerrit commit. The comment contains a link to the Baseline test report, so the commit author and reviewers can inspect the rendering differences caused by the patch set.

In contrast to other bots, LanceBot does not give any -1 votes, since it cannot judge if the mismatch is benign or not. Naturally, reviewers are not expected to approve a commit that causes unacceptable rendering regressions.

False LanceBot reports

Long experience has shown that false reports are pretty rare. However one may on occasion experience situations like the following:

  • Your code does not touch graphics at all, but still mismatches are shown. This may happen if you do changes to e.g. the qml engine. It means your code causes general regressions, and would probably fail in the CI, and they hit on the lancelot test scenes too.
  • A LanceBot comment is posted, but it lacks a report link or other relevant information. This is caused by a problem in the LanceBot system itself; please contact the maintainer (see below).

Contact

For questions, contact the Oslo Graphics Team on Teams chat (Qt Company internal), or the maintainer on email.

Details about some of the Baseline tests

The QPainter Baseline test

Path: <code>qtbase/tests/baseline/painting</code>

This test exercises Qt's 2D graphics engine. The test scenes are specified in Qt Painting Script (QPS), a simple special-purpose script language. QPS scripts map directly over to QPainter API calls.

The test has a number of test functions. Each function paints the suite of test scenes with different color depths/pixel format, in order to try out different painting code paths. Also, two of the functions use the OpenGL paint engine.

To render a test scene locally, use the lance tool. Example:

./qtbase/tests/manual/lance/lance -i qtbase/tests/baseline/painting/scripts/aliasing.qps

The Qt Quick Baseline test

Path: <code>qtdeclarative/tests/baseline/scenegraph</code>

This test exercises the Quick scene graph rendering. The test scenes are specified as normal qml scripts. For obvious reasons, the scenes must be non-interactive and static.

For best coverage, LanceBot will normally run this test once for each supported graphics stack on the test host machine. For Qt 6.x, that means the supported RHI backends, plus software rendering.

To render a scene locally, simply use the qml tool. Example:

/myqtdir/bin/qml qtdeclarative/tests/baseline/scenegraph/data/borderimages/borderimage.qml

One can also use the qmlscenegrabber tool that is built as part of the test, ref. below.

The Qt Quick3D Baseline test

Path: <code>qtquick3d/tests/baseline</code>

This test exercises the Quick3D rendering. The test scenes are specified as normal qml scripts. For obvious reasons, the scenes must be non-interactive and static - i.e. animation, if any, must freeze within a few seconds so that a deterministic window grab can be taken.

LanceBot normally runs this test once with each of the RHI backends supported by the host machine.

To render a scene locally, use the qquick3d_qmlscenegrabber tool that is built as part of the test. Example, for only viewing:

qquick3d_scenegrabber -viewonly qtquick3d/tests/auto/quick3d_lancelot/data/camera/DefaultCamera.qml

To test the grabbing locally, do for example:

qquick3d_/qmlscenegrabber -o out.png qtquick3d/tests/auto/quick3d_lancelot/data/camera/DefaultCamera.qml

That will store the scene grab in foo.png.