Lancelot Graphics Testing

The Lancelot system is an extension of the QTest unit test framework. Its purpose is to help catch regressions in Qt's graphics rendering. It facilitates rendering a suite of test scenes, and comparing the produced output images with the images produced from a previous rendering run. When mismatches are detected, a simple web-based report is generated. The report shows the produced "before" and "after" images, and highlights the differences.

Not all mismatches are regressions!

The Lancelot system was created on the realization that rendering output is not suited for normal unit testing. The reason is simply that a hard-coded, automated test can not decide if a rendering difference is an improvement or a regression. Hence, such a test would reject a bug-fixing code change just as a bug-introducing code change. Hence, manual inspection of the mismatches is needed to make the decision.

For illustration, here is an example report.

There are currently two Lancelot-based tests in Qt, one in qtbase and one in qtdeclarative, both detailed below.

The Lancelot system is designed around a networked Baseline server, where the produced images are stored for usage as comparison on later runs. The Baseline server also creates and serves the mismatch reports.

LanceBot: the early warning system

A bot is set up to watch for patch sets uploaded to gerrit. When a patch set touches parts of qtbase or qtdeclarative relevant for graphics rendering, it triggers a run of the Lancelot tests. The rendering output produced by the current HEAD of the target Qt branch is used as baseline. This is then compared to the output produced by Qt with the patch set applied on top of current HEAD.

If a mismatch is detected, the bot posts a comment to the gerrit change. The comment contains a link to the Lancelot report, so the commit author and reviewers can inspect the rendering differences caused by the patch set. In contrast to the Sanity Bot, LanceBot does not give any -1 votes, since it cannot judge if the mismatch is benign or not. Naturally, reviewers are not expected to approve a commit that causes unacceptable rendering regressions.

It may be noted that LanceBot does not post any +1 votes when no mismatches is detected either.Since that is the case for the vast majority of gerrit changes, such votes would be spam.

False LanceBot reports

Long experience has shown that false reports are very rare. But you may on occasion experience situations like the following:

• Your code does not touch graphics at all, but still mismatches are shown. This may happen if you do changes to e.g. the qml engine. It means your code causes general regressions, and would probably fail in the CI, and they hit on the lancelot test scenes too.
• A LanceBot comment is posted, but it lacks a report link or other relevant information. This is caused by a problem in the LanceBot system itself; please contact the maintainer (see below).

Contact

For questions, contact the maintainer on email or IRC: aavit on Freenode / eiaavits on QtC.

The Lancelot tests

The QPainter Lancelot test

Path: qtbase/tests/auto/other/lancelot

This test exercises Qt's 2D graphics engine. The test scenes are specified in Qt Painting Script (QPS), a simple special-purpose script language. QPS scripts map directly over to QPainter API calls.

The test has a number of test functions. Each function paints the suite of test scenes with different color depths/pixel format, in order to try out different painting code paths. Also, two of the functions use the OpenGL paint engine.

To render a test scene locally, use the lance tool. Example:

./qtbase/tests/manual/lance/lance -i qtbase/tests/auto/other/lancelot/scripts/aliasing.qps

The Qt Quick Lancelot test

Path: qtdeclarative/tests/manual/scenegraph_lancelot

This test exercises the Quick scene graph rendering. The test scenes are specified as normal qml scripts. For obvious reasons, the scenes must be non-interactive and static.

The test has two test functions. Both renders the suite of test scenes, but one function removes all texts from the scenes. This allows focusing on graphics changes independent of font rendering.

This test is normally run twice: once with the default (OpenGL) rendering backend, and once with QT_QUICK_BACKEND=software to test the 2D rendering backend.

To render a scene locally, simply use the qmlscene tool. Example:

./qtbase/bin/qmlscene qtdeclarative/tests/manual/scenegraph_lancelot/data/borderimages/borderimage.qml