Lancelot Graphics Testing

The Lancelot system is an extension of the QTest unit test framework. Its purpose is to help catch regressions in Qt's graphics rendering. It facilitates rendering a suite of test scenes, and comparing the produced output images with the images produced from a previous rendering run. When differences are detected, a simple web-based report is generated. The report shows the produced "before" and "after" images, and highlights the differences.

Not all differences are regressions!

The Lancelot system was created on the realization that rendering output is not suited for normal unit testing. The reason is simply that a hard-coded, automated test can not decide if a rendering difference is an improvement or a regression. Hence, such a test would reject a bug-fixing patch set just as a bug-introducing patch set. Hence, manual inspection is needed to make the decision.

For illustration, here is an example report.

There are currently two Lancelot-based tests in Qt, one in qtbase and one in qtdeclarative.

The QPainter Lancelot test

Path: qtbase/tests/auto/other/lancelot

This test exercises Qt's 2D graphics engine. The test scenes are specified in Qt Painting Script (QPS), a simple special-purpose script language. QPS scripts map directly over to QPainter API calls. The test has a number of test functions. Each function paints the suite of test scenes with different color depths/pixel format, in order to try out different painting code paths. Also, two of the functions use the OpenGL paint engine.

To render a test scene locally, use the lance tool. Example:

./qtbase/tests/manual/lance/lance -i qtbase/tests/auto/other/lancelot/scripts/aliasing.qps

The Qt Quick Lancelot test

Path: qtdeclarative/tests/manual/scenegraph_lancelot

This test exercises the Quick scene graph rendering. The test scenes are specified as normal qml scripts. For obvious reasons, the scenes must be non-interactive and static. The test has two test functions. Both renders the suite of test scenes, but one function removes all texts from the scenes. This allows focusing on graphics changes independent of font rendering.

To render a scene locally, simply use the qmlscene tool. Example:

./qtbase/bin/qmlscene qtdeclarative/tests/manual/scenegraph_lancelot/data/borderimages/borderimage.qml