Most common sources of flakiness on openSuSE

Flaky tests are tests that are passing but occasionally fail, usually for unknown reasons. It is important to note that these tests have issues in the testing code, not in the functionality being tested.

To ensure that a test is not flaky, it should be ran several thousand times. In some rare cases, a test can run successfully a hundred times and fail on the hundred and first attempt.

Here are some of the most common ways to fix flaky tests on openSuSE. These fixes are generally very simple and yet have significantly contributed to reduce the amount of flakiness on openSuSE between October 2024 and April 2025.

QVERIFY() QCOMPARE() vs QTRY_VERIFY()/ QTRY_COMPARE()

A test condition may rely on event processing or the completion of another thread to evaluate as true.

QVERIFY/QCOMPARE macros check a condition once while the QTRY_VERIFY / QTRY_COMPARE macros process events in the UI thread for a set length of time, until the condition is true. If that condition is not evaluated as true before the time expires, it returns false.

This simple use of the QTRY_VERIFY/QTRY_COMPARE macros has helped identify and correct many flaky tests, particularly on OpenSuSE. Both QVERIFY and QCOMPARE serve similar purposes in this context.

By using QTRY_VERIFY we repeatedly check if the textEditObject meets the isCursorVisible condition. In contrast, QVERIFY checks the condition only once, immediately failing the test if the condition is not met.

An example patch fixing a flaky test by only changing the QVERIFY for the QTRY_VERIFY macro

Change for QTRY_VERIFY flaky tst_qquicktextedit::cursorDelegate (624128) · Gerrit Code Review

ENSURE RELIABLE INITIALIZATION

In asynchronous processing, variables may unexpectedly be initialized as a nullptr, particularly when asynchronous operations have not been completed yet.

This can happen when a method that returns a child object is invoked before the child object has been fully instantiated .

Verification of  child process result typically occurs afterwards. While this practice generally does not cause issues, it can still lead to bugs in asynchronous OS environments.

The code snippet below is an illustration:

For instance, this flaky test was failing with QVERIFY of line 7054 returning false. The reason being that cursorDelegate was actually instantiated as a nullptr, even though a child is actually present.

To ensure that the test passes, we should first verify that textInput->findChild<>() returns a value that is not a nullptr. Once confirmed, we can then assign this value.

MOUSE EVENTS/TOUCH EVENTS/GESTURES

In case of tests handling such as mouse events, touch events, and gestures, the position of the window is crucial. In asynchronous operating processing, a QWindow is initially shown without any frame (window.framePosition() == window.position()).

The frame is added afterward, causing the window to be moved down by the height of the frame. If a touch event is sent too early, it will be slightly off, and once the window position has been adjusted, the touch event will be sent

into the window frame rather than in the actual window. To address this, it is sometimes necessary to use a static position for the window,  obtain from example QGuiApplication::primaryScreen()->availableGeometry().topLeft().

IS THE WINDOW EXPOSED/ACTIVE/VISIBLE?

When checking if something on a window can be interacted with, it's important to ensure the window itself is visible/active. Some tests require clicking a button or testing a functionality within a QWidget.

Sometimes, the click event is sent prior to verifying the window's visibility, resulting in the button not being clicked even though the event was sent correctly. To address this, use QTest::qWaitForWindowActive(),

QTest::qWaitForWindowExposed() and window.isVisible(). Additionally, QTest::qWait() can be added to manually diagnose if the window is being shown or not.

HOW TO USE QTEST::QWAIT()

QTest::qWait() is a valuable debugging tool that allows us to observe the behavior of automated tests that would otherwise be executed too quickly to see. While it can help to understand the causes of flakiness, it is not a solution for fixing it and should not be used for that purpose.

Using QTest::qWait() permanently in a test is discouraged.

For instance, instead of waiting for a set time limit, we should wait until the window becomes visible.

QVERIFY(QTest::qWaitFor([&]{ return window.isVisible(); }));

QTest::qWait() should be used to more easily identify the causes of flakiness, but never as to fix them. A longer QTest::qWait() can also be used to manually perform the action that would otherwise have been done automatically in a specific test.

This approach helps determine whether the issue lies within the functionality being tested or in the test itself. It's easier to see if clicking a button has the desired effect when manually clicking it, rather then if the button is automatically clicked.

USING QDEBUG() << “some text” << something;

The use of qDebug() is useful because it provides a convenient way to output debugging information in Qt applications. qDebug() prints the output directly to the terminal when in debugging mode.

RESULTS

By utilizing these straightforward mechanisms to prevent flakiness, the reliability of passing tests was significantly enhanced. Here are a few examples of these fixed tests on openSuSE 15.

Set clear geometry for tst_QWidget::hoverPosition to remove flakiness (597821) · Gerrit Code Review

Change for QTRY_VERIFY2 in manual test largerThanScreen_QTBUG30142 (598362) · Gerrit Code Review

Change for QTRY_COMPARE in flaky auto test saveAndRestore() (602858) · Gerrit Code Review

Fix flakiness in tst_QWidget::saveRestoreGeometry() (607593) · Gerrit Code Review