QtCS2024 Flaky
We currently blacklist flaky tests, which means real failures they might have brought to our attention can get ignored. This should be a temporary state, but some have been blacklisted for many years. Each should be associated with a Jira ticket that won't be closed until the test is fixed and blacklisting removed; it's not clear that's always followed, though. How can we improve on this situation ? See also: QTQAINFRA-6400 and https://bugreports.qt.io/browse/QTBUG-96295 QTBUG-96295] and the notes Eddy prepared in advance.
Notes
- Can we delete blacklisted *tests* with an age older than X years? No, because:
- They might work on other platforms
- BLACKLISTS provides intelligence: They can e.g. give an indication to which areas there are problems in Qt.
- Ideally, nothing should be blacklisted. Unfortunately we don't live in an ideal world with deadlines or CI integration that blocks progress for many contributors.
- There are similar tools available:
- QSKIP (used instead of BLACKLIST if the test can cause crashes)
- QEXPECT_FAIL (if you *know* the test fails)
- BLACKLIST (for e.g. flaky tests)
Blacklisted tests should be run. We need to monitor their state over time (is it fixed?)
Every time a test is blacklisted, it should be accompanied with a JIRA issue (maybe gerrit bot can require a QTBUG-xxxxx)
We have a page that explains what we can do to reduce flakiness of a test.
Use "ci" blacklist tag to specify that it is a problem in CI only.