QtCS2024 Flaky

We currently blacklist flaky tests, which means real failures they might have brought to our attention can get ignored. This should be a temporary state, but some have been blacklisted for many years. Each should be associated with a Jira ticket that won't be closed until the test is fixed and blacklisting removed; it's not clear that's always followed, though. How can we improve on this situation ? See also: QTQAINFRA-6400 and https://bugreports.qt.io/browse/QTBUG-96295 QTBUG-96295] and the notes Eddy prepared in advance.

Notes

• Can we delete blacklisted *tests* with an age older than X years? No, because:
  • They might work on other platforms
  • BLACKLISTS provides intelligence: They can e.g. give an indication to which areas there are problems in Qt.
• Ideally, nothing should be blacklisted. Unfortunately we don't live in an ideal world with deadlines or CI integration that blocks progress for many contributors.
• There are similar tools available:
  • QSKIP (used instead of BLACKLIST if the test can cause crashes)
  • QEXPECT_FAIL (if you *know* the test fails)
  • BLACKLIST (for e.g. flaky tests)

Blacklisted tests should be run. We need to monitor their state over time (is it fixed?)

Every time a test is blacklisted, it should be accompanied with a JIRA issue (maybe gerrit bot can require a QTBUG-xxxxx)

We have a page that explains what we can do to reduce flakiness of a test.

Use "ci" blacklist tag to specify that it is a problem in CI only.