QtCS2024 Flaky: Difference between revisions
No edit summary |
No edit summary |
||
| Line 1: | Line 1: | ||
We currently blacklist flaky tests, which means real failures they might have brought to our attention can get ignored. This should be a temporary state, but some have been blacklisted for many years. Each should be associated with a Jira ticket that won't be closed until the test is fixed and blacklisting removed; it's not clear that's always followed, though. How can we improve on this situation ? See also: [https://bugreports.qt.io/browse/QTQAINFRA-6400 QTQAINFRA-6400] and https://bugreports.qt.io/browse/QTBUG-96295 QTBUG-96295] and the [http://www.chaos.org.uk/~eddy/craft/FlakyTests.html notes Eddy prepared in advance]. | We currently blacklist flaky tests, which means real failures they might have brought to our attention can get ignored. This should be a temporary state, but some have been blacklisted for many years. Each should be associated with a Jira ticket that won't be closed until the test is fixed and blacklisting removed; it's not clear that's always followed, though. How can we improve on this situation ? See also: [https://bugreports.qt.io/browse/QTQAINFRA-6400 QTQAINFRA-6400] and https://bugreports.qt.io/browse/QTBUG-96295 QTBUG-96295] and the [http://www.chaos.org.uk/~eddy/craft/FlakyTests.html notes Eddy prepared in advance]. | ||
== Notes == | |||
* Can we delete blacklisted *tests* with an age older than X years? No, because: | |||
** They might work on other platforms | |||
** BLACKLISTS provides intelligence: They can e.g. give an indication to which areas there are problems in Qt. | |||
* Ideally, nothing should be blacklisted. Unfortunately we don't live in an ideal world with deadlines or CI integration that blocks progress for many contributors. | |||
* There are similar tools available: | |||
** QSKIP (used instead of BLACKLIST if the test can cause crashes) | |||
** QEXPECT_FAIL (if you *know* the test fails) | |||
** BLACKLIST (for e.g. flaky tests) | |||
Blacklisted tests should be run. We need to monitor their state over time (is it fixed?) | |||
Every time a test is blacklisted, it should be accompanied with a JIRA issue (maybe gerrit bot can require a QTBUG-xxxxx) | |||
We have a [[How to reproduce autotest fails|page]] that explains what we can do to reduce flakiness of a test. | |||
Use "ci" blacklist tag to specify that it is a problem in CI only. | |||
[[Category:QtCS2024]] | [[Category:QtCS2024]] | ||
Revision as of 13:41, 6 September 2024
We currently blacklist flaky tests, which means real failures they might have brought to our attention can get ignored. This should be a temporary state, but some have been blacklisted for many years. Each should be associated with a Jira ticket that won't be closed until the test is fixed and blacklisting removed; it's not clear that's always followed, though. How can we improve on this situation ? See also: QTQAINFRA-6400 and https://bugreports.qt.io/browse/QTBUG-96295 QTBUG-96295] and the notes Eddy prepared in advance.
Notes
- Can we delete blacklisted *tests* with an age older than X years? No, because:
- They might work on other platforms
- BLACKLISTS provides intelligence: They can e.g. give an indication to which areas there are problems in Qt.
- Ideally, nothing should be blacklisted. Unfortunately we don't live in an ideal world with deadlines or CI integration that blocks progress for many contributors.
- There are similar tools available:
- QSKIP (used instead of BLACKLIST if the test can cause crashes)
- QEXPECT_FAIL (if you *know* the test fails)
- BLACKLIST (for e.g. flaky tests)
Blacklisted tests should be run. We need to monitor their state over time (is it fixed?)
Every time a test is blacklisted, it should be accompanied with a JIRA issue (maybe gerrit bot can require a QTBUG-xxxxx)
We have a page that explains what we can do to reduce flakiness of a test.
Use "ci" blacklist tag to specify that it is a problem in CI only.