QtCS2024 Flaky: Difference between revisions
Jump to navigation
Jump to search
(Stub page to introduce the topic.) |
m (Use of first person pronoun -> name.) |
||
| Line 1: | Line 1: | ||
We currently blacklist flaky tests, which means real failures they might have brought to our attention can get ignored. This should be a temporary state, but some have been blacklisted for many years. Each should be associated with a Jira ticket that won't be closed until the test is fixed and blacklisting removed; it's not clear that's always followed, though. How can we improve on this situation ? See also: [https://bugreports.qt.io/browse/QTQAINFRA-6400 QTQAINFRA-6400] and https://bugreports.qt.io/browse/QTBUG-96295 QTBUG-96295] and the [http://www.chaos.org.uk/~eddy/craft/FlakyTests.html notes | We currently blacklist flaky tests, which means real failures they might have brought to our attention can get ignored. This should be a temporary state, but some have been blacklisted for many years. Each should be associated with a Jira ticket that won't be closed until the test is fixed and blacklisting removed; it's not clear that's always followed, though. How can we improve on this situation ? See also: [https://bugreports.qt.io/browse/QTQAINFRA-6400 QTQAINFRA-6400] and https://bugreports.qt.io/browse/QTBUG-96295 QTBUG-96295] and the [http://www.chaos.org.uk/~eddy/craft/FlakyTests.html notes Eddy prepared in advance]. | ||
Revision as of 14:12, 3 September 2024
We currently blacklist flaky tests, which means real failures they might have brought to our attention can get ignored. This should be a temporary state, but some have been blacklisted for many years. Each should be associated with a Jira ticket that won't be closed until the test is fixed and blacklisting removed; it's not clear that's always followed, though. How can we improve on this situation ? See also: QTQAINFRA-6400 and https://bugreports.qt.io/browse/QTBUG-96295 QTBUG-96295] and the notes Eddy prepared in advance.