QtCS2021 - Testing upstream changes with downstream modules

Session Summary

A discussion to find the best solution to the issue of changes in one module (upstream) not being tested with modules (downstream) that depend on that module.

The goal of this work is to save the time of both upstream and downstream developers by providing faster feedback for upstream changes that may affect downstream modules.

Session Owners

• Mitch Curtis (mitch.curtis@qt.io)

Notes

Run tests for specific dependent modules

Have some way of telling CI that this module requires tests from other modules (doesn't have to be all dependent modules; can be a subset) to pass in addition to the module's own tests.

Pros

• Saves developer time (excluding CI integrations, which don’t count since they shouldn’t be wasting developer time).
  • qtdeclarative devs don't need to build qtquickcontrols2.
  • Git operations for both modules don't get slower.
  • Working on the modules in Creator doesn't get slower.

Cons

• Requires extra infrastructure to support it in CI.
• In the case of qtquickcontrols2, we miss out on the ability to use controls in the examples of qtdeclarative and its dependent modules.
• Some changes need to break downstream modules (e.g. API changes, etc.), so blocking CI because of downstream modules would be an issue.

Notes

• Could do a one-off system where the developer can press a button on Gerrit to run tests?
• Could do a smaller, important subset of all downstream tests.
• We do already have a way of running one-off tests of other modules:
  • You prepare a change like this https://codereview.qt-project.org/c/qt/qt5/+/352361 and press Precheck, where the submodule sha1s are unmerged changes. This is how you do it for per-modules: https://codereview.qt-project.org/c/qt/qtquickcontrols2/+/352695. Not a nice developer experience.

Un-modularise the CI system

This is probably partially related to the first solution. Taken from https://bugreports.qt.io/browse/COIN-133:

Pros

• There is only one "module" in the integration setup: qt5.
• This means that we get full revdep coverage.
• All changes staged for the same branch in different submodules are integrated at the same time.
  • This conveniently introduces atomicity, and thus solves the problem introduced by revdeps in a modular CI.
• When an integration succeeds, the supermodule sha1 are instantly updated as well.
• The CI should also test the packaging stages, so we have end-to-end coverage.

Cons

• Requires a lot of infrastructure work.
• Significantly increases the performance demands of the system. https://bugreports.qt.io/browse/COIN-134 could help with that.

Move specific modules into the repository of the module they depend on

Where it makes sense, dependent modules can simply be moved into the repository of the module they depend on.

Pros

• Fairly straightforward solution that requires no extra infrastructure.
• In the case of qtquickcontrols2, we gain the ability to use controls in the examples of qtdeclarative and its dependent modules.

Cons

• Slower git operations.
• Slower build times for qtdeclarative devs. You can use e.g. ninja targets, but you would have to do this every time you build. Ideally we’d add a configure option for this, like -no-gui in qtbase, so that they don’t need to build qtquickcontrols2.
• Creator would probably be quite a bit slower to work with having both of these in one project.
• May need to rename qtdeclarative.git?
• Doesn't scale; doesn't make sense for all other modules.

Notes

• Could split QML out into its own repository, and have Quick and Controls in one repo.
• Graft history in, don't merge it.
• Git bisect could be problematic.
  • If the bisect crosses the merge point, qtquickcontrols2 would appear/disappear.