CI Overview

Introduction

Shortening the time between the injection of a defect and detecting/fixing the problem has significant cost benefits. The QA team is maintaining a set of tools that provides a Continuous Integration (CI) infrastructure that enables SW development and QA to reach those goals.

The goals for the system are:

• Implement an automated quality gate that blocks poor quality changes from being merged in Git.
• Make test results available to developer in a clear, concise, and user friendly way.

Responsibilities

Task	Task Description	Point of Contact
General HW/VM maintenance	Ensuring test equipment remains stable and working. Responsible for the architecture of the infrastructure.	Tony Sarajärvi
Coin	Make sure our Continuous Integration software (Coin) is up. Leads the development of the software.	Aapo Keskimölö
Open governance (Gerrit)	Ensure that Gerrit's Git database is working and troubleshoot various error situations with it	Oswald Buddenhagen
Support	Front-line assistance with any issues arising from CI system	qt.ci@qt.io

Continuous Integration (CI) System

All changes to Qt go through Gerrit (http://codereview.qt-project.org). A majority of projects there are tracked by a continuous integration system (Coin). A tracked project doesn't have a submit possibility for the commit, but instead has a merge patch for staging option. When a commit has been staged, it is judged by Coin whether or not it can be merged into upstream.

CI status

• Patches currently being tested can be viewed with a Gerrit search: http://codereview.qt-project.org/#q,status:integrating,n,z
• Patches staged for testing in the next build can also be viewed: http://codereview.qt-project.org/#q,status:staged,n,z
• Coin's dashboard is visible here: https://testresults.qt.io/coin
• Current status of the infrastructure is found here: https://testresults.qt.io/grafana

Test Results

All logs are published here https://testresults.qt.io/logs/, but knowing which log belongs to your build is quite difficult from that view. Instead one should see the links attached to the Gerrit commit or from Coin's dashboard to be able to find the right logs for your build.

Qt5 Repository High Level Structure

In Qt 5, various components of Qt have been split out into separate repositories, as part of the Qt modularization project.

This is intended to simplify the development of individual modules. Likewise, the repository and continuous integration setup with respect to individual modules is simpler, with only one level of continuous integration.

The main components in the repository structure are:

Superproject repository (qt/qt5.git)

• This repository contains several git submodules. Cloning and building this repository is conceptually similar to doing a full build of pre-modularization Qt.

Module repositories

• Git repositories exist for each Qt submodule (e.g. qtbase, qtdeclarative, qtqa, ...)

Feature and other branches

• Dedicated feature branches are useful to gain more flexibility during development and experiment more freely, without risking the quality of other branches. This is the other main area in which development happens.

What exactly does Coin do?

When Coin receives a notification that commits have been staged for a specific Qt module and a certain branch, it waits a while if further commits are being staged. This gives us the possibility to stage multiple commits in one go.

Coin goes through the following steps to determine if the commits are OK for merging:

• Compile the module and its dependencies
• Run the autotest suite for the module under test (via `make check')
• If the autotest suite succeed => the change gets integrated.
• If one autotest fail => we run it again:
  • if it succeed, it's considered flaky => the change gets integrated anyway.
  • if it fails again, it's an stable regression => the change gets rejected.

The results are then published to http://testresults.qt.io and Gerrit is notified whether or not it should merge or reject the commit(s).

Why commits might get rejected by the CI System?

For Qt 5 (mostly all the modules in Gerrit) all the autotests are supposed to compile and run successfully. The CI rejects the commit(s) for the following reasons:

• Sources don't compile
  • Even though sources were tested locally by the developer, the CI also compiles the sources on multiple operating systems and different compilers.
• Autotests fail
  • The commit(s) can either cause real regression, or the CI could have hit a flaky case somewhere else that's not related to the change at all. The CI unfortunately does not know this and rejects the commit(s).
• The CI infra fails
  • This could be a bug in Coin, crashing hosts, network connection issues, time-outs etc. These problems are related to the stability of the CI. It's a continuous work as we try to improve the infrastructure so that we minimize the amount of rejects the infra itself causes.

Whenever a change gets rejected, you will get a new comment on Gerrit.

Reproducing an autotest failure

You can always try to reproduce the failure locally by just running make and make check under the tests/auto folder. However, understandably developers don't have every platform where the CI tests at their disposal. If encountering a situation like this, feel free to contact qt-ci@qt.io and we can discuss how to provide you with an environment where this can be reproduced.

How do I customize my test's execution?

If you need to change some aspect of your test's execution, such as the handling of the exit code, the test's maximum permitted runtime, or the commands used to execute the test, see CI Autotest Metadata.

What tests are we running ?

See CI Configurations.

Exactly how are the CI machines configured?

The virtual machines on which builds are done upon are set up by using scripts. These scripts are stored in Git, in qt5.git to be precise. In qt5.git repository there's a folder called /coin/platform_configurations that contains plain text files that list all the configuration that are run. The folder /coin/provisioning on the other hand contains all the scripts that are run to configure these VMs. The configuration scripts are publicly available and for anyone to modify and update.

How can I see what the CI system is doing?

At the moment we don't have a real time view of what Coin is doing. However, as soon as a build is complete, its status is uploaded and dashboard at http://testresults.qt.io/coin is updated. The status page does not work if the CI system is down for maintenance or other reasons.

Reports for each test run, and system logs when something goes wrong, are mailed to the public mailing list at http://lists.qt-project.org/pipermail/ci-reports/ . Mails contain various "X-Qt-CI-*" headers which may be useful for filtering.

Reporting Bugs

Bugs relating to the public autotest infrastructure should be reported on https://bugreports.qt.io/ against the QTQAINFRA project.