CI Overview: Difference between revisions

From Qt Wiki
Jump to navigation Jump to search
No edit summary
 
No edit summary
Line 1: Line 1:
==Introduction==
[[Category:Developing_Qt::QA]]
 
[toc align_right="yes" depth="2"]
 
== Introduction ==


Shortening the time between the injection of a defect and detecting/fixing the problem has significant cost benefits. The QA team is maintaining a set of tools that provides a Continuous Integration (CI) infrastructure that enables SW development and QA to reach those goals.
Shortening the time between the injection of a defect and detecting/fixing the problem has significant cost benefits. The QA team is maintaining a set of tools that provides a Continuous Integration (CI) infrastructure that enables SW development and QA to reach those goals.
Line 9: Line 13:
* Make test results available to developer in a clear, concise, and user friendly way.
* Make test results available to developer in a clear, concise, and user friendly way.


==Responsibilities==
== Responsibilities ==


{| class="infotable line"
|''. Task |''. Task Description |_. Point of Contact |<br />| General HW/VM maintenance | Ensuring test equipment remains stable and working<br />Ensuring we have the necessary hardware and software to meet Tier 1 testing needs | Tony Sarajärvi (https://qt.io/member/132527) |<br />| General scripting | Generic scripting work, i.e. the glue that makes the system work<br />Maintenance of &lt;code&amp;gt;qtqa&amp;lt;/code&amp;gt; Qt5 module | Tony Sarajärvi (https://qt.io/member/132527) |<br />| Open governance (Gerrit) | Scripting/maintenance relating to open test infrastructure (e.g. public build logs)<br />Community point of contact for CI change requests | Tony Sarajärvi (https://qt.io/member/132527) |<br />| Support | Front-line assistance with any issues arising from CI system | Tony Sarajärvi (https://qt.io/member/132527) |
! Task
! Task Description
! Point of Contact
|-
| General HW/VM maintenance
| Ensuring test equipment remains stable and working <br /> Ensuring we have the necessary hardware and software to meet Tier 1 testing needs
|
Tony Sarajärvi (https://qt.io/member/132527)
|-
| General scripting
| Generic scripting work, i.e. the glue that makes the system work <br /> Maintenance of <code>qtqa</code> Qt5 module
|
Tony Sarajärvi (https://qt.io/member/132527)
|-
| Open governance (Gerrit)
| Scripting/maintenance relating to open test infrastructure (e.g. public build logs) <br /> Community point of contact for CI change requests
|
Tony Sarajärvi (https://qt.io/member/132527)
|-
| Support
| Front-line assistance with any issues arising from CI system
|
Tony Sarajärvi (https://qt.io/member/132527)
|}


==Continuous Integration (CI) System==
== Continuous Integration (CI) System ==


All changes to Qt must be tested before integrated upstream:
All changes to Qt must be tested before integrated upstream:
Line 43: Line 23:
* All feature development must happen in a feature branch
* All feature development must happen in a feature branch
* All bug fixing goes into Gerrit
* All bug fixing goes into Gerrit
** Continuously tested; check build and autotest results before integrating
** Continuously tested; check build and autotest results before integrating  


Jenkins is used for continuous building and running autotests.
Jenkins is used for continuous building and running autotests.


===CI status===
=== CI status ===


<nowiki>*Patches currently being tested can be viewed with a gerrit search: </nowiki>http://codereview.qt.io/#q,status:integrating,n,z<br /> *Patches staged for testing in the next build can also be viewed: http://codereview.qt.io/#q,status:staged,n,z
*Patches currently being tested can be viewed with a gerrit search: http://codereview.qt.io/#q,status:integrating,n,z<br />*Patches staged for testing in the next build can also be viewed: http://codereview.qt.io/#q,status:staged,n,z


===Test Results===
=== Test Results ===


Results are made available on http://testresults.qt.io/ci
Results are made available on http://testresults.qt.io/ci
Line 57: Line 37:
At time of writing, results are made available at the end of each test run; a test run currently in progress cannot be viewed.
At time of writing, results are made available at the end of each test run; a test run currently in progress cannot be viewed.


===Qt5 Repository High Level Structure===
=== Qt5 Repository High Level Structure ===


In Qt 5, various components of Qt have been split out into separate repositories, as part of the Qt modularization project.
In Qt 5, various components of Qt have been split out into separate repositories, as part of the Qt modularization project.
Line 65: Line 45:
The main components in the repository structure are:
The main components in the repository structure are:


'''Superproject repository'''
'''Superproject repository'''<br />'''''' This repository contains several git submodules. Cloning and building this repository is conceptually similar to doing a full build of pre-modularization Qt.<br />'''''' '''Mainline superproject repository (e.g. qt/qt5.git)''':<br />Only the Continuous Integration system pushes to this repository.<br />'''Module repositories'''<br />'''''' Git repositories exist for each Qt submodule (e.g. qtbase, qtdeclarative, qtqta …)<br />* '''Mainline module repository (e.g. qt/qtbase.git)''':<br />Only the Continuous Integration system pushes to this repository.<br />'''Feature and other branches'''<br />'''''' Dedicated feature branches are useful to gain more flexibility during development and experiment more freely, without risking the quality of other branches. This is the other main area in which development happens.


* This repository contains several git submodules. Cloning and building this repository is conceptually similar to doing a full build of pre-modularization Qt.
== Why commits might get rejected by the CI System? ==
* '''Mainline superproject repository (e.g. qt/qt5.git)''':<br /> Only the Continuous Integration system pushes to this repository.


<br />'''Module repositories'''
For Qt 5 (mostly all the modules in [https://codereview.qt.io/ Gerrit]) all the autotests are supposed to compile and run successfully. So the actual procedure to reject a change is '''<span class="roughly"></span>''':
 
* Git repositories exist for each Qt submodule (e.g. qtbase, qtdeclarative, qtqta …)
 
* '''Mainline module repository (e.g. qt/qtbase.git)''':<br /> Only the Continuous Integration system pushes to this repository.
 
<br />'''Feature and other branches'''
 
* Dedicated feature branches are useful to gain more flexibility during development and experiment more freely, without risking the quality of other branches. This is the other main area in which development happens.
 
==Why commits might get rejected by the CI System?==
 
For Qt 5 (mostly all the modules in [https://codereview.qt.io/ Gerrit]) all the autotests are supposed to compile and run successfully. So the actual procedure to reject a change is '''(roughly)''':


* Compile the module and its dependencies
* Compile the module and its dependencies
* Run the autotest suite for the module under test (via `<code>make check</code>’)
* Run the autotest suite for the module under test (via `&amp;lt;code&amp;gt;make check&amp;lt;/code&amp;gt;')
* If the autotest suite succeed =&gt; the change gets '''integrated'''.
* If the autotest suite succeed =&gt; the change gets '''integrated'''.
* If one autotest fail =&gt; we run it again:
* If one autotest fail =&gt; we run it again:
** if it succeed, it’s considered flaky =&gt; the change gets '''integrated''' anyway.
** if it succeed, it's considered flaky =&gt; the change gets '''integrated''' anyway.
** if it fails again, it’s an stable regression =&gt; the change gets '''rejected'''.
** if it fails again, it's an stable regression =&gt; the change gets '''rejected'''.


Whenever a change gets rejected, you will get a new comment on Gerrit. The rejection might be caused by either a build failure or an autotests failure (regression).
Whenever a change gets rejected, you will get a new comment on Gerrit. The rejection might be caused by either a build failure or an autotests failure (regression).


===Reproducing an autotest failure===
=== Reproducing an autotest failure ===


You might want to try to reproduce what the CI System is doing (as there is a high hope that your changes are compiling and passing the autotests in your local environment).
You might want to try to reproduce what the CI System is doing (as there is a high hope that your changes are compiling and passing the autotests in your local environment).
Line 99: Line 66:
For this to be done, there are two relevant repositories containing the scripts that the CI System uses:
For this to be done, there are two relevant repositories containing the scripts that the CI System uses:


* http://qt.gitorious.org/qt/qtqa &lt;- the test scripts themselves
* http://qt.gitorious.org/qt/qtqa &lt;- the test scripts themselves  
* http://qt.gitorious.org/qtqa/testconfig &lt;- the precise data for which scripts are used in which test configurations
* http://qt.gitorious.org/qtqa/testconfig &lt;- the precise data for which scripts are used in which test configurations


Note: You might have to setup your perl dependencies for the <code>qtqa</code> module the very first time by reading:
Note: You might have to setup your perl dependencies for the &lt;code&amp;gt;qtqa&amp;lt;/code&amp;gt; module the very first time by reading:
 
The scripts in qt/qtqa module may be used independently.<br /> However, if you want to reproduce the same options and environment as used in the Qt CI system, you can invoke them through the test.pl script in the testconfig repository.<br /> So, the basic steps are (for Linux):
 
==How do I customize my test’s execution?==
 
If you need to change some aspect of your test’s execution, such as the handling of the exit code, the test’s maximum permitted runtime, or the commands used to execute the test, see [[CI Autotest Metadata|CI_Autotest_Metadata]].
 
==What tests are we running ?==
 
See [[CI Configurations|CI_Configurations]].
 
==Exactly how are the CI machines configured?==
 
See [[CI Machine Configuration|CI_Machine_Configuration]].
 
==How can I see what the CI system is doing?==
 
The CI system’s status for each project can be observed in near-realtime at http://testresults.qt.io/ci/status/<br /> (or raw data at http://testresults.qt.io/cgi-bin/ci-api?pretty=true if you prefer). The status page does not work if the CI system is down for maintenance or other reasons.
 
Reports for each test run, and system logs when something goes wrong, are mailed to the public mailing list at http://lists.qt.io/pipermail/ci-reports/ . Mails contain various “X-Qt-CI-*” headers which may be useful for filtering.


==Reporting Bugs==
<code><br /># git clone git://gitorious.org/qt/qtqa.git _qtqa_latest<br /># perldoc _qtqa_latest/scripts/setup.pl<br /></code>


Bugs relating to the public autotest infrastructure should be reported on https://bugreports.qt.io/ against the <span class="caps">QTQAINFRA</span> project.
The scripts in qt/qtqa module may be used independently.<br />However, if you want to reproduce the same options and environment as used in the Qt CI system, you can invoke them through the test.pl script in the testconfig repository.<br />So, the basic steps are (for Linux):


===Categories:===
<code><br /># git clone git://gitorious.org/qt/qtbase.git<br /># cd qtbase<br /># git clone git://gitorious.org/qtqa/testconfig.git _testconfig<br /># perl _testconfig/test.pl —project 'QtBase master Integration' —stage 'linux-g++<s>32 Ubuntu 10.04 x86' 2&amp;gt;&amp;1 | tee testlog.txt<br /># &amp;lt;wait for the results&amp;amp;gt;<br /></code>
<br />h2. How do I customize my test's execution?
<br />If you need to change some aspect of your test's execution, such as the handling of the exit code, the test's maximum permitted runtime, or the commands used to execute the test, see [[CI_Autotest_Metadata]].
<br />h2. What tests are we running ?
<br />See [[CI_Configurations]].
<br />h2. Exactly how are the CI machines configured?
<br />See [[CI_Machine_Configuration]].
<br />h2. How can I see what the CI system is doing?
<br />The CI system's status for each project can be observed in near-realtime at http://testresults.qt.io/ci/status/<br />(or raw data at http://testresults.qt.io/cgi-bin/ci-api?pretty=true if you prefer). The status page does not work if the CI system is down for maintenance or other reasons.
<br />Reports for each test run, and system logs when something goes wrong, are mailed to the public mailing list at http://lists.qt.io/pipermail/ci-reports/ . Mails contain various &quot;X-Qt-CI</s>*&quot; headers which may be useful for filtering.


* [[:Category:Developing Qt|Developing_Qt]]
== Reporting Bugs ==
** [[:Category:Developing Qt::QA|QA]]

Revision as of 07:44, 24 February 2015


[toc align_right="yes&quot; depth="2&quot;]

Introduction

Shortening the time between the injection of a defect and detecting/fixing the problem has significant cost benefits. The QA team is maintaining a set of tools that provides a Continuous Integration (CI) infrastructure that enables SW development and QA to reach those goals.

The goals for the system are:

  • Implement an automated quality gate that blocks poor quality changes from entering the stable branch,
  • Run an exhaustive set of configuration tests in a nightly fashion,
  • Make test results available to developer in a clear, concise, and user friendly way.

Responsibilities

|. Task |. Task Description |_. Point of Contact |
| General HW/VM maintenance | Ensuring test equipment remains stable and working
Ensuring we have the necessary hardware and software to meet Tier 1 testing needs | Tony Sarajärvi (https://qt.io/member/132527) |
| General scripting | Generic scripting work, i.e. the glue that makes the system work
Maintenance of <code&gt;qtqa&lt;/code&gt; Qt5 module | Tony Sarajärvi (https://qt.io/member/132527) |
| Open governance (Gerrit) | Scripting/maintenance relating to open test infrastructure (e.g. public build logs)
Community point of contact for CI change requests | Tony Sarajärvi (https://qt.io/member/132527) |
| Support | Front-line assistance with any issues arising from CI system | Tony Sarajärvi (https://qt.io/member/132527) |

Continuous Integration (CI) System

All changes to Qt must be tested before integrated upstream:

  • All feature development must happen in a feature branch
  • All bug fixing goes into Gerrit
    • Continuously tested; check build and autotest results before integrating

Jenkins is used for continuous building and running autotests.

CI status

Test Results

Results are made available on http://testresults.qt.io/ci

At time of writing, results are made available at the end of each test run; a test run currently in progress cannot be viewed.

Qt5 Repository High Level Structure

In Qt 5, various components of Qt have been split out into separate repositories, as part of the Qt modularization project.

This is intended to simplify the development of individual modules. Likewise, the repository and continuous integration setup with respect to individual modules is simpler, with only one level of continuous integration.

The main components in the repository structure are:

Superproject repository
' This repository contains several git submodules. Cloning and building this repository is conceptually similar to doing a full build of pre-modularization Qt.
'
Mainline superproject repository (e.g. qt/qt5.git):
Only the Continuous Integration system pushes to this repository.
Module repositories
' Git repositories exist for each Qt submodule (e.g. qtbase, qtdeclarative, qtqta …)
*
Mainline module repository (e.g. qt/qtbase.git):
Only the Continuous Integration system pushes to this repository.
Feature and other branches
'
Dedicated feature branches are useful to gain more flexibility during development and experiment more freely, without risking the quality of other branches. This is the other main area in which development happens.

Why commits might get rejected by the CI System?

For Qt 5 (mostly all the modules in Gerrit) all the autotests are supposed to compile and run successfully. So the actual procedure to reject a change is :

  • Compile the module and its dependencies
  • Run the autotest suite for the module under test (via `&lt;code&gt;make check&lt;/code&gt;')
  • If the autotest suite succeed => the change gets integrated.
  • If one autotest fail => we run it again:
    • if it succeed, it's considered flaky => the change gets integrated anyway.
    • if it fails again, it's an stable regression => the change gets rejected.

Whenever a change gets rejected, you will get a new comment on Gerrit. The rejection might be caused by either a build failure or an autotests failure (regression).

Reproducing an autotest failure

You might want to try to reproduce what the CI System is doing (as there is a high hope that your changes are compiling and passing the autotests in your local environment).

For this to be done, there are two relevant repositories containing the scripts that the CI System uses:

Note: You might have to setup your perl dependencies for the <code&gt;qtqa&lt;/code&gt; module the very first time by reading:

<br /># git clone git://gitorious.org/qt/qtqa.git _qtqa_latest<br /># perldoc _qtqa_latest/scripts/setup.pl<br />

The scripts in qt/qtqa module may be used independently.
However, if you want to reproduce the same options and environment as used in the Qt CI system, you can invoke them through the test.pl script in the testconfig repository.
So, the basic steps are (for Linux):

<br /># git clone git://gitorious.org/qt/qtbase.git<br /># cd qtbase<br /># git clone git://gitorious.org/qtqa/testconfig.git _testconfig<br /># perl _testconfig/test.pl —project 'QtBase master Integration' —stage 'linux-g++<s>32 Ubuntu 10.04 x86' 2&amp;gt;&amp;1 | tee testlog.txt<br /># &amp;lt;wait for the results&amp;amp;gt;<br />


h2. How do I customize my test's execution?
If you need to change some aspect of your test's execution, such as the handling of the exit code, the test's maximum permitted runtime, or the commands used to execute the test, see CI_Autotest_Metadata.
h2. What tests are we running ?
See CI_Configurations.
h2. Exactly how are the CI machines configured?
See CI_Machine_Configuration.
h2. How can I see what the CI system is doing?
The CI system's status for each project can be observed in near-realtime at http://testresults.qt.io/ci/status/
(or raw data at http://testresults.qt.io/cgi-bin/ci-api?pretty=true if you prefer). The status page does not work if the CI system is down for maintenance or other reasons.
Reports for each test run, and system logs when something goes wrong, are mailed to the public mailing list at http://lists.qt.io/pipermail/ci-reports/ . Mails contain various "X-Qt-CI*" headers which may be useful for filtering.

Reporting Bugs