Qt Quality Gate Criteria

From Qt Wiki
Revision as of 00:19, 24 March 2016 by Wieland (talk | contribs) (clean-up)
Jump to navigation Jump to search


Qt Quality Gate Criteria

These criteria are the minimum standard for submitting code to any Qt stable branch.

Definitions

  • The Submission is any change (addition, modification, or removal) that is submitted to a Qt branch. It may consist of code, comments, documentation, resources, or other content.
  • Building The Package refers to compiling and linking the DLLs and executables, including test asset, that resides in a Qt package.

Basic Principles

  • Development teams are responsible to ensure all criteria are met before any code is submitted to master.
  • These criteria are a minimum set. Teams are encouraged to set their standards higher.
  • These quality criteria will be applied to all contributors.
  • Code needs to compile on all Tier 1 platforms (no regressions)
  • Has been (API) reviewed (i.e. follows coding conventions)
  • The code can be demonstrated (by means of demo/example/tests)
  • Meets a defined minimum threshold of coverage by means of Unit/System tests
  • All tests for the new feature pass plus no regressions in existing tests
  • Well documented (functional specification, class documentation, architectural overview, etc)
  • Maintenance agreed
  • Test Plan available

Bug-fixes (i.e. work done in resolving a defect) are only accepted into the Master Code Line if it meets the following criteria:

  • Code needs to compile on all Tier 1 platforms and work (no regressions)
  • Has been reviewed (i.e. follows coding conventions)
  • Fix fixes the reported error, and may also be verified by a new test (preferably automated Unit/System test)
  • Corresponding documentation is updated as needed (functional specification, class documentation, architectural overview, etc)

Compiling

Rule Responsibility Explanation
Compiles against recent version of master Contributor The Submission has been integrated with up-to-date Qt master.
Compiles on all Tier 1 platforms Contributor with QA Engineer The Package builds on all Tier 1 platforms that are defined for the product. Note that The Package includes not only the DLLs, but also other executables and code, such as the test asset, performance tests and examples. This package is able to be installed on target platforms completely and without user intervention.

Static/code-quality criteria

Rule Responsibility Explanation
Compliant coding convention Contributor For QML see: qml coding conventions
qdoc - zero warnings Community Documentation for the submission should be complete and relative to the standard for the submission. For instance a major API contribution requires significant documentation whereas a feature will require only documentation related directly to its use.
Compile - zero warnings on Linux and minimal on remaining Tier 1 platforms Contributor Some warnings are more serious than the others, so we obviously care most about the important ones.

Performance criteria

Rule Responsibility Explanation
Autotest performance Contributor with Integration Engineer The measured unit test performance no worse unless approved if module has been submitted previously
Performance test coverage Contributor with QA Engineer Use-cases are covered by performance tests.
Streamlined platform interaction Contributor Native Qt classes no more than half speed of equivalent native implementation

Error management

Rule Responsibility Explanation
P0/1 tasks - zero open QA Engineer Monitoring of all teams active tasks against Qt and ensuring consistent error management and progressing of issues.
Untriaged tasked - zero Contributor with QA Engineer All bugs should be triaged and a priority assigned.
Relative reduction in bug count Contributor with QA Engineer Weighted bug total no worse than previous release.

Future Wishlist (i.e. pointless to implement this until we get the items above under control)

Rule Responsibility Explanation
Autotests - 80% function coverage, 70% executable Contributor and QA Engineer Autotests must, in bare minimum, call most functions in exported public API. Furthermore, line coverage of autotests should exceed 70%, unless there's a genuine technical reason why that cannot be reached. Note that these numbers are not necessarily the definition of good autotest coverage, but our attempt to define what would be the bare acceptable minimum.
Autotests pass-rate > 90% Contributor and Integration Engineer In principle: autotests (in other words, automatically executed unit and module tests) must pass. Regressions are not allowed.