Qt Metrics 2 Description: Difference between revisions

From Qt Wiki
Jump to navigation Jump to search
(The metrics page link changed as the official page was launced)
(Wrote ch. Requirements and Target Audience)
Line 19: Line 19:
=== System Overview ===
=== System Overview ===


The Qt Metrics system consists of three main components: the parser, the report builder and the database. See the picture below. The parser reads the build logs, test results, or other files that are created in the development and integration process, and pushes the data into the database. The report builder reads the data from the database, and shows predefined metrics and reports based on user's area of interest. The report builder with PHP (Slim framework), Twitter Bootstrap, HTML, CSS, and JavaScript with some jQuery and D3 graphics components. The parser is implemented with Perl (implementation for the 'new-CI' not done yet and solution therefore still open). The database is a MariaDB database.
The Qt Metrics system consists of three main components: the parser, the report builder and the database. See the picture below. The parser reads the build logs, test results, or other files that are created in the development and integration process, and pushes the data into the database. The report builder reads the data from the database, and shows predefined metrics and reports based on user's area of interest. The report builder is implemented with PHP (Slim framework), Twitter Bootstrap, HTML, CSS, and JavaScript with some jQuery and D3 graphics components. The parser is implemented with Perl (implementation for the 'new-CI' not done yet and solution therefore still open). The database is a MariaDB database.


[[Image:Qtmetricssystem.png]]
[[Image:Qtmetricssystem.png]]
Line 29: Line 29:
=== Requirements and Target Audience ===
=== Requirements and Target Audience ===


{| class="wikitable"
==== System Requirements ====
|-
 
| '''TBD'''
The following general requirements have been identified:
|}
* Because Qt is based on open source, it is recommended to use open source technologies on both the metrics system backend and the frontend solutions. The selected technologies and technical solutions shall fit also for other areas (in addition to CI) that will be implemented later on.
* Anyone in the Qt developer community should be able to create new reports on the metrics pages.
* The user of the web site should be able to filter the information based on different criteria and to browse relevant information for his or her needs.
 
==== User stories ====
 
The target audience for the metrics site are the Qt integration teams (in The Qt Company) and the global Qt developer community.
 
The following user needs have been identified (a need may interest several roles although only one listed):
 
* As a Qt developer I want to see have autotests on my area failed recently (e.g. tst_qdbusconnection)
* As a Qt developer I want to know about a Qt module/project (e.g. qtbase) and see why it fails so often
* As a Qt developer I want to see which test I should fix to improve our coverage (e.g. I look at the module and see that tst_qwidget is red on most platforms, I click a link and dive deeper and how see how this test fails)
* As a module integrator I want to see the status of a Qt module/project (e.g. qtdeclarative)
* As a platform integrator I want to see the status of my platform (e.g. Windows), see which tests fail often and which ones are insignificant on this platform, and to click a link to a test to see more details.
* As a platform integrator I want to easily study a recent build log
* As a platform integrator I want to see if a configuration is untested because it's insignificant
* As a platform integrator I want to see a report of the most statistically relevant failing/flaky tests of the past week in order to start fixing the most relevant things step by step and see the status to improve
* As a tech lead I want to be able to dive from overview level into detailed data
* As a tech lead I want to see statistics
* As a release manager I want to see the status of a forthcoming release (e.g. 5.5.1)
* As a CI support person I want to see which test failed lately (randomly)
* As a CI support person I want to see which tests could be raised from insignificant to significant
* As a manager I want to see an easy-to-read overview of the CI status
 
See also the ch. [[#FUTURE DEVELOPMENT|FUTURE DEVELOPMENT]].


<br>
<br>

Revision as of 09:45, 1 October 2015

UNDER CONSTRUCTION

INTRODUCTION

The Qt Metrics 2 web portal visualizes the progress of Qt's quality, focusing on Continuous Integration (CI). The goal is to automate laborious manual work that is required, for example, in the CI release cycle, to report the key information, as well as providing real time data with good performance. The Qt integration teams (in The Qt Company) and the global Qt developer community are the target audience.

Qt Metrics 2: http://testresults.qt.io/qtmetrics/
Version: This document is updated against the report builder v0.20

This document is divided in chapters; each chapter having its own purpose and target audience:
2. Using the System: What (what the users see)
3. Detailed Description: How it works (quick overview to implementation)
4. Developing and Maintenance: How to develop it (how to create new metrics/pages)

System Overview

The Qt Metrics system consists of three main components: the parser, the report builder and the database. See the picture below. The parser reads the build logs, test results, or other files that are created in the development and integration process, and pushes the data into the database. The report builder reads the data from the database, and shows predefined metrics and reports based on user's area of interest. The report builder is implemented with PHP (Slim framework), Twitter Bootstrap, HTML, CSS, and JavaScript with some jQuery and D3 graphics components. The parser is implemented with Perl (implementation for the 'new-CI' not done yet and solution therefore still open). The database is a MariaDB database.

Qtmetricssystem.png
PICTURE. Generic system architecture



Requirements and Target Audience

System Requirements

The following general requirements have been identified:

  • Because Qt is based on open source, it is recommended to use open source technologies on both the metrics system backend and the frontend solutions. The selected technologies and technical solutions shall fit also for other areas (in addition to CI) that will be implemented later on.
  • Anyone in the Qt developer community should be able to create new reports on the metrics pages.
  • The user of the web site should be able to filter the information based on different criteria and to browse relevant information for his or her needs.

User stories

The target audience for the metrics site are the Qt integration teams (in The Qt Company) and the global Qt developer community.

The following user needs have been identified (a need may interest several roles although only one listed):

  • As a Qt developer I want to see have autotests on my area failed recently (e.g. tst_qdbusconnection)
  • As a Qt developer I want to know about a Qt module/project (e.g. qtbase) and see why it fails so often
  • As a Qt developer I want to see which test I should fix to improve our coverage (e.g. I look at the module and see that tst_qwidget is red on most platforms, I click a link and dive deeper and how see how this test fails)
  • As a module integrator I want to see the status of a Qt module/project (e.g. qtdeclarative)
  • As a platform integrator I want to see the status of my platform (e.g. Windows), see which tests fail often and which ones are insignificant on this platform, and to click a link to a test to see more details.
  • As a platform integrator I want to easily study a recent build log
  • As a platform integrator I want to see if a configuration is untested because it's insignificant
  • As a platform integrator I want to see a report of the most statistically relevant failing/flaky tests of the past week in order to start fixing the most relevant things step by step and see the status to improve
  • As a tech lead I want to be able to dive from overview level into detailed data
  • As a tech lead I want to see statistics
  • As a release manager I want to see the status of a forthcoming release (e.g. 5.5.1)
  • As a CI support person I want to see which test failed lately (randomly)
  • As a CI support person I want to see which tests could be raised from insignificant to significant
  • As a manager I want to see an easy-to-read overview of the CI status

See also the ch. FUTURE DEVELOPMENT.



USING THE SYSTEM

This section introduces the Qt Metrics system from the user point of view.

Report Builder Overview

TBD

The report builder is a collection of PHP files running on the testresults server (running Apache) on the qt.io domain. It utilizes Ajax, JavaScript, and HTML/CSS (HTML5/CSS3 for some visual appearance). Picture 2 below describes the logical elements and functionality of the report builder (please note that the latest changes may not be reflected in all the pictures in this document).

Use Cases

TBD

List roles incl admin.



DETAILED DESCRIPTION

This section discusses the Qt Metrics system components and their implementation into a certain level of detail. The target audience is the SW designer(s) maintaining and developing the system.

Report Builder

TBD

The report builder implementation consists of one metrics page (CI metrics) having a filter box, status box, and two metrics boxes (Project dashboard and Autotest dashboard). The file structure of the implementation is described in Picture 6 and Picture 7 (complete list of files shown in Table 2 Directories and files).

Database

The database is a MariaDB database, and all tables use MyISAM storage engine. The database is running on the Qt testresults server on the qt.io domain.

PICTURE. Database tables, their fields, and field types

PICTURE. Database tables, their fields, and field types
PICTURE. Database tables, their fields, and field types

Data is parsed from the build log files by default, or from the XML test reports if separately mentioned so in the tables below.

Table Purpose
project Project names, both build projects (currently only the 'Qt5' included) and the parent projects for the testsets (like 'qtbase')
branch Branch name (like '5.5' or 'dev')
state State name (currently only the 'state' builds are included)
platform Target or host platform (like 'linux' / Ubuntu_11.10' / 'x86')
compiler Target or host compiler (like 'g++')
conf Build configuration consisting of target and host platform and compiler, features and full name (like 'linux-g++_shadow-build_Ubuntu_11.10_x86')
phase Phase of the configuration build (like 'configuring Qt')
testset Testset for its parent project (like 'tst_qftp')
testfunction Testfunction for its parent testset (like 'initTestCase'), parsed from XML reports:

<TestFunction name="initTestCase"> <Incident line="0" file="" type="pass"/> <Duration msecs="17.570825"/> </TestFunction>

testrow Testrow for its parent testfunction (like 'WithoutProxy:ok01'), parsed from XML reports:

<TestFunction name="connectToHost"> <Incident line="0" file="" type="pass"> <DataTag> <![CDATA[WithoutProxy:ok01]]> </DataTag> </Incident>...

TABLE. Database tables for 'fixed' data

Table Purpose
project_run Project build data. On most Qt Metrics 2 pages only the Qt5 state build data is shown (unless stated otherwise on the page info)
conf_run Configuration build data for its parent project_run
phase_run Configuration build phase data for its parent conf_run
testset_run Testset result data for its parent conf_run
testfunction_run Testfunction result data for its parent testset_run, parsed from XML reports:

<TestFunction name="initTestCase"> <Incident line="0" file="" type="pass"/> <Duration msecs="17.570825"/> </TestFunction>

testrow_run Testrow result data for its parent testfunction_run, parsed from XML reports:

<TestFunction name="connectToHost"> <Incident line="0" file="" type="pass"> <DataTag> <![CDATA[WithoutProxy:ok01]]> </DataTag> </Incident>...

TABLE. Database tables for build data

Table Purpose
db_status Information on database updates by the parser

TABLE. Database tables for other data

Parser

TBD

(are all methods available?)

New data is added to the database only by the parser. For CI, data is inserted from the new log files when a new build has completed. This method is called a "single scan". The data already in the database is never updated since build results never change after a build. However, if there is a need for example to add a new database field or to change a field type, the whole database must be rebuilt from the log files. This method is referred as a "full scan". There is also a "catch up" method to compare the database against the build log files residing in the server, and to add the possibly missing data. This is needed for example if single scan operations failed for some reason.


Key Functionality

TBD

JavaScript Libraries

TBD

(from https://qt-metrics-jusippol.c9.io/app-design.php)

D3

D3 (Data-Driven Documents) is a JavaScript library for manipulating documents based on data, and helps bringing data to life using HTML, SVG and CSS. The http://d3js.org/ and https://github.com/mbostock/d3/wiki/Gallery include guidance and examples, and because of the popularity of D3 many detailed examples are available in the Internet.

Qt Metrics system uses the library with a CDN from http://d3js.org. A horizontal stacked bar graph is used currently in the Project dashboard level 2 to visualize the build phases.

jQuery UI

jQuery UI is a curated set of user interface interactions, effects, widgets, and themes built on top of the jQuery JavaScript Library.

Twitter Bootstrap



DEVELOPMENT AND MAINTENANCE

This chapter describes a few typical development and maintenance cases as a supplement to the detailed description in the previous chapter. The SW designers who are maintaining and developing the system are the target audience.

Directory Structure and Naming Conventions

TBD

(from https://qt-metrics-jusippol.c9.io/app-design.php)

The list of current directories are shown below.

TABLE. Directories

Directory/File Purpose
(main folder) Common implementation files
images/ Directory for common images like logos and icons

Site map

The site map graph has been made with www.draw.io. The source xml file (site_map.xml) is located in the images directory. To save the image, use export as picture (png). To save the source xml, use export as xml (normal).

Maintaining the Version Info

The report builder version is visible via the About header menu item (about.html) and it shall be updated on every change affecting the UI.

Requests for New Data

New fields should be added to the database only with proper reason and use case. Only the parser adds data into the database. Therefore, if new data fields are needed to the database while specifying or implementing new content of the existing or new metrics boxes or pages, a change request to the parser is needed as well. The change request should include information as to whether the new data is already available, for example in some log files, or whether the change requires a change to the log files.

Submitting Code

The development of the Qt metrics system can be done on any server environment that supports PHP and MySQL/MariaDB. The implemented open source code is made available to the master branch of the qtqa/sysadmin Qt repository in the sysadmin/non-puppet/qtmetrics folder. The submissions follow the common Qt contribution guidelines, including the use of the Gerrit Code Review tool.

Quality Assurance

The following actions should be taken to ensure the quality of the change commits.

Qt coding conventions

Use the Qt Commit Policy, Qt Coding Style, and Qt Coding Conventions where applicable for the PHP, JavaScript, HTML and CSS implementation, plus others if used.

Commits

As a general rule for all Qt contributions, it is important to make atomic commits. That means that each commit should contain exactly one self-contained change. Unrelated changes, or new functionality and error corrections, must not be mixed. Likewise, unrelated small fixes, for example to coding style, should not be hidden in bigger commits, but should be handled in a separate commit.

The Gerrit Early Warning System will make certain checks to the code. It is a good practice to make the check before pushing the changes to Gerrit to avoid unnecessary fix patches. The sanitizer tool, a script made with Perl, can be used for that.

Validating the scripts

It is recommended to follow the HTML5 standard and use the HTML5/CSS3 approach, for example to make the style definitions in the CSS file instead of in the PHP or HTML code. Please utilize the HTML markup validator and the CSS validator by the World Wide Web Consortium (W3C). In addition, use for example JavaScript Lint to validate JavaScript code.

Unit testing

The development of the Qt metrics system can be done using any available web development environment that supports PHP and MySQL/MariaDB, like XAMPP, LAMP, WAMP or Cloud9 cloud IDE for example. PHPUnit shall be used for unit testing the classes (test classes are in src/test directory). For that some sample data shall be stored into MySQL/MariaDB database tables (SQL scripts in src/test directory). New functionality or content may require modifications or additions to the sample data.

Verification on target server

The changes can be verified on the testresults server in a specific verification folder (http://testresults.qt.io/qtmetrics-dev/new) before submitting changed code. The CI team will assist with the access rights.

Verification with different browsers

Verification in all major browsers is essential for any web based service. The Qt metrics system follows responsive design with Twitter Bootstrap to adjust to different screen resolutions. However, it must be noted that, although the pages may utilize large screens, the visible content area should be defined to be as narrow as possible and the most important content to be placed on the left to prevent unnecessary horizontal scrolling.

All changes should be verified at least with the following browsers: Chrome, Firefox, Opera, IE and Safari. Use of different operating systems and browser versions, as well as mobile browsers, is also recommended. It is proposed to list the verified browsers in the Gerrit commit description.

Security and Data Validation

Source data and validation

The source data used in the Qt metrics system is publicly available. The log files for the CI metrics page, for example, can be found in http://testresults.qt.io/ci/. Hence the data itself does not need any user access control.

The parser takes care of data validation before storing data to the database.

Database user access

The report builder of the Qt metrics system uses one user id to read the data from the database. This user has only a read access to the database. The parser uses another user id with write access to store the data to the database. A separate admin user is used for the administration actions, which are authenticated with HTTP basic authentication (see separate chapter).

The database management system cannot be accessed outside the testresults server, where both the report builder and the parser reside.

Web security

TBD



FUTURE DEVELOPMENT

The target is that anyone from the Qt developer community could contribute to the development and improvement of the Qt Metrics page.

Backlog

The development items are listed in the:



SUPPORT

In case of questions, proposals, and improvement ideas, please contact the development team:

  • Juha Sippola (Developer: report builder, database)
  • Tony Sarajärvi (Developer: parser; Use case designer, Key user, Reviewer)
  • Frederik Gladhorn (Use case designer, Key user, Reviewer)
  • Simon Hausman (Key user, Reviewer)
  • Oswald Buddenhagen (Key user, Reviewer)
  • Friedemann Kleint (Key user, Reviewer)