Qt Metrics 2 Description
- 1 INTRODUCTION
- 2 USING THE SYSTEM
- 3 DETAILED DESCRIPTION
- 4 DEVELOPMENT AND MAINTENANCE
- 5 FUTURE DEVELOPMENT
- 6 SUPPORT
The Qt Metrics web portal visualizes the progress of Qt's quality, focusing on Continuous Integration (CI), code coverage, autotest, release test automation, manual test metrics, and other reports later on. The goal is to automate laborious manual work that is required, for example, in the CI release cycle, to report the key information, as well as providing real time data with good performance. The Qt integration teams (in Digia) and the global Qt developer community are the target audience.
Qt Metrics: http://testresults.qt.io/qtmetrics-dev/new/ (Note: this is a development version)
Version: This document is updated against the report builder v2.8
This document is divided in chapters; each chapter having its own purpose and target audience:
2. Using the System: What (what the users see)
3. Detailed Description: How it works (quick overview to implementation)
4. Developing and Maintenance: How to develop it (how to create new metrics/pages)
The Qt Metrics system consists of three main components: the parser, the report builder and the database. See Picture 1. The parser reads the build logs, test results, or other files that are created in the development process, and pushes the data into the database. The report builder reads the data from the database, and shows predefined metrics and reports where the user is able to scope and filter the information from. The parser is implemented with Perl and the report builder with HTML, CSS, PHP and Ajax with some jQuery UI and D3 graphics components. The database is a MySQL database.
Requirements and Target Audience
USING THE SYSTEM
This section introduces the Qt Metrics system from the user point of view.
Report Builder Overview
List roles incl admin.
This section discusses the Qt Metrics system components and their implementation into a certain level of detail. The target audience is the SW designer(s) maintaining and developing the system.
The report builder implementation consists of one metrics page (CI metrics) having a filter box, status box, and two metrics boxes (Project dashboard and Autotest dashboard). The file structure of the implementation is described in Picture 6 and Picture 7 (complete list of files shown in Table 2 Directories and files).
The database is a MySQL database, and all tables use MyISAM storage engine. The database is running on the Qt testresults server on the qt.io domain.
Database tables and their field types can be seen in the picture below.
PICTURE. Database tables, and their fields and field types
TABLE. Database tables
| Project build results
(ci includes all builds while ci_latest includes only the latest build for each project)
TBD (are all methods available)
New data is added to the database only by the parser. For CI, data is inserted from the new log files when a new build has completed. This method is called a “single scan”. The data already in the database is never updated since build results never change after a build. However, if there is a need for example to add a new database field or to change a field type, the whole database must be rebuilt from the log files. This method is referred as a “full scan”. There is also a “catch up” method to compare the database against the build log files residing in the server, and to add the possibly missing data. This is needed for example if single scan operations failed for some reason.
Qt Metrics system uses the library with a CDN from http://d3js.org. A horizontal stacked bar graph is used currently in the Project dashboard level 2 to visualize the build phases.
DEVELOPMENT AND MAINTENANCE
This chapter describes a few typical development and maintenance cases as a supplement to the detailed description in the previous chapter. The SW designers who are maintaining and developing the system are the target audience.
Directory Structure and Naming Conventions
The list of current directories are shown below.
|(main folder)||Common implementation files|
|commondefinitions.php||Definitions for common flags and values|
|commonfunctions.php||Functions for common use in several metrics pages|
|connect.php||To establish the MySQL connection|
|connectionclose.php||To close the MySQL connection (code currently commented out)|
|connectiondefinitions.php||Database server definitions and flags for MySQL API and connection type|
|index.php||Default file redirecting to metricspageci.php|
|menu.php||Main menu of metrics pages|
|metricspageci.php||CI Metrics page implementation|
|styles.css||Common style definitions|
|calendar/||Calendar component from http://www.triconsole.com/php/calendar_datepicker.php|
|ci/||CI Metrics page implementation files|
|ci/calendar/||Directory for one calendar component image|
|ci/definitions.php||Common definitions used in CI Metrics implementation to tailor views or functionalities|
|ci/functions.php||Common functions used in CI Metrics implementation|
|ci/getautotestvalues.php||To load all the autotest values from database|
|ci/getconfvalues.php||To load all the configuration values from database|
|ci/getdatabasestatus.php||To load the database status, for example the update time, from the database|
|ci/getfilters.php||To layout the filters and call files to load the filter values|
|ci/getprojectvalues.php||To load all the project values from database|
|ci/graphbuildphases.php||To show the build phases graph in project dashboard (level 2)|
|ci/listbuilds.php||To show the project and configuration build history in project dashboard (level 2 and 3)|
|ci/listconfigurations.php||To show the configurations in project dashboard (level 2)|
|ci/listfailingautotests.php||To show the failed autotests in project dashboard (level 2 and 3)|
|ci/listgeneraldata.php||To show the general data for project and configuration build in project dashboard (level 2 and 3)|
|ci/listprojects.php||To show the projects in project dashboard (level 1)|
|ci/metricsboxdefinitions.php||To list the metrics boxes and define their functionality regarding to the filters|
|ci/msgautotestdashboardlevel1.html||Metrics description for autotest dashboard (level 1)|
|ci/msgautotestdashboardlevel2.html||Metrics description for autotest dashboard (level 2)|
|ci/msgautotestresultdescription.html||Description for autotest result history notation used in autotest dashboard (level 2)|
|ci/msgfailuredescription.html||Description for autotest failure categories used in autotest dashboard (level 1 and 2)|
|ci/msgprojectdashboardlevel1.html||Metrics description for project dashboard (level 1)|
|ci/msgprojectdashboardlevel2.html||Metrics description for project dashboard (level 2)|
|ci/msgprojectdashboardlevel3.html||Metrics description for project dashboard (level 3)|
|ci/showautotestdashboard.php||Autotest dashboard metrics box implementation (all sections in the same file)|
|ci/showprojectdashboard.php||Project dashboard metrics box implementation (separate sections implemented in their own files)|
|images/||Directory for common images like logos and icons|
Maintaining the Version Info
The report builder version is visible via the About header menu item (about.html) and it shall be updated on every change affecting the UI.
Requests for New Data
New fields should be added to the database only with proper reason and use case. Only the parser adds data into the database. Therefore, if new data fields are needed to the database while specifying or implementing new content of the existing or new metrics boxes or pages, a change request to the parser is needed as well. The change request should include information as to whether the new data is already available, for example in some log files, or whether the change requires a change to the log files.
The development of the Qt metrics system can be done on any server environment that supports PHP and MySQL. The implemented open source code is made available to the master branch of the qtqa/sysadmin Qt repository in the sysadmin/non-puppet/qtmetrics folder. The submissions follow the common Qt contribution guidelines, including the use of the Gerrit Code Review tool.
The following actions should be taken to ensure the quality of the change commits.
Qt coding conventions
As a general rule for all Qt contributions, it is important to make atomic commits. That means that each commit should contain exactly one self-contained change. Unrelated changes, or new functionality and error corrections, must not be mixed. Likewise, unrelated small fixes, for example to coding style, should not be hidden in bigger commits, but should be handled in a separate commit.
The Gerrit Early Warning System will make certain checks to the code. It is a good practice to make the check before pushing the changes to Gerrit to avoid unnecessary fix patches. The sanitizer tool, a script made with Perl, can be used for that.
Validating the scripts
The development of the Qt metrics system can be done using any available web development environment that supports PHP and MySQL, like XAMPP, LAMP, WAMP or Cloud9 cloud IDE for example. PHPUnit shall be used for unit testing the classes (test classes are in src/test directory). For that some sample data shall be stored into MySQL database tables (MySQL scripts in src/test directory). New functionality or content may require modifications or additions to the sample data.
Verification on target server
The changes can be verified on the testresults server in a specific verification folder (http://testresults.qt.io/qtmetrics-dev/new) before submitting changed code. The CI team will assist with the access rights.
Verification with different browsers
Verification in all major browsers is essential for any web based service. The Qt metrics system follows responsive design with Twitter Bootstrap to adjust to different screen resolutions. However, it must be noted that, although the pages may utilize large screens, the visible content area should be defined to be as narrow as possible and the most important content to be placed on the left to prevent unnecessary horizontal scrolling.
All changes should be verified at least with the following browsers: Chrome, Firefox, Opera, IE and Safari. Use of different operating systems and browser versions, as well as mobile browsers, is also recommended. It is proposed to list the verified browsers in the Gerrit commit description.
Security and Data Validation
Source data and validation
The source data used in the Qt metrics system is publicly available. The log files for the CI metrics page, for example, can be found in http://testresults.qt.io/ci/. Hence the data itself does not need any user access control.
The parser takes care of data validation before storing data to the database.
Database user access
The report builder of the Qt metrics system uses one user id to read the data from the database. This user has only a read access to the database. The parser uses another user id with write access to store the data to the database. A separate admin user is used for the administration actions, which are authenticated with HTTP basic authentication (see separate chapter).
The database management system cannot be accessed outside the testresults server, where both the report builder and the parser reside.
The target is that anyone from the Qt developer community could contribute to the development and improvement of the Qt Metrics page. The development items are listed in the:
- Qt Metrics 2 backlog: Qt Metrics 2 Backlog
In case of questions, proposals, and improvement ideas, please contact the development team:
- Juha Sippola (Developer: report builder, database)
- Tony Sarajärvi (Developer: parser; Use case designer, Key user, Reviewer)
- Frederik Gladhorn (Use case designer, Key user, Reviewer)
- Simon Hausman (Key user, Reviewer)
- Oswald Buddenhagen (Key user, Reviewer)
- Friedemann Kleint (Key user, Reviewer)