Qt Metrics Page Backlog

From Qt Wiki
Jump to navigation Jump to search

This is the development backlog for Qt Metrics page: http://testresults.qt.io/qtmetrics/

For more detailed description of the Qt Metrics page please see the wiki page: http://wiki.qt.io/Qt_Metrics_Page_Description

In case of proposals or improvement ideas please contact the development team: Juha Sippola: report builder (qt.io/member/137658) Tony Sarajärvi: parser and database (qt.io/member/132527)

See the DONE items after v1.0 at the bottom of the page.

The Backlog

STATUS VERSION PRIORITY TAG PAGE / BOX ITEM CODE REVIEW
in progress *** Usability Common Enable direct URL usage by filtering via URL parameters (e.g. to mail direct link to a specific report). This is a known problem in Ajax implementations.
in progress *** Usability Common Browser Back button does not work with nested metrics (you must use the top links in a box instead). This is related to the direct URL usage item.
idle *** Functionality CI / Graphs CI graphs to visualize CI performance: a) autotest set execution time (to visualize sequential and parallel runs), b) the trend of phase duration (one, or several if feasible) across the last 20 builds in selected Project Configuration. Draft indicators drawn for both of these. Utilize the data already available in the database.
idle *** Functionality Common / Graphs Create improvement progress trend graphs, e.g. on build success rates, build execution times, Configuration and Autotest force success and insignificant flag removals.
idle ** Look'n'feel CI The build history information in the dashboards is currently shown in a table format. A passed/failed % graphics solution could provide additional readability and improved visual appearance, both in the current build histories and in new trend graphs where needed.
idle ** Functionality New Report generator to create a detailed text & graph report from a view. This would be applicable for such data that takes long time to extract and format. The content of the reports must be specified first.
idle ** Functionality CI / Project dashboard Modify the list of projects on level 1 to read only selected Configuration data. Currently the Configuration filter affects only to what projects are listed, not to the data that is shown for the listed projects (the data is from Project level). This is an improvement to the v2.6.
idle ** Functionality CI / Project dashboard Include a mechanism to sort the lists by title (like in Autotest dashboard)
idle ** Functionality CI / Autotest dashboard Have a mechanism to be able to see what went wrong with the autotests, and then assigning a corrective task to the right person to fix it. This needs to be specified in more details (test case metrics provide a partial solution already).
idle ** Functionality New page Add a new metric/page for the build phases data (which CI build machines failed most often and which CI builds and phases take the longest time to execute). This is already supported in the parser and the data is stored into the database. The metric report needs to be specified first. The graph in Project dashboard already touches this a bit.
idle ** Functionality CI / Autotest dashboard Include the bug id’s for test cases. This requires first a database or other solution where to get the data.
idle ** Usability Common Include the mostly needed links to header e.g. as a tag cloud or similar, for example with items like "to whom", "what", "how", "examples", "support", "contribute". This would improve the overall user experience.
idle ** Usability CI / Project dashboard Have the number of shown builds configurable in the build history, instead of the current fixed amount.
idle ** Implementation Common Add Google Analytics to get information on the portal usage.
idle * Functionality Common / Graphs Send graphs by email, or register to a list to receive certain graphs at certain intervals.
idle * Functionality CI / Project dashboard Tick box to select if idle times are included or not in the Build phases graph. This is an improvement to the v2.8.
idle * Functionality CI / Autotest dashboard The number (and list) of projects where autotest (set) is run could be included in the Autotest level 1.
idle * Functionality CI / Autotest dashboard Have the test case passed/failed/skipped summary in autotest dashboard on level 2 (could be calculated e.g. on the level 1 where already read from db and saved to session vars); v2.7 already lists data into build history table as pop-ups but some kind of summary of latest/selected build could be added. This is an improvement to the v2.7.
idle * Look’n’feel CI / Autotest dashboard ‘Official’ icons for info and sorting (see pics directory), and maybe the build log file links in autotest dashboard on level 2 (e.g. from iconmonstr.com).
idle * Security Common Qt metrics page cannot be accessed via https currently
idle * Usability Filters Have a ‘clear the filter’ link/icon in each displayed filter in dashboard filter areas (so that you don't have to select All from the filter box)
idle * Usability Filters Indicate somehow that some filter is active and Clear selections ‘can’ be pressed
idle * Usability CI / Project dashboard Project dashboard to include number of significant and insignificant vs. total autotests in latest Build -> both in 1st (Project list) and 2nd (Conf list) level
idle * Usability CI / Project dashboard Configuration result to the Build phases graph (e.g. to conf title). This is an improvement to the v2.8.
idle * Usability CI / Project dashboard Add tooltips into Build phases graph (e.g phase name and value). This is an improvement to the v2.8.
idle * Usability CI / Project dashboard The Project Build history and Configurations show ‘latest build’ for the previously latest one if there is a new build created during the browsing. User need to refresh to get this in synch again. This is an improvement to the v2.3.
idle * Usability CI / Autotest dashboard Show expected duration to load the xml data in the Autotest dashboard level 2
idle * Usability CI Check log file availability (it may not always be available)
idle * Usability CI Error handling for cases where e.g. Project, Conf or Autotest drops from the database (due to database rebuild); unsure how frequently these kind of situations would happen
idle * Implementation Common / Graphs Allow multiple graph scripts per metrics box (add loop to metricsboxci.php). This is an improvement to the v2.8.

Done Items

Changes after v1.0:

STATUS VERSION PRIORITY TAG PAGE / BOX ITEM CODE REVIEW
DONE **** Implementation Common Changed references with testresults.qt-project.org to testresults.qt.io. link
DONE v2.8 *** Functionality CI Build phases graph to visualize CI performance (project build phases by configurations). Show how long do the builds take to execute, and are the different configurations run effectively in parallel. Utilize the data already available in the database (phases table). link
DONE v2.7 *** Functionality Autotest dashboard Autotest case details: Include the number of reruns in the autotest views. By analyzing the so-called flaky tests, those autotests or test cases that first fail but then pass on second round without any changes, could help to decrease the overall build times. This is already supported in the parser and the database implementation. Purpose is to automate the flaky tests part in the currently manually generated report. In addition, the autotest result history colors (autotest dashboard level 2) could be made more precise with the new data in the database. link
DONE v2.6 *** Filters CI Group Configuration filter values to a Platform filter (the current filters and the links in the dashboards provide the same functionality and are therefore unnecessary duplicate to each other). In addition, when only Configuration (or Platform) filtered, show the Project list for the Conf (or Platform) instead of the current note to select a Project in the Project dashboard. link
DONE v2.5 *** Filters CI Split Project filter to project name (easier filtering for project owners) and to Branch (dev,release,stable) to filter e.g. Project and Autotest list at the 1st level (the current filters and the links in the dashboards provide the same functionality and are therefore unnecessary duplicate to each other). link
DONE v2.4 *** Usability / Performance Autotest dashboard Autotest Failed % calculation performance: The data on level 1 when Project is filtered takes time to calculate. Replace the current approach of reading from the XML files with reading from database. In addition, Failed % on level 1 could be shown also when Project is not filtered to have an overall view to the most failing autotests like originally requested. New database table(s) are needed with changes to the parser. link
DONE v2.3 *** Usability CI / Project dashboard Project dashboard for any build i.e. to show the data (general, builds etc.) for any build (by clicking e.g. the build number); now the shows only the latest build link
DONE v2.2 **** Functionality Autotest dashboard Autotest case results: Detail the results of the autotest sets into test case level, which would make it easier to point out the detailed failure reasons and therefore to get the corrections faster as well. Purpose is to automate the currently manually generated report. link
DONE v2.2 *** Usability CI / Autotest dashboard Autotest dashboard to include number of executed autotests (passed/failed). Specially after per-step precision is implemented. link
DONE v2.1 **** Usability / Performance CI Performance improvements to display the requested information in reasonable time after request (in about 5 to 7 seconds instead of several minutes in the worst case). Currently identified problems are a) initial page loading (reading complete data and saving it into browser session variables), b) page loading during a database rebuild, and c) first read operations after a database update. Possible causes are the speed of the testresults server, speed of the database, rebuild of the DBMS cache after each update and the report builder application logic. Solutions include modifying the database server and its configurations, plus the database parameters and structure, adding database indexing, and improving the report builder application logic. link
DONE v2.0 *** Functionality Common Implement the main menu support for several metrics pages and to identify active and inactive menu items link
DONE v2.0 *** Security Common Store the database login information to a secured file (although this is currently not a problem because the user has only read access to the database, and the login is restricted by the server) link
DONE v1.5.3 *** Implementation Common Ajax and metrics box structure refactoring to make it easier to add new metrics pages and to avoid duplicate code. link
DONE v1.5.1, v1.5 ** Implementation CI / Project dashboard Solve the URL encoding issue with correct PHP/JavaScript functions instead of the current string manipulation link, link
DONE *** Usability Common Automatic frequent database updates from the build log files. See also here. Note the possible negative affects to the performance. link
DONE v1.4 *** Usability CI / Autotest dashboard Sorting based on any column in the list (in level 1 view) link
DONE v1.3 *** Functionality CI / Autotest dashboard Add result history for Project/Conf builds link
DONE v1.2 **** Filters CI Add timescale filter (e.g. since a date) and apply it to Project dashboard and Autotest dashboard link
DONE v1.1.1 *** Usability CI Change the order in which the boxes are loaded on initial page load to indicate the possible database rebuild progress instead of welcome text when loading the page during rebuild link