QtMetrics How To Read: Difference between revisions

From Qt Wiki
Jump to navigation Jump to search
(Convert ExpressionEngine section headers)
(Images)
Line 6: Line 6:
Below is the main view you receive when you enter the QtMetrics page. The first column shows different builds run. There are runs for all submodules, their master Qt5 (including all the submodules), and most if not all branches for them.
Below is the main view you receive when you enter the QtMetrics page. The first column shows different builds run. There are runs for all submodules, their master Qt5 (including all the submodules), and most if not all branches for them.


[[Image:http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/1-metrics_front_page.png|QtMetrics front page]]
http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/1-metrics_front_page.png


On the left column listing the modules, select Qt5_release_Integration for example
On the left column listing the modules, select Qt5_release_Integration for example


[[Image:http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/2-qt5_release_emphasized.png|Qt5_release emphasized]]
http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/2-qt5_release_emphasized.png


It will open a page like this. Now the left hand part is replaced with all the configurations run inside this job. Meaning, Qt5_release_Integration has been run on all those platforms and configurations.
It will open a page like this. Now the left hand part is replaced with all the configurations run inside this job. Meaning, Qt5_release_Integration has been run on all those platforms and configurations.
[[Image:http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/3-qt5_release_front_page.png|Qt5_release front page]]
 
http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/3-qt5_release_front_page.png


Going through these topics
Going through these topics


[[Image:http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/4-qt5_relese_topics_emphasized.png|Topics zoomed]]
http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/4-qt5_relese_topics_emphasized.png


* First column shows you if your build passed or not. This should be pretty clear. Abandoned means that it has been cancelled for some reason.
* First column shows you if your build passed or not. This should be pretty clear. Abandoned means that it has been cancelled for some reason.
Line 26: Line 27:
Next these ones
Next these ones


[[Image:http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/5-qt5_relese_topics_emphasized2.png|Numbers zoomed]]
http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/5-qt5_relese_topics_emphasized2.png


These numbers simply show you how many tests have failed in this configuration. 'Insignificant' tests marked here are tests that are marked as insignificant '''in the .pro file''' of the test. '''It is not the CI system that has marked these as insignificant'''.
These numbers simply show you how many tests have failed in this configuration. 'Insignificant' tests marked here are tests that are marked as insignificant '''in the .pro file''' of the test. '''It is not the CI system that has marked these as insignificant'''.
Line 33: Line 34:


Below the project view you have the list of all '''failing autotests'''. If we listed all autotests, the page would just bloat.
Below the project view you have the list of all '''failing autotests'''. If we listed all autotests, the page would just bloat.
[[Image:http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/6-qt5_release_autotest_main.png|Autotest main]]
 
http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/6-qt5_release_autotest_main.png


Now this part is important, and you really need to understand this part. Tests can be marked as insignificant at two levels:
Now this part is important, and you really need to understand this part. Tests can be marked as insignificant at two levels:
Line 43: Line 45:


The same thing in QtMetrics. These two topics separate the significant and insignificant tests in the .pro -file level.
The same thing in QtMetrics. These two topics separate the significant and insignificant tests in the .pro -file level.
[[Image:http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/7-qt5_release_autotest_main_topics.png|Autotest topics]]
 
http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/7-qt5_release_autotest_main_topics.png
 
Meaning, the left column shows how many times a test failed that the developer thought should be significant, alas we have a blocking problem here. The right column shows the tests that already the developer knows to be buggy, and are not relevant (There should be a bug report for all these. These tests are broken).
Meaning, the left column shows how many times a test failed that the developer thought should be significant, alas we have a blocking problem here. The right column shows the tests that already the developer knows to be buggy, and are not relevant (There should be a bug report for all these. These tests are broken).


The colored columns
The colored columns
[[Image:http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/8-qt5_release_autotest_main_topics2.png|Autotest topics 2]]
http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/8-qt5_release_autotest_main_topics2.png
explain the situation at the CI level. At start, every configuration is insignificant. But as tests pass once, the flag is removed, and tests start to be significant, and move to the blocking configuration.
explain the situation at the CI level. At start, every configuration is insignificant. But as tests pass once, the flag is removed, and tests start to be significant, and move to the blocking configuration.


Line 60: Line 64:
How to find random failures, not appearing in the last build.
How to find random failures, not appearing in the last build.
On the top of the page you have a time scale filter. Let's move that back a couple of weeks
On the top of the page you have a time scale filter. Let's move that back a couple of weeks
[[Image:http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/9-qt5_release_time_filter.png|Time filter]]
http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/9-qt5_release_time_filter.png
[[Image:http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/10-qt5_release_time_filter_2.png|Changed time filter]]
http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/10-qt5_release_time_filter_2.png


The update of the autotests below takes some time now. The QtMetrics page will go through a lot of XML files to parse for the requested data. But after it is done, you can notice how the numbers increased in this column. It will show you in total numbers how many times the test has failed and how many times it was run during that period. It will also show you a pass rate percentage.
The update of the autotests below takes some time now. The QtMetrics page will go through a lot of XML files to parse for the requested data. But after it is done, you can notice how the numbers increased in this column. It will show you in total numbers how many times the test has failed and how many times it was run during that period. It will also show you a pass rate percentage.
[[Image:http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/11-qt5_release_autotests_more.png|Autotests increased]]
 
http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/11-qt5_release_autotests_more.png
 
You can also find new test cases appearing. These are the really random ones I was referring to. That test didn't fail during the last build, but it did fail once during that time frame we set in the time filter.
You can also find new test cases appearing. These are the really random ones I was referring to. That test didn't fail during the last build, but it did fail once during that time frame we set in the time filter.
[[Image:http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/12-qt5_release_autotests_highlighted.png|Autotests highlighted]]
[[Image:http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/12-qt5_release_autotests_highlighted.png|Autotests highlighted]]


By clicking on the test case, you can open further information about it.
By clicking on the test case, you can open further information about it.
[[Image:http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/13qt5_release_test_case.png|Test case data]]
 
http://i209.photobucket.com/albums/bb139/tsarajar/QtMetrics/13qt5_release_test_case.png
 
There you can see which test function is the failing one, which configuration it failed in, and which build it failed in.
There you can see which test function is the failing one, which configuration it failed in, and which build it failed in.


I hope this page helps. If it still needs some more clarifications, please contact me [http://qt.io/mebmer/132527 tosaraja]
I hope this page helps. If it still needs some more clarifications, please contact me [http://qt.io/mebmer/132527 tosaraja]

Revision as of 11:27, 30 March 2015

This article may require cleanup to meet the Qt Wiki's quality standards. Reason: Auto-imported from ExpressionEngine.
Please improve this article if you can. Remove the {{cleanup}} tag and add this page to Updated pages list after it's clean.

How to read autotest metrics from QtMetrics

The QtMetrics is a powerful tool, but can intimidate the everyday user as it presents you with a lot of numbers and statistics. This is a guide on how you find the relevant autotest data to you.

Below is the main view you receive when you enter the QtMetrics page. The first column shows different builds run. There are runs for all submodules, their master Qt5 (including all the submodules), and most if not all branches for them.

1-metrics_front_page.png

On the left column listing the modules, select Qt5_release_Integration for example

2-qt5_release_emphasized.png

It will open a page like this. Now the left hand part is replaced with all the configurations run inside this job. Meaning, Qt5_release_Integration has been run on all those platforms and configurations.

3-qt5_release_front_page.png

Going through these topics

4-qt5_relese_topics_emphasized.png

  • First column shows you if your build passed or not. This should be pretty clear. Abandoned means that it has been cancelled for some reason.
  • The second column 'forcesuccess' tells you if the configuration has been flagged to forcefully pass even though there might have been a build break or a test failure.
  • The third column is a bit less severe. 'Insignificant' means that tests are flagged as insignificant (at the CI level). This configuration needs to pass the compilation, but autotests may fail.

Neither forcesuccess or insignificant should be enabled. These are problems that need attention!

Next these ones

5-qt5_relese_topics_emphasized2.png

These numbers simply show you how many tests have failed in this configuration. 'Insignificant' tests marked here are tests that are marked as insignificant in the .pro file of the test. It is not the CI system that has marked these as insignificant.

Let's look closer at the autotests (further down the page).

Below the project view you have the list of all failing autotests. If we listed all autotests, the page would just bloat.

6-qt5_release_autotest_main.png

Now this part is important, and you really need to understand this part. Tests can be marked as insignificant at two levels:

  • In the .pro file, usually marked by the developer
  • In the CI configuration, usually marked by the CI team

If the test is marked insignificant in the .pro file, it will only mark the tests inside that tst_ -set as insignificant, and nothing else. If the CI configuration is marked as insignificant, all the tests from that build configuration (e.g. linux-g++_static_Ubuntu_12.04_x64) are insignificant. It's important to get rid of all these configurations, because no regression can be caught in that configuration as long as tests are marked as insignificant. Every time a new platform or configuration is introduced to the CI, it is initially marked as insignificant at this level so that it doesn't block anything on the go and developers get time to fix these.

The same thing in QtMetrics. These two topics separate the significant and insignificant tests in the .pro -file level.

7-qt5_release_autotest_main_topics.png

Meaning, the left column shows how many times a test failed that the developer thought should be significant, alas we have a blocking problem here. The right column shows the tests that already the developer knows to be buggy, and are not relevant (There should be a bug report for all these. These tests are broken).

The colored columns 8-qt5_release_autotest_main_topics2.png explain the situation at the CI level. At start, every configuration is insignificant. But as tests pass once, the flag is removed, and tests start to be significant, and move to the blocking configuration.

The colors explain it too. The most fatal one is the darkest red. Whenever both the developer and the CI thinks, that a tests should pass, but didn't, it fails the build and the commit doesn't go through.

The second red column contains the situations where the developer thinks that his test is ok, but the CI is still marking that configuration as insignificant. These are usually remains that should be cleaned. Corrective action would be to go mark the test insignificant in the .pro file, so that the whole configuration could be marked as significant.

The orange columns contain the status of tests that have been marked insignificant already by the developer, as explained earlier. They are just split up accordingly to the CI configuration as well.

If I lost you now, I recommend you read the above a second time, or a third, and if that doesn't help, contact me tosaraja for further help. That way I also know I need to explain this further.

How to find random failures, not appearing in the last build. On the top of the page you have a time scale filter. Let's move that back a couple of weeks 9-qt5_release_time_filter.png 10-qt5_release_time_filter_2.png

The update of the autotests below takes some time now. The QtMetrics page will go through a lot of XML files to parse for the requested data. But after it is done, you can notice how the numbers increased in this column. It will show you in total numbers how many times the test has failed and how many times it was run during that period. It will also show you a pass rate percentage.

11-qt5_release_autotests_more.png

You can also find new test cases appearing. These are the really random ones I was referring to. That test didn't fail during the last build, but it did fail once during that time frame we set in the time filter. Autotests highlighted

By clicking on the test case, you can open further information about it.

13qt5_release_test_case.png

There you can see which test function is the failing one, which configuration it failed in, and which build it failed in.

I hope this page helps. If it still needs some more clarifications, please contact me tosaraja