Sanity-Test-Guidelines: Difference between revisions

From Qt Wiki
Jump to navigation Jump to search
No edit summary
(Formatting)
 
(5 intermediate revisions by 2 users not shown)
Line 1: Line 1:
[[Category:Release::Testing]]
[[Category:Release::Testing]]


[toc align_right="yes" depth="2"]


= Sanity Test =
= Sanity Test =
Line 7: Line 7:
Sanity Test is a part of the releasing cycle. The purpose is to ensure that the Qt SDK is ready for further testing.
Sanity Test is a part of the releasing cycle. The purpose is to ensure that the Qt SDK is ready for further testing.


{color:blue}'''RTA work ongoing:'''<br />Part of the Sanity test cases are covered by Release test automation (RTA). See more information on the {color:blue}'''RTA''' (covered by RTA) and {color:red}'''RTA''' (not yet covered by RTA) sections below.
<span style="color:blue">'''RTA work ongoing:''' </span>
Part of the Sanity test cases are covered by Release test automation (RTA). See more information on the <span style="color:blue">'''RTA'''</span> (covered by RTA) and <span style="color:red">'''RTA'''</span> (not yet covered by RTA) sections below.


== Build checks ==
== Build checks ==
Line 13: Line 14:
* '''Asks about license'''
* '''Asks about license'''


{color:blue}'''RTA:''' RTA verifies that license key is asked while running the configure (valid license key, invalid license key). When the installer is run the installing with a license key is verified by RTA.
<span style="color:blue">'''RTA:'''</span> RTA verifies that license key is asked while running the configure (valid license key, invalid license key). When the installer is run the installing with a license key is verified by RTA.


* '''Compiles with minimal options (eg. -opensource -confirm-license)'''<br />Run configure command with the following options<br /><code><br />configure -developer-build -opensource -nomake examples -nomake tests -confirm-license<br /></code><br />{color:blue}'''RTA''' Linux, Mac, Windows: covered by RTA
* '''Compiles with minimal options (eg. -opensource -confirm-license)'''
Run configure command with the following options
<code>
configure -developer-build -opensource -nomake examples -nomake tests -confirm-license
</code>
<span style="color:blue">'''RTA'''</span> Linux, Mac, Windows: covered by RTA


* '''Compiles as static build (where supported)'''<br />Run configure command with '-static':<br /><code><br />configure -static …<br /></code><br />{color:blue}'''RTA''' Linux, Mac, Windows: covered by RTA
* '''Compiles as static build (where supported)'''
Run configure command with '-static':
<code>
configure -static …
</code>
<span style="color:blue">'''RTA'''</span> Linux, Mac, Windows: covered by RTA


* '''Compiles in namespace (where supported)'''<br />Run configure command with '-qtnamespace &lt;namespace&amp;gt;':<br /><code><br />configure -qtnamespace CustomName …<br /></code><br />{color:blue}'''RTA''' Linux, Mac, Windows: covered by RTA
* '''Compiles in namespace (where supported)'''
Run configure command with '-qtnamespace <namespace>':
<code>
configure -qtnamespace CustomName …
</code>
<span style="color:blue">'''RTA'''</span> Linux, Mac, Windows: covered by RTA


* '''Compiles with shadow build'''<br />Create a new folder. cd to the new folder. Run the configure command from the source directory.<br />{color:blue}'''RTA''' Linux, Mac, Windows: covered by RTA
* '''Compiles with shadow build'''
Create a new folder. cd to the new folder. Run the configure command from the source directory.
<span style="color:blue">'''RTA'''</span> Linux, Mac, Windows: covered by RTA


* '''Compiles cross-compiled'''<br />Use -xplatform parameter, e.g.<br /><code><br />configure -embedded arm -xplatform qws/linux-arm-g++ …<br /></code>
* '''Compiles cross-compiled'''
Use -xplatform parameter, e.g.
<code>
configure -embedded arm -xplatform qws/linux-arm-g++ …
</code>


== General checks ==
== General checks ==


* '''Text files have the correct EOL'''<br />Open a few text files (README, .cpp, .h, .pro, etc.) and check if they look OK.
* '''Text files have the correct EOL'''
Open a few text files (README, .cpp, .h, .pro, etc.) and check if they look OK.
* '''Files/dirs have sane perm. and timestamps'''
* '''Files/dirs have sane perm. and timestamps'''
* '''Tags have been replaced properly'''<br />The list of tags: qmake -query (???)
* '''Tags have been replaced properly'''
* '''README has valid information'''<br />Complete information on how to build Qt from sources should be given.
The list of tags: qmake -query (???)
* '''README has valid information'''
Complete information on how to build Qt from sources should be given.


== Installer checks ==
== Installer checks ==


* '''Right version of Qt Installer Framework is used'''<br />Launch the installer with '—version' . For Qt 5.4 -beta it should show 'IFW Version: &quot;1.5.82&amp;quot;, Installer base SHA1: &quot;f67db7e&amp;quot;,<br />{color:blue}'''RTA:''' Linux, Mac covered by RTA.<br />{color:red}'''RTA:''' Windows: Manual testing required.
* '''Right version of Qt Installer Framework is used'''
Launch the installer with '—version' . For Qt 5.4 -beta it should show 'IFW Version: "1.5.82", Installer base SHA1: "f67db7e",
<span style="color:blue">'''RTA:'''</span> Linux, Mac covered by RTA.
<span style="color:red">'''RTA:'''</span> Windows: Manual testing required.


* '''Fresh install works correctly'''<br />Install Qt SDK on a newly installed OS.
* '''Fresh install works correctly'''
Install Qt SDK on a newly installed OS.


{color:blue}'''RTA:''' RTA verifies that clean installation works ok. Clean means that the installation directory does not contain any other Qt version.<br />{color:red}'''RTA:''' Whole OS is not installed as a fresh installation during the RTA. Study how to do this added into CI's ta
<span style="color:blue">'''RTA:'''</span> RTA verifies that clean installation works ok. Clean means that the installation directory does not contain any other Qt version.
<span style="color:red">'''RTA:'''</span> Whole OS is not installed as a fresh installation during the RTA. Study how to do this added into CI's ta


* '''Fresh install to place requiring elevated rights works correctly'''<br />Install Qt SDK as a normal user to a place requiring elevated rights. The installer should allow this by asking for elevated rights / the sudo password.
* '''Fresh install to place requiring elevated rights works correctly'''
Install Qt SDK as a normal user to a place requiring elevated rights. The installer should allow this by asking for elevated rights / the sudo password.


* '''Upgrade install works correctly'''<br />Install higher version of Qt having older one, e.g. install Qt 5.3.2 on Qt 5.2.0.
* '''Upgrade install works correctly'''
Install higher version of Qt having older one, e.g. install Qt 5.3.2 on Qt 5.2.0.


* '''Parallel install works correctly'''<br />Install two Qt installation in parallel, e.g. Qt 4.8.6 and 5.3.2. Check that both packages work
* '''Parallel install works correctly'''
* '''Warns when installing over an installation'''<br />Run the installer the second time. A warning pop-up windows should be displayed.
Install two Qt installation in parallel, e.g. Qt 4.8.6 and 5.3.2. Check that both packages work
* '''Warns when installing over an installation'''
Run the installer the second time. A warning pop-up windows should be displayed.


* '''Is correctly signed (vendor/certificate, not untrusted)'''<br />Check where applicable.
* '''Is correctly signed (vendor/certificate, not untrusted)'''
Check where applicable.


* '''Displays appropriate gfx &amp; strings &amp; version no.'''<br />This check is about general look and correct package version number.
* '''Displays appropriate gfx & strings & version no.'''
This check is about general look and correct package version number.


* '''Offers the correct license(s)'''
* '''Offers the correct license(s)'''


{color:blue}'''RTA:''' RTA verifies that installation works with the valid license key.<br />{color:red}'''RTA:''' Visual verification of the installer and installing with the invalid license keys needs to be done manually.
<span style="color:blue">'''RTA:'''</span> RTA verifies that installation works with the valid license key.
<span style="color:blue">'''RTA:'''</span> Visual verification of the installer and installing with the invalid license keys needs to be done manually.


* '''Offers sane default install directory (w/version number)'''<br />Check of the default folder is similar to ../Qt5.3.2/<br />{color:blue}'''RTA:''' RTA verifies the default directory.
* '''Offers sane default install directory (w/version number)'''
* '''Correctly installs to default directory'''<br />{color:blue}'''RTA:''' RTA verifies installing into the default directory.
Check of the default folder is similar to ../Qt5.3.2/
* '''Correctly installs to non-default loc.'''<br />Change the target folder during installation from ../Qt5.3.2/ to a different name.<br />{color:blue}'''RTA:''' RTA verifies installing into the non-default directory.
<span style="color:blue">'''RTA:'''</span> RTA verifies the default directory.
* '''Sanely reports progress and completion'''<br />Click on the Details button to see the progress. Wait for successful completion of the installation process.
* '''Correctly installs to default directory'''
<span style="color:blue">'''RTA:'''</span> RTA verifies installing into the default directory.
* '''Correctly installs to non-default loc.'''
Change the target folder during installation from ../Qt5.3.2/ to a different name.
<span style="color:blue">'''RTA:'''</span> RTA verifies installing into the non-default directory.
* '''Sanely reports progress and completion'''
Click on the Details button to see the progress. Wait for successful completion of the installation process.


* '''Installs only selected components'''<br />Try to use Deselect All button, and select a few additional components. Component selector should works sanely. After installation is done, only selected components should be in the target folder.<br />{color:blue}'''RTA:''' RTA verifies that only selected components are installed.
* '''Installs only selected components'''
* '''Shortcuts from last page of installer works'''<br />Select/deselect two check boxes (to run Qt Creator and see notes) on the final screen.
Try to use Deselect All button, and select a few additional components. Component selector should works sanely. After installation is done, only selected components should be in the target folder.
<span style="color:blue">'''RTA:'''</span> RTA verifies that only selected components are installed.
* '''Shortcuts from last page of installer works'''
Select/deselect two check boxes (to run Qt Creator and see notes) on the final screen.
* '''Correctly creates Desktop shortcuts'''
* '''Correctly creates Desktop shortcuts'''
* '''Correctly creates Start Menu shortcuts'''<br />Check if Qt Creator can be found by the OS (start menu, launcher, dash home, etc.). Check the icon.
* '''Correctly creates Start Menu shortcuts'''
Check if Qt Creator can be found by the OS (start menu, launcher, dash home, etc.). Check the icon.
* '''Sets correct environment variables for shell'''
* '''Sets correct environment variables for shell'''
* '''Package shows up in Control Panel/Pkg. Manager'''
* '''Package shows up in Control Panel/Pkg. Manager'''
* '''Patching of files is done correctly'''
* '''Patching of files is done correctly'''


* '''Cancel button is available'''<br />Use cancel button. Check if there are no changes done by the installer when the installation is interrupted.<br />{color:blue}'''RTA:''' RTA verifies that the installation can be cancelled. Note: RTA does not support cancelling the installing after the 'install' button is pressed i.e. while the installing of the components is ongoing.
* '''Cancel button is available'''
Use cancel button. Check if there are no changes done by the installer when the installation is interrupted.
<span style="color:blue">'''RTA:'''</span> RTA verifies that the installation can be cancelled. Note: RTA does not support cancelling the installing after the 'install' button is pressed i.e. while the installing of the components is ongoing.
* '''Cleans up after install failure'''
* '''Cleans up after install failure'''


Line 75: Line 124:
== Uninstaller checks ==
== Uninstaller checks ==


* '''Uninstalls properly'''<br />{color:blue}'''RTA:''' RTA verifies that uninstalling is done properly.
* '''Uninstalls properly'''
* '''Removes installed files properly'''<br />{color:blue}'''RTA:''' RTA verifies that all installed files are removed.
<span style="color:blue">'''RTA:'''</span> RTA verifies that uninstalling is done properly.
* '''Removes empty directories'''<br />{color:blue}'''RTA:''' RTA verifies that the installation directory is removed.
* '''Removes installed files properly'''
<span style="color:blue">'''RTA:'''</span> RTA verifies that all installed files are removed.
* '''Removes empty directories'''
<span style="color:blue">'''RTA:'''</span> RTA verifies that the installation directory is removed.
* '''Removes registry keys'''
* '''Removes registry keys'''
* '''Reverses any other changes'''
* '''Reverses any other changes'''
Line 83: Line 135:
== Both source/binary package checks ==
== Both source/binary package checks ==


* '''License is correct'''<br />{color:blue}'''RTA:''' RTA verifies that proper license files are found from $QTDIR\Licenses directory.
* '''License is correct'''
* '''Assistant works correctly'''<br />Run Assistant (e.g. ./5.0.2/gcc_64/bin/assistant). Check the basic functionality, browse through, search for documentation.<br />{color:blue}'''RTA:''' RTA verifies that Assistant can be opened &amp; closed, browsing and searching works.
<span style="color:blue">'''RTA:'''</span> RTA verifies that proper license files are found from $QTDIR\Licenses directory.
* '''Assistant works correctly'''
Run Assistant (e.g. ./5.0.2/gcc_64/bin/assistant). Check the basic functionality, browse through, search for documentation.
<span style="color:blue">'''RTA:'''</span> RTA verifies that Assistant can be opened & closed, browsing and searching works.
* '''Designer works correctly'''
* '''Designer works correctly'''
* '''Qt Creator works correctly'''<br />Qt Creator can be tested using the following test cases: &quot;Qt Creator Tests:&quot;:http://wiki.qt.io/Category:Tools::QtCreator::QualityAssurance
* '''Qt Creator works correctly'''
* '''Demos and examples launch without crashing'''<br />{color:blue}'''Each example''' should be run with no error.<br />{color:blue}'''RTA:''' RTA launches and closes each Qt example (except console examples).
Qt Creator can be tested using the following test cases: [http://wiki.qt.io/Category:Tools::QtCreator::QualityAssurance Qt Creator Tests:]
* '''Demos and examples launch without crashing'''
<span style="color:blue">'''Each example'''</span> should be run with no error.
<span style="color:blue">'''RTA:'''</span> RTA launches and closes each Qt example (except console examples).
* '''Demos and examples function acceptably'''
* '''Demos and examples function acceptably'''


'''RTA:''' Sanity level testing; see the QtModule sections below.
'''RTA:''' Sanity level testing; see the QtModule sections below.


* '''Demos and examples can be rebuilt'''<br />{color:blue}'''RTA:''' RTA compiles the examples and demos from /Src and /bin directories.
* '''Demos and examples can be rebuilt'''
* '''External apps build (Qt Creator and other applications)'''<br />Create your own application, compile it and run.
<span style="color:blue">'''RTA:'''</span> RTA compiles the examples and demos from /Src and /bin directories.
* '''External apps build (Qt Creator and other applications)'''
Create your own application, compile it and run.


* '''&quot;DLL Swapping&amp;quot; work on an application'''
* '''"DLL Swapping" work on an application'''


* '''GUI stress-testing works ok'''<br />Try to perform many keystrokes, mouse clicks, etc.
* '''GUI stress-testing works ok'''
Try to perform many keystrokes, mouse clicks, etc.
* '''Audio/Video w/QtMultimedia works'''
* '''Audio/Video w/QtMultimedia works'''
* '''Raster engine works'''<br />Run ./qtbase/examples/gui/rasterwindow example.
* '''Raster engine works'''
Run ./qtbase/examples/gui/rasterwindow example.
* '''Image formats work'''
* '''Image formats work'''


* '''GraphicsView works'''
* '''GraphicsView works'''
* '''OpenGL works'''<br />Run ./qtbase/examples/gui/openglwindow example.<br />This is also checked by most of the other examples.
* '''OpenGL works'''
Run ./qtbase/examples/gui/openglwindow example.
This is also checked by most of the other examples.
* '''Printing works'''
* '''Printing works'''
* '''QML 1 apps work (qmlviewer)'''<br />{color:blue}'''RTA:''' RTA runs few QtQML1 examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.
* '''QML 1 apps work (qmlviewer)'''
* '''QML 2 apps work (qmlscene)'''<br />{color:blue}'''RTA:''' RTA runs few QtQML2 examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.
<span style="color:blue">'''RTA:'''</span> RTA runs few QtQML1 examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.
* '''QtNetwork works'''<br />{color:blue}'''RTA:''' RTA runs few QtNetwork examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.
* '''QML 2 apps work (qmlscene)'''
* '''QtSql works'''<br />{color:blue}'''RTA:''' RTA runs few QtSql examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.
<span style="color:blue">'''RTA:'''</span> RTA runs few QtQML2 examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.
* '''QtSvg works'''<br />{color:blue}'''RTA:''' RTA runs few QtSvg examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.
* '''QtNetwork works'''
* '''QtXml works'''<br />{color:blue}'''RTA:''' RTA runs few QtXML examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.
<span style="color:blue">'''RTA:'''</span> RTA runs few QtNetwork examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.
* '''QtSql works'''
<span style="color:blue">'''RTA:'''</span> RTA runs few QtSql examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.
* '''QtSvg works'''
<span style="color:blue">'''RTA:'''</span> RTA runs few QtSvg examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.
* '''QtXml works'''
<span style="color:blue">'''RTA:'''</span> RTA runs few QtXML examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.


* '''QtWebEngine works'''
* '''QtWebEngine works'''
Line 118: Line 188:
* '''QtWebkit 2 works'''
* '''QtWebkit 2 works'''


* '''Qt Script 2 works'''<br />Run ./qtscript/examples/script/qstetrix example. Use Debug option. context2d can be also run.<br />{color:blue}'''RTA:''' RTA runs few QtScript examples and verifies that some detailed level Qt functionality can be used. Example ./qtscript/examples/script/qstetrix is included into the testing. They are launched via the command prompt i.e. not from the Qt Creator.
* '''Qt Script 2 works'''
Run ./qtscript/examples/script/qstetrix example. Use Debug option. context2d can be also run.
<span style="color:blue">'''RTA:'''</span> RTA runs few QtScript examples and verifies that some detailed level Qt functionality can be used. Example ./qtscript/examples/script/qstetrix is included into the testing. They are launched via the command prompt i.e. not from the Qt Creator.


== Qt Creator Checks ==
== Qt Creator Checks ==


* '''Right Qt Creator version is installed'''<br />Check Qt Creator version in About dialog.
* '''Right Qt Creator version is installed'''
Check Qt Creator version in About dialog.


{color:blue}'''RTA:''' Linux, Mac: Qt Creator version is verified from the About dialog.<br />{color:red}'''RTA:''' Windows: not yet part of RTA (some testing tool issues; work ongoing to solve this). Manual testing required.
<span style="color:blue">'''RTA:'''</span> Linux, Mac: Qt Creator version is verified from the About dialog.
<span style="color:red">'''RTA:'''</span> Windows: not yet part of RTA (some testing tool issues; work ongoing to solve this). Manual testing required.


* &quot;Qt Creator Tests:&quot;:http://wiki.qt.io/Category:Tools::QtCreator::QualityAssurance
* [http://wiki.qt.io/Category:Tools::QtCreator::QualityAssurance Qt Creator Tests:]


{color:blue}'''RTA:''' Linux, Mac: Qt Creator is opened and closed in RTA tests.<br />{color:red}'''RTA:''' Windows: Qt Creator open &amp; close not yet covered by RTA (some testing tool issues; work ongoing to solve this). In general, other test cases except open&amp;amp;close are not yet part of RTA. Manual testing required.
<span style="color:blue">'''RTA:'''</span> Linux, Mac: Qt Creator is opened and closed in RTA tests.
<span style="color:red">'''RTA:'''</span> Windows: Qt Creator open & close not yet covered by RTA (some testing tool issues; work ongoing to solve this). In general, other test cases except open&close are not yet part of RTA. Manual testing required.


== CMake Checks ==
== CMake Checks ==
Line 140: Line 215:
QT_INSTALLATION_PREFIX=$PWD/gcc_64
QT_INSTALLATION_PREFIX=$PWD/gcc_64


for d in `find . -name &quot;cmake.pro&amp;quot;`; do<br /> b=`dirname $d`
for d in `find . -name "cmake.pro"`; do
b=`dirname $d`


m=`dirname $b`<br /> m=`dirname $m`<br /> m=`dirname $m`<br /> m=`basename $m`<br /> if [ x$m != x&amp;quot;qtactiveqt&amp;quot; ]; then<br /> pushd $b &gt; /dev/null<br /> rm -rf build<br /> $QT_INSTALLATION_PREFIX/bin/qmake<br /> make check<br /> popd &gt; /dev/null<br /> fi<br />done
m=`dirname $b`
m=`dirname $m`
m=`dirname $m`
m=`basename $m`
if [ x$m != x"qtactiveqt" ]; then
pushd $b > /dev/null
rm -rf build
$QT_INSTALLATION_PREFIX/bin/qmake
make check
popd > /dev/null
fi
done


</code>
</code>
Line 150: Line 237:
<code>
<code>


set QT_INSTALLATION_PREFIX=&amp;#37;cd:&amp;#37;/msvc2010<br />set scriptbase=&amp;#37;cd&amp;amp;#37;<br />FOR &amp;#37;&amp;#37;? IN (qtactiveqt qtbase qtdeclarative qtmultimedia qtquick1 qtscript qtsensors qtserialport qtsvg qttools qtxmlpatterns) DO (<br /> cd Srcamp;#37;&amp;#37;?rd /S /Q build<br /> &amp;#37;QT_INSTALLATION_PREFIX&amp;amp;#37;/bin/qmake<br /> nmake check<br /> cd &amp;#37;scriptbase&amp;amp;#37;<br />)<br />cd Src\qtwebkit\Source\WebKit\qt\tests\cmake<br />rd /S /Q build<br />&amp;#37;QT_INSTALLATION_PREFIX&amp;amp;#37;/bin/qmake<br />nmake check<br />cd &amp;#37;scriptbase&amp;amp;#37;
set QT_INSTALLATION_PREFIX=%cd:%/msvc2010
set scriptbase=%cd%
FOR %%? IN (qtactiveqt qtbase qtdeclarative qtmultimedia qtquick1 qtscript qtsensors qtserialport qtsvg qttools qtxmlpatterns) DO (
cd Srcamp;#37;%?rd /S /Q build
%QT_INSTALLATION_PREFIX%/bin/qmake
nmake check
cd %scriptbase%
)
cd Src\qtwebkit\Source\WebKit\qt\tests\cmake
rd /S /Q build
%QT_INSTALLATION_PREFIX%/bin/qmake
nmake check
cd %scriptbase%


</code>
</code>
Line 158: Line 257:
<code>
<code>


set QT_INSTALLATION_PREFIX=&amp;#37;cd:&amp;#37;/mingw48_32<br />set scriptbase=&amp;#37;cd&amp;amp;#37;<br />FOR &amp;#37;&amp;#37;? IN (qtactiveqt qtbase qtdeclarative qtmultimedia qtquick1 qtscript qtsensors qtserialport qtsvg qttools qtxmlpatterns) DO (<br /> cd Srcamp;#37;&amp;#37;?rd /S /Q build<br /> &amp;#37;QT_INSTALLATION_PREFIX&amp;amp;#37;/bin/qmake<br /> mingw32-make check<br /> cd &amp;#37;scriptbase&amp;amp;#37;<br />)<br />cd Src\qtwebkit\Source\WebKit\qt\tests\cmake<br />rd /S /Q build<br />&amp;#37;QT_INSTALLATION_PREFIX&amp;amp;#37;/bin/qmake<br />mingw32-make check
set QT_INSTALLATION_PREFIX=%cd:%/mingw48_32
set scriptbase=%cd%
FOR %%? IN (qtactiveqt qtbase qtdeclarative qtmultimedia qtquick1 qtscript qtsensors qtserialport qtsvg qttools qtxmlpatterns) DO (
cd Srcamp;#37;%?rd /S /Q build
%QT_INSTALLATION_PREFIX%/bin/qmake
mingw32-make check
cd %scriptbase%
)
cd Src\qtwebkit\Source\WebKit\qt\tests\cmake
rd /S /Q build
%QT_INSTALLATION_PREFIX%/bin/qmake
mingw32-make check


</code>
</code>
Line 166: Line 276:
= Links =
= Links =


* &quot;Building Qt 5 from Git&amp;quot;:http://wiki.qt.io/Building-Qt-5-from-Git
* [http://wiki.qt.io/Building-Qt-5-from-Git Building Qt 5 from Git]
* &quot;Building static Qt on Windows&amp;quot;:http://www.qtcentre.org/wiki/index.php?title=Building_static_Qt_on_Windows
* [http://www.qtcentre.org/wiki/index.php?title=Building_static_Qt_on_Windows Building static Qt on Windows]
* &quot;What is the usage of the qtlibinfix and qtnamespace configuration options?&quot;:http://qt.io/faq/answer/what_is_the_usage_of_the_qtlibinfix_and_qtnamespace_configuration_options
* [http://qt.io/faq/answer/what_is_the_usage_of_the_qtlibinfix_and_qtnamespace_configuration_options What is the usage of the qtlibinfix and qtnamespace configuration options?]
* &quot;Bugtracker&amp;quot;:https://bugreports.qt.io/secure/Dashboard.jspa
* [https://bugreports.qt.io/secure/Dashboard.jspa Bugtracker]

Latest revision as of 07:49, 15 April 2015


Sanity Test

Sanity Test is a part of the releasing cycle. The purpose is to ensure that the Qt SDK is ready for further testing.

RTA work ongoing: Part of the Sanity test cases are covered by Release test automation (RTA). See more information on the RTA (covered by RTA) and RTA (not yet covered by RTA) sections below.

Build checks

  • Asks about license

RTA: RTA verifies that license key is asked while running the configure (valid license key, invalid license key). When the installer is run the installing with a license key is verified by RTA.

  • Compiles with minimal options (eg. -opensource -confirm-license)

Run configure command with the following options 

configure -developer-build -opensource -nomake examples -nomake tests -confirm-license

RTA Linux, Mac, Windows: covered by RTA

  • Compiles as static build (where supported)

Run configure command with '-static': 

configure -static …

RTA Linux, Mac, Windows: covered by RTA

  • Compiles in namespace (where supported)

Run configure command with '-qtnamespace <namespace>': 

configure -qtnamespace CustomName …

RTA Linux, Mac, Windows: covered by RTA

  • Compiles with shadow build

Create a new folder. cd to the new folder. Run the configure command from the source directory. RTA Linux, Mac, Windows: covered by RTA

  • Compiles cross-compiled

Use -xplatform parameter, e.g. 

configure -embedded arm -xplatform qws/linux-arm-g++ …

General checks

  • Text files have the correct EOL

Open a few text files (README, .cpp, .h, .pro, etc.) and check if they look OK.

  • Files/dirs have sane perm. and timestamps
  • Tags have been replaced properly

The list of tags: qmake -query (???)

  • README has valid information

Complete information on how to build Qt from sources should be given.

Installer checks

  • Right version of Qt Installer Framework is used

Launch the installer with '—version' . For Qt 5.4 -beta it should show 'IFW Version: "1.5.82", Installer base SHA1: "f67db7e", RTA: Linux, Mac covered by RTA. RTA: Windows: Manual testing required.

  • Fresh install works correctly

Install Qt SDK on a newly installed OS.

RTA: RTA verifies that clean installation works ok. Clean means that the installation directory does not contain any other Qt version. RTA: Whole OS is not installed as a fresh installation during the RTA. Study how to do this added into CI's ta

  • Fresh install to place requiring elevated rights works correctly

Install Qt SDK as a normal user to a place requiring elevated rights. The installer should allow this by asking for elevated rights / the sudo password.

  • Upgrade install works correctly

Install higher version of Qt having older one, e.g. install Qt 5.3.2 on Qt 5.2.0.

  • Parallel install works correctly

Install two Qt installation in parallel, e.g. Qt 4.8.6 and 5.3.2. Check that both packages work

  • Warns when installing over an installation

Run the installer the second time. A warning pop-up windows should be displayed.

  • Is correctly signed (vendor/certificate, not untrusted)

Check where applicable.

  • Displays appropriate gfx & strings & version no.

This check is about general look and correct package version number.

  • Offers the correct license(s)

RTA: RTA verifies that installation works with the valid license key. RTA: Visual verification of the installer and installing with the invalid license keys needs to be done manually.

  • Offers sane default install directory (w/version number)

Check of the default folder is similar to ../Qt5.3.2/ RTA: RTA verifies the default directory.

  • Correctly installs to default directory

RTA: RTA verifies installing into the default directory.

  • Correctly installs to non-default loc.

Change the target folder during installation from ../Qt5.3.2/ to a different name. RTA: RTA verifies installing into the non-default directory.

  • Sanely reports progress and completion

Click on the Details button to see the progress. Wait for successful completion of the installation process.

  • Installs only selected components

Try to use Deselect All button, and select a few additional components. Component selector should works sanely. After installation is done, only selected components should be in the target folder. RTA: RTA verifies that only selected components are installed.

  • Shortcuts from last page of installer works

Select/deselect two check boxes (to run Qt Creator and see notes) on the final screen.

  • Correctly creates Desktop shortcuts
  • Correctly creates Start Menu shortcuts

Check if Qt Creator can be found by the OS (start menu, launcher, dash home, etc.). Check the icon.

  • Sets correct environment variables for shell
  • Package shows up in Control Panel/Pkg. Manager
  • Patching of files is done correctly
  • Cancel button is available

Use cancel button. Check if there are no changes done by the installer when the installation is interrupted. RTA: RTA verifies that the installation can be cancelled. Note: RTA does not support cancelling the installing after the 'install' button is pressed i.e. while the installing of the components is ongoing.

  • Cleans up after install failure
  • Cleanly fails on insufficient disk space

Uninstaller checks

  • Uninstalls properly

RTA: RTA verifies that uninstalling is done properly.

  • Removes installed files properly

RTA: RTA verifies that all installed files are removed.

  • Removes empty directories

RTA: RTA verifies that the installation directory is removed.

  • Removes registry keys
  • Reverses any other changes

Both source/binary package checks

  • License is correct

RTA: RTA verifies that proper license files are found from $QTDIR\Licenses directory.

  • Assistant works correctly

Run Assistant (e.g. ./5.0.2/gcc_64/bin/assistant). Check the basic functionality, browse through, search for documentation. RTA: RTA verifies that Assistant can be opened & closed, browsing and searching works.

  • Designer works correctly
  • Qt Creator works correctly

Qt Creator can be tested using the following test cases: Qt Creator Tests:

  • Demos and examples launch without crashing

Each example should be run with no error. RTA: RTA launches and closes each Qt example (except console examples).

  • Demos and examples function acceptably

RTA: Sanity level testing; see the QtModule sections below.

  • Demos and examples can be rebuilt

RTA: RTA compiles the examples and demos from /Src and /bin directories.

  • External apps build (Qt Creator and other applications)

Create your own application, compile it and run.

  • "DLL Swapping" work on an application
  • GUI stress-testing works ok

Try to perform many keystrokes, mouse clicks, etc.

  • Audio/Video w/QtMultimedia works
  • Raster engine works

Run ./qtbase/examples/gui/rasterwindow example.

  • Image formats work
  • GraphicsView works
  • OpenGL works

Run ./qtbase/examples/gui/openglwindow example. This is also checked by most of the other examples.

  • Printing works
  • QML 1 apps work (qmlviewer)

RTA: RTA runs few QtQML1 examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.

  • QML 2 apps work (qmlscene)

RTA: RTA runs few QtQML2 examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.

  • QtNetwork works

RTA: RTA runs few QtNetwork examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.

  • QtSql works

RTA: RTA runs few QtSql examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.

  • QtSvg works

RTA: RTA runs few QtSvg examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.

  • QtXml works

RTA: RTA runs few QtXML examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.

  • QtWebEngine works
  • QtWebKit 1 works
  • QtWebkit 2 works
  • Qt Script 2 works

Run ./qtscript/examples/script/qstetrix example. Use Debug option. context2d can be also run. RTA: RTA runs few QtScript examples and verifies that some detailed level Qt functionality can be used. Example ./qtscript/examples/script/qstetrix is included into the testing. They are launched via the command prompt i.e. not from the Qt Creator.

Qt Creator Checks

  • Right Qt Creator version is installed

Check Qt Creator version in About dialog.

RTA: Linux, Mac: Qt Creator version is verified from the About dialog. RTA: Windows: not yet part of RTA (some testing tool issues; work ongoing to solve this). Manual testing required.

RTA: Linux, Mac: Qt Creator is opened and closed in RTA tests. RTA: Windows: Qt Creator open & close not yet covered by RTA (some testing tool issues; work ongoing to solve this). In general, other test cases except open&close are not yet part of RTA. Manual testing required.

CMake Checks

Ensure that the binary package was installed with the ‘source components’ checked.

On Linux, install cmake from a repository (at least CMake version 2.8.3) and copy a script like this to your Qt installation and run it: 

#!/bin/bash

QT_INSTALLATION_PREFIX=$PWD/gcc_64

for d in `find . -name "cmake.pro"`; do
 b=`dirname $d`

m=`dirname $b`
 m=`dirname $m`
 m=`dirname $m`
 m=`basename $m`
 if [ x$m != x"qtactiveqt" ]; then
 pushd $b > /dev/null
 rm -rf build
 $QT_INSTALLATION_PREFIX/bin/qmake
 make check
 popd > /dev/null
 fi
done

On Windows with msvc use a bat script instead (download cmake from http://www.cmake.org/cmake/resources/software.html ): 

set QT_INSTALLATION_PREFIX=%cd:%/msvc2010
set scriptbase=%cd%
FOR %%? IN (qtactiveqt qtbase qtdeclarative qtmultimedia qtquick1 qtscript qtsensors qtserialport qtsvg qttools qtxmlpatterns) DO (
 cd Srcamp;#37;%?rd /S /Q build
 %QT_INSTALLATION_PREFIX%/bin/qmake
 nmake check
 cd %scriptbase%
)
cd Src\qtwebkit\Source\WebKit\qt\tests\cmake
rd /S /Q build
%QT_INSTALLATION_PREFIX%/bin/qmake
nmake check
cd %scriptbase%

If testing with MinGW on Windows, use a script like this; 

set QT_INSTALLATION_PREFIX=%cd:%/mingw48_32
set scriptbase=%cd%
FOR %%? IN (qtactiveqt qtbase qtdeclarative qtmultimedia qtquick1 qtscript qtsensors qtserialport qtsvg qttools qtxmlpatterns) DO (
 cd Srcamp;#37;%?rd /S /Q build
 %QT_INSTALLATION_PREFIX%/bin/qmake
 mingw32-make check
 cd %scriptbase%
)
cd Src\qtwebkit\Source\WebKit\qt\tests\cmake
rd /S /Q build
%QT_INSTALLATION_PREFIX%/bin/qmake
mingw32-make check

On all platforms, the result should be output showing the cmake tests running and passing. Please report failures to the releasing mailing list.

Links

Retrieved from "https://wiki.qt.io/index.php?title=Sanity-Test-Guidelines&oldid=15781"