Sanity-Test-Guidelines

[toc align_right="yes" depth="2"]

Sanity Test

Sanity Test is a part of the releasing cycle. The purpose is to ensure that the Qt SDK is ready for further testing.

{color:blue}RTA work ongoing:
Part of the Sanity test cases are covered by Release test automation (RTA). See more information on the {color:blue}RTA (covered by RTA) and {color:red}RTA (not yet covered by RTA) sections below.

Build checks

• Asks about license

{color:blue}RTA: RTA verifies that license key is asked while running the configure (valid license key, invalid license key). When the installer is run the installing with a license key is verified by RTA.

• Compiles with minimal options (eg. -opensource -confirm-license)
  Run configure command with the following options
  

  <br />configure -developer-build -opensource -nomake examples -nomake tests -confirm-license<br />
  

  
  {color:blue}RTA Linux, Mac, Windows: covered by RTA

• Compiles as static build (where supported)
  Run configure command with '-static':
  

  <br />configure -static …<br />
  

  
  {color:blue}RTA Linux, Mac, Windows: covered by RTA

• Compiles in namespace (where supported)
  Run configure command with '-qtnamespace <namespace&gt;':
  

  <br />configure -qtnamespace CustomName …<br />
  

  
  {color:blue}RTA Linux, Mac, Windows: covered by RTA

• Compiles with shadow build
  Create a new folder. cd to the new folder. Run the configure command from the source directory.
  {color:blue}RTA Linux, Mac, Windows: covered by RTA

• Compiles cross-compiled
  Use -xplatform parameter, e.g.
  

  <br />configure -embedded arm -xplatform qws/linux-arm-g++ …<br />
  

General checks

• Text files have the correct EOL
  Open a few text files (README, .cpp, .h, .pro, etc.) and check if they look OK.
• Files/dirs have sane perm. and timestamps
• Tags have been replaced properly
  The list of tags: qmake -query (???)
• README has valid information
  Complete information on how to build Qt from sources should be given.

Installer checks

• Right version of Qt Installer Framework is used
  Launch the installer with '—version' . For Qt 5.4 -beta it should show 'IFW Version: "1.5.82&quot;, Installer base SHA1: "f67db7e&quot;,
  {color:blue}RTA: Linux, Mac covered by RTA.
  {color:red}RTA: Windows: Manual testing required.

• Fresh install works correctly
  Install Qt SDK on a newly installed OS.

{color:blue}RTA: RTA verifies that clean installation works ok. Clean means that the installation directory does not contain any other Qt version.
{color:red}RTA: Whole OS is not installed as a fresh installation during the RTA. Study how to do this added into CI's ta

• Fresh install to place requiring elevated rights works correctly
  Install Qt SDK as a normal user to a place requiring elevated rights. The installer should allow this by asking for elevated rights / the sudo password.

• Upgrade install works correctly
  Install higher version of Qt having older one, e.g. install Qt 5.3.2 on Qt 5.2.0.

• Parallel install works correctly
  Install two Qt installation in parallel, e.g. Qt 4.8.6 and 5.3.2. Check that both packages work
• Warns when installing over an installation
  Run the installer the second time. A warning pop-up windows should be displayed.

• Is correctly signed (vendor/certificate, not untrusted)
  Check where applicable.

• Displays appropriate gfx & strings & version no.
  This check is about general look and correct package version number.

• Offers the correct license(s)

{color:blue}RTA: RTA verifies that installation works with the valid license key.
{color:red}RTA: Visual verification of the installer and installing with the invalid license keys needs to be done manually.

• Offers sane default install directory (w/version number)
  Check of the default folder is similar to ../Qt5.3.2/
  {color:blue}RTA: RTA verifies the default directory.
• Correctly installs to default directory
  {color:blue}RTA: RTA verifies installing into the default directory.
• Correctly installs to non-default loc.
  Change the target folder during installation from ../Qt5.3.2/ to a different name.
  {color:blue}RTA: RTA verifies installing into the non-default directory.
• Sanely reports progress and completion
  Click on the Details button to see the progress. Wait for successful completion of the installation process.

• Installs only selected components
  Try to use Deselect All button, and select a few additional components. Component selector should works sanely. After installation is done, only selected components should be in the target folder.
  {color:blue}RTA: RTA verifies that only selected components are installed.
• Shortcuts from last page of installer works
  Select/deselect two check boxes (to run Qt Creator and see notes) on the final screen.
• Correctly creates Desktop shortcuts
• Correctly creates Start Menu shortcuts
  Check if Qt Creator can be found by the OS (start menu, launcher, dash home, etc.). Check the icon.
• Sets correct environment variables for shell
• Package shows up in Control Panel/Pkg. Manager
• Patching of files is done correctly

• Cancel button is available
  Use cancel button. Check if there are no changes done by the installer when the installation is interrupted.
  {color:blue}RTA: RTA verifies that the installation can be cancelled. Note: RTA does not support cancelling the installing after the 'install' button is pressed i.e. while the installing of the components is ongoing.
• Cleans up after install failure

• Cleanly fails on insufficient disk space

Uninstaller checks

• Uninstalls properly
  {color:blue}RTA: RTA verifies that uninstalling is done properly.
• Removes installed files properly
  {color:blue}RTA: RTA verifies that all installed files are removed.
• Removes empty directories
  {color:blue}RTA: RTA verifies that the installation directory is removed.
• Removes registry keys
• Reverses any other changes

Both source/binary package checks

• License is correct
  {color:blue}RTA: RTA verifies that proper license files are found from $QTDIR\Licenses directory.
• Assistant works correctly
  Run Assistant (e.g. ./5.0.2/gcc_64/bin/assistant). Check the basic functionality, browse through, search for documentation.
  {color:blue}RTA: RTA verifies that Assistant can be opened & closed, browsing and searching works.
• Designer works correctly
• Qt Creator works correctly
  Qt Creator can be tested using the following test cases: "Qt Creator Tests:":http://wiki.qt.io/Category:Tools::QtCreator::QualityAssurance
• Demos and examples launch without crashing
  {color:blue}Each example should be run with no error.
  {color:blue}RTA: RTA launches and closes each Qt example (except console examples).
• Demos and examples function acceptably

RTA: Sanity level testing; see the QtModule sections below.

• Demos and examples can be rebuilt
  {color:blue}RTA: RTA compiles the examples and demos from /Src and /bin directories.
• External apps build (Qt Creator and other applications)
  Create your own application, compile it and run.

• "DLL Swapping&quot; work on an application

• GUI stress-testing works ok
  Try to perform many keystrokes, mouse clicks, etc.
• Audio/Video w/QtMultimedia works
• Raster engine works
  Run ./qtbase/examples/gui/rasterwindow example.
• Image formats work

• GraphicsView works
• OpenGL works
  Run ./qtbase/examples/gui/openglwindow example.
  This is also checked by most of the other examples.
• Printing works
• QML 1 apps work (qmlviewer)
  {color:blue}RTA: RTA runs few QtQML1 examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.
• QML 2 apps work (qmlscene)
  {color:blue}RTA: RTA runs few QtQML2 examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.
• QtNetwork works
  {color:blue}RTA: RTA runs few QtNetwork examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.
• QtSql works
  {color:blue}RTA: RTA runs few QtSql examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.
• QtSvg works
  {color:blue}RTA: RTA runs few QtSvg examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.
• QtXml works
  {color:blue}RTA: RTA runs few QtXML examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.

• QtWebEngine works

• QtWebKit 1 works

• QtWebkit 2 works

• Qt Script 2 works
  Run ./qtscript/examples/script/qstetrix example. Use Debug option. context2d can be also run.
  {color:blue}RTA: RTA runs few QtScript examples and verifies that some detailed level Qt functionality can be used. Example ./qtscript/examples/script/qstetrix is included into the testing. They are launched via the command prompt i.e. not from the Qt Creator.

Qt Creator Checks

• Right Qt Creator version is installed
  Check Qt Creator version in About dialog.

{color:blue}RTA: Linux, Mac: Qt Creator version is verified from the About dialog.
{color:red}RTA: Windows: not yet part of RTA (some testing tool issues; work ongoing to solve this). Manual testing required.

• "Qt Creator Tests:":http://wiki.qt.io/Category:Tools::QtCreator::QualityAssurance

{color:blue}RTA: Linux, Mac: Qt Creator is opened and closed in RTA tests.
{color:red}RTA: Windows: Qt Creator open & close not yet covered by RTA (some testing tool issues; work ongoing to solve this). In general, other test cases except open&amp;close are not yet part of RTA. Manual testing required.

CMake Checks

Ensure that the binary package was installed with the ‘source components’ checked.

On Linux, install cmake from a repository (at least CMake version 2.8.3) and copy a script like this to your Qt installation and run it:

#!/bin/bash

QT_INSTALLATION_PREFIX=$PWD/gcc_64

for d in `find . -name &quot;cmake.pro&amp;quot;`; do<br /> b=`dirname $d`

m=`dirname $b`<br /> m=`dirname $m`<br /> m=`dirname $m`<br /> m=`basename $m`<br /> if [ x$m != x&amp;quot;qtactiveqt&amp;quot; ]; then<br /> pushd $b &gt; /dev/null<br /> rm -rf build<br /> $QT_INSTALLATION_PREFIX/bin/qmake<br /> make check<br /> popd &gt; /dev/null<br /> fi<br />done

On Windows with msvc use a bat script instead (download cmake from http://www.cmake.org/cmake/resources/software.html ):

set QT_INSTALLATION_PREFIX=&amp;#37;cd:&amp;#37;/msvc2010<br />set scriptbase=&amp;#37;cd&amp;amp;#37;<br />FOR &amp;#37;&amp;#37;? IN (qtactiveqt qtbase qtdeclarative qtmultimedia qtquick1 qtscript qtsensors qtserialport qtsvg qttools qtxmlpatterns) DO (<br /> cd Srcamp;#37;&amp;#37;?rd /S /Q build<br /> &amp;#37;QT_INSTALLATION_PREFIX&amp;amp;#37;/bin/qmake<br /> nmake check<br /> cd &amp;#37;scriptbase&amp;amp;#37;<br />)<br />cd Src\qtwebkit\Source\WebKit\qt\tests\cmake<br />rd /S /Q build<br />&amp;#37;QT_INSTALLATION_PREFIX&amp;amp;#37;/bin/qmake<br />nmake check<br />cd &amp;#37;scriptbase&amp;amp;#37;

If testing with MinGW on Windows, use a script like this;

set QT_INSTALLATION_PREFIX=&amp;#37;cd:&amp;#37;/mingw48_32<br />set scriptbase=&amp;#37;cd&amp;amp;#37;<br />FOR &amp;#37;&amp;#37;? IN (qtactiveqt qtbase qtdeclarative qtmultimedia qtquick1 qtscript qtsensors qtserialport qtsvg qttools qtxmlpatterns) DO (<br /> cd Srcamp;#37;&amp;#37;?rd /S /Q build<br /> &amp;#37;QT_INSTALLATION_PREFIX&amp;amp;#37;/bin/qmake<br /> mingw32-make check<br /> cd &amp;#37;scriptbase&amp;amp;#37;<br />)<br />cd Src\qtwebkit\Source\WebKit\qt\tests\cmake<br />rd /S /Q build<br />&amp;#37;QT_INSTALLATION_PREFIX&amp;amp;#37;/bin/qmake<br />mingw32-make check

On all platforms, the result should be output showing the cmake tests running and passing. Please report failures to the releasing mailing list.

Links

• "Building Qt 5 from Git&quot;:http://wiki.qt.io/Building-Qt-5-from-Git
• "Building static Qt on Windows&quot;:http://www.qtcentre.org/wiki/index.php?title=Building_static_Qt_on_Windows
• "What is the usage of the qtlibinfix and qtnamespace configuration options?":http://qt.io/faq/answer/what_is_the_usage_of_the_qtlibinfix_and_qtnamespace_configuration_options
• "Bugtracker&quot;:https://bugreports.qt.io/secure/Dashboard.jspa