Sanity-Test-Guidelines
| This article may require cleanup to meet the Qt Wiki's quality standards. Reason: Auto-imported from ExpressionEngine. Please improve this article if you can. Remove the {{cleanup}} tag and add this page to Updated pages list after it's clean. |
[toc align_right="yes" depth="2"]
Sanity Test
Sanity Test is a part of the releasing cycle. The purpose is to ensure that the Qt SDK is ready for further testing.
{color:blue}RTA work ongoing: Part of the Sanity test cases are covered by Release test automation (RTA). See more information on the {color:blue}RTA (covered by RTA) and {color:red}RTA (not yet covered by RTA) sections below.
Build checks
- Asks about license
{color:blue}RTA: RTA verifies that license key is asked while running the configure (valid license key, invalid license key). When the installer is run the installing with a license key is verified by RTA.
- Compiles with minimal options (eg. -opensource -confirm-license)
Run configure command with the following options
configure -developer-build -opensource -nomake examples -nomake tests -confirm-license
{color:blue}RTA Linux, Mac, Windows: covered by RTA
- Compiles as static build (where supported)
Run configure command with '-static':
configure -static …
{color:blue}RTA Linux, Mac, Windows: covered by RTA
- Compiles in namespace (where supported)
Run configure command with '-qtnamespace <namespace>':
configure -qtnamespace CustomName …
{color:blue}RTA Linux, Mac, Windows: covered by RTA
- Compiles with shadow build
Create a new folder. cd to the new folder. Run the configure command from the source directory. {color:blue}RTA Linux, Mac, Windows: covered by RTA
- Compiles cross-compiled
Use -xplatform parameter, e.g.
configure -embedded arm -xplatform qws/linux-arm-g++ …
General checks
- Text files have the correct EOL
Open a few text files (README, .cpp, .h, .pro, etc.) and check if they look OK.
- Files/dirs have sane perm. and timestamps
- Tags have been replaced properly
The list of tags: qmake -query (???)
- README has valid information
Complete information on how to build Qt from sources should be given.
Installer checks
- Right version of Qt Installer Framework is used
Launch the installer with '—version' . For Qt 5.4 -beta it should show 'IFW Version: "1.5.82", Installer base SHA1: "f67db7e", {color:blue}RTA: Linux, Mac covered by RTA. {color:red}RTA: Windows: Manual testing required.
- Fresh install works correctly
Install Qt SDK on a newly installed OS.
{color:blue}RTA: RTA verifies that clean installation works ok. Clean means that the installation directory does not contain any other Qt version. {color:red}RTA: Whole OS is not installed as a fresh installation during the RTA. Study how to do this added into CI's ta
- Fresh install to place requiring elevated rights works correctly
Install Qt SDK as a normal user to a place requiring elevated rights. The installer should allow this by asking for elevated rights / the sudo password.
- Upgrade install works correctly
Install higher version of Qt having older one, e.g. install Qt 5.3.2 on Qt 5.2.0.
- Parallel install works correctly
Install two Qt installation in parallel, e.g. Qt 4.8.6 and 5.3.2. Check that both packages work
- Warns when installing over an installation
Run the installer the second time. A warning pop-up windows should be displayed.
- Is correctly signed (vendor/certificate, not untrusted)
Check where applicable.
- Displays appropriate gfx & strings & version no.
This check is about general look and correct package version number.
- Offers the correct license(s)
{color:blue}RTA: RTA verifies that installation works with the valid license key. {color:red}RTA: Visual verification of the installer and installing with the invalid license keys needs to be done manually.
- Offers sane default install directory (w/version number)
Check of the default folder is similar to ../Qt5.3.2/ {color:blue}RTA: RTA verifies the default directory.
- Correctly installs to default directory
{color:blue}RTA: RTA verifies installing into the default directory.
- Correctly installs to non-default loc.
Change the target folder during installation from ../Qt5.3.2/ to a different name. {color:blue}RTA: RTA verifies installing into the non-default directory.
- Sanely reports progress and completion
Click on the Details button to see the progress. Wait for successful completion of the installation process.
- Installs only selected components
Try to use Deselect All button, and select a few additional components. Component selector should works sanely. After installation is done, only selected components should be in the target folder. {color:blue}RTA: RTA verifies that only selected components are installed.
- Shortcuts from last page of installer works
Select/deselect two check boxes (to run Qt Creator and see notes) on the final screen.
- Correctly creates Desktop shortcuts
- Correctly creates Start Menu shortcuts
Check if Qt Creator can be found by the OS (start menu, launcher, dash home, etc.). Check the icon.
- Sets correct environment variables for shell
- Package shows up in Control Panel/Pkg. Manager
- Patching of files is done correctly
- Cancel button is available
Use cancel button. Check if there are no changes done by the installer when the installation is interrupted. {color:blue}RTA: RTA verifies that the installation can be cancelled. Note: RTA does not support cancelling the installing after the 'install' button is pressed i.e. while the installing of the components is ongoing.
- Cleans up after install failure
- Cleanly fails on insufficient disk space
Uninstaller checks
- Uninstalls properly
{color:blue}RTA: RTA verifies that uninstalling is done properly.
- Removes installed files properly
{color:blue}RTA: RTA verifies that all installed files are removed.
- Removes empty directories
{color:blue}RTA: RTA verifies that the installation directory is removed.
- Removes registry keys
- Reverses any other changes
Both source/binary package checks
- License is correct
{color:blue}RTA: RTA verifies that proper license files are found from $QTDIR\Licenses directory.
- Assistant works correctly
Run Assistant (e.g. ./5.0.2/gcc_64/bin/assistant). Check the basic functionality, browse through, search for documentation. {color:blue}RTA: RTA verifies that Assistant can be opened & closed, browsing and searching works.
- Designer works correctly
- Qt Creator works correctly
Qt Creator can be tested using the following test cases: Qt Creator Tests:
- Demos and examples launch without crashing
{color:blue}Each example should be run with no error. {color:blue}RTA: RTA launches and closes each Qt example (except console examples).
- Demos and examples function acceptably
RTA: Sanity level testing; see the QtModule sections below.
- Demos and examples can be rebuilt
{color:blue}RTA: RTA compiles the examples and demos from /Src and /bin directories.
- External apps build (Qt Creator and other applications)
Create your own application, compile it and run.
- "DLL Swapping" work on an application
- GUI stress-testing works ok
Try to perform many keystrokes, mouse clicks, etc.
- Audio/Video w/QtMultimedia works
- Raster engine works
Run ./qtbase/examples/gui/rasterwindow example.
- Image formats work
- GraphicsView works
- OpenGL works
Run ./qtbase/examples/gui/openglwindow example. This is also checked by most of the other examples.
- Printing works
- QML 1 apps work (qmlviewer)
{color:blue}RTA: RTA runs few QtQML1 examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.
- QML 2 apps work (qmlscene)
{color:blue}RTA: RTA runs few QtQML2 examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.
- QtNetwork works
{color:blue}RTA: RTA runs few QtNetwork examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.
- QtSql works
{color:blue}RTA: RTA runs few QtSql examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.
- QtSvg works
{color:blue}RTA: RTA runs few QtSvg examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.
- QtXml works
{color:blue}RTA: RTA runs few QtXML examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.
- QtWebEngine works
- QtWebKit 1 works
- QtWebkit 2 works
- Qt Script 2 works
Run ./qtscript/examples/script/qstetrix example. Use Debug option. context2d can be also run. {color:blue}RTA: RTA runs few QtScript examples and verifies that some detailed level Qt functionality can be used. Example ./qtscript/examples/script/qstetrix is included into the testing. They are launched via the command prompt i.e. not from the Qt Creator.
Qt Creator Checks
- Right Qt Creator version is installed
Check Qt Creator version in About dialog.
{color:blue}RTA: Linux, Mac: Qt Creator version is verified from the About dialog. {color:red}RTA: Windows: not yet part of RTA (some testing tool issues; work ongoing to solve this). Manual testing required.
{color:blue}RTA: Linux, Mac: Qt Creator is opened and closed in RTA tests. {color:red}RTA: Windows: Qt Creator open & close not yet covered by RTA (some testing tool issues; work ongoing to solve this). In general, other test cases except open&close are not yet part of RTA. Manual testing required.
CMake Checks
Ensure that the binary package was installed with the ‘source components’ checked.
On Linux, install cmake from a repository (at least CMake version 2.8.3) and copy a script like this to your Qt installation and run it:
#!/bin/bash
QT_INSTALLATION_PREFIX=$PWD/gcc_64
for d in `find . -name "cmake.pro"`; do
b=`dirname $d`
m=`dirname $b`
m=`dirname $m`
m=`dirname $m`
m=`basename $m`
if [ x$m != x"qtactiveqt" ]; then
pushd $b > /dev/null
rm -rf build
$QT_INSTALLATION_PREFIX/bin/qmake
make check
popd > /dev/null
fi
done
On Windows with msvc use a bat script instead (download cmake from http://www.cmake.org/cmake/resources/software.html ):
set QT_INSTALLATION_PREFIX=%cd:%/msvc2010
set scriptbase=%cd%
FOR %%? IN (qtactiveqt qtbase qtdeclarative qtmultimedia qtquick1 qtscript qtsensors qtserialport qtsvg qttools qtxmlpatterns) DO (
cd Srcamp;#37;%?rd /S /Q build
%QT_INSTALLATION_PREFIX%/bin/qmake
nmake check
cd %scriptbase%
)
cd Src\qtwebkit\Source\WebKit\qt\tests\cmake
rd /S /Q build
%QT_INSTALLATION_PREFIX%/bin/qmake
nmake check
cd %scriptbase%
If testing with MinGW on Windows, use a script like this;
set QT_INSTALLATION_PREFIX=%cd:%/mingw48_32
set scriptbase=%cd%
FOR %%? IN (qtactiveqt qtbase qtdeclarative qtmultimedia qtquick1 qtscript qtsensors qtserialport qtsvg qttools qtxmlpatterns) DO (
cd Srcamp;#37;%?rd /S /Q build
%QT_INSTALLATION_PREFIX%/bin/qmake
mingw32-make check
cd %scriptbase%
)
cd Src\qtwebkit\Source\WebKit\qt\tests\cmake
rd /S /Q build
%QT_INSTALLATION_PREFIX%/bin/qmake
mingw32-make check
On all platforms, the result should be output showing the cmake tests running and passing. Please report failures to the releasing mailing list.