Sanity-Test-Guidelines: Difference between revisions

From Qt Wiki
Jump to navigation Jump to search
No edit summary
 
No edit summary
Line 1: Line 1:
=Sanity Test=


Sanity Test is a part of the releasing cycle. The purpose is to ensure that the Qt <span class="caps">SDK</span> is ready for further testing.
<font color="blue">'''<span class="caps">RTA</span> work ongoing:'''</font><br /> Part of the Sanity test cases are covered by Release test automation (<span class="caps">RTA</span>). See more information on the <font color="blue">'''<span class="caps">RTA</span>'''</font> (covered by <span class="caps">RTA</span>) and <font color="red">'''<span class="caps">RTA</span>'''</font> (not yet covered by <span class="caps">RTA</span>) sections below.
==Build checks==
* '''Asks about license'''
<font color="blue">'''<span class="caps">RTA</span>:'''</font> <span class="caps">RTA</span> verifies that license key is asked while running the configure (valid license key, invalid license key). When the installer is run the installing with a license key is verified by <span class="caps">RTA</span>.
* '''Compiles with minimal options (eg. -opensource -confirm-license)'''<br /> Run configure command with the following options <br /><font color="blue">'''<span class="caps">RTA</span>'''</font> Linux, Mac, Windows: covered by <span class="caps">RTA</span>
* '''Compiles as static build (where supported)'''<br /> Run configure command with ‘-static’:<br /><font color="blue">'''<span class="caps">RTA</span>'''</font> Linux, Mac, Windows: covered by <span class="caps">RTA</span>
* '''Compiles in namespace (where supported)'''<br /> Run configure command with ‘-qtnamespace &lt;namespace&gt;’:<br /><font color="blue">'''<span class="caps">RTA</span>'''</font> Linux, Mac, Windows: covered by <span class="caps">RTA</span>
* '''Compiles with shadow build'''<br /> Create a new folder. cd to the new folder. Run the configure command from the source directory.<br /><font color="blue">'''<span class="caps">RTA</span>'''</font> Linux, Mac, Windows: covered by <span class="caps">RTA</span>
* '''Compiles cross-compiled'''<br /> Use -xplatform parameter, e.g.<br />
==General checks==
* '''Text files have the correct <span class="caps">EOL</span>'''<br /> Open a few text files (<span class="caps">README</span>, .cpp, .h, .pro, etc.) and check if they look OK.
* '''Files/dirs have sane perm. and timestamps'''
* '''Tags have been replaced properly'''<br /> The list of tags: qmake -query (???)
* '''<span class="caps">README</span> has valid information'''<br /> Complete information on how to build Qt from sources should be given.
==Installer checks==
* '''Right version of Qt Installer Framework is used'''<br /> Launch the installer with ‘—version’ . For Qt 5.4 -beta it should show ‘<span class="caps">IFW</span> Version: “1.5.82”, Installer base SHA1: “f67db7e”,<br /><font color="blue">*RTA:'''% Linux, Mac covered by <span class="caps">RTA</span>.<br />'''</font>{color:red}*RTA:% Windows: Manual testing required.
* '''Fresh install works correctly'''<br /> Install Qt <span class="caps">SDK</span> on a newly installed OS.
<font color="blue">'''<span class="caps">RTA</span>:'''</font> <span class="caps">RTA</span> verifies that clean installation works ok. Clean means that the installation directory does not contain any other Qt version.<br /><font color="red">'''<span class="caps">RTA</span>:'''</font> Whole OS is not installed as a fresh installation during the <span class="caps">RTA</span>. Study how to do this added into CI’s ta
* '''Fresh install to place requiring elevated rights works correctly'''<br /> Install Qt <span class="caps">SDK</span> as a normal user to a place requiring elevated rights. The installer should allow this by asking for elevated rights / the sudo password.
* '''Upgrade install works correctly'''<br /> Install higher version of Qt having older one, e.g. install Qt 5.3.2 on Qt 5.2.0.
* '''Parallel install works correctly'''<br /> Install two Qt installation in parallel, e.g. Qt 4.8.6 and 5.3.2. Check that both packages work
* '''Warns when installing over an installation'''<br /> Run the installer the second time. A warning pop-up windows should be displayed.
* '''Is correctly signed (vendor/certificate, not untrusted)'''<br /> Check where applicable.
* '''Displays appropriate gfx &amp; strings &amp; version no.'''<br /> This check is about general look and correct package version number.
* '''Offers the correct license(s)'''
<font color="blue">'''<span class="caps">RTA</span>:'''</font> <span class="caps">RTA</span> verifies that installation works with the valid license key. <br /><font color="red">'''<span class="caps">RTA</span>:'''</font> Visual verification of the installer and installing with the invalid license keys needs to be done manually.
* '''Offers sane default install directory (w/version number)'''<br /> Check of the default folder is similar to ../Qt5.3.2/<br /><font color="blue">'''<span class="caps">RTA</span>:'''</font> <span class="caps">RTA</span> verifies the default directory.
* '''Correctly installs to default directory'''<br /><font color="blue">'''<span class="caps">RTA</span>:'''</font> <span class="caps">RTA</span> verifies installing into the default directory.
* '''Correctly installs to non-default loc.'''<br /> Change the target folder during installation from ../Qt5.3.2/ to a different name.<br /><font color="blue">'''<span class="caps">RTA</span>:'''</font> <span class="caps">RTA</span> verifies installing into the non-default directory.
* '''Sanely reports progress and completion'''<br /> Click on the Details button to see the progress. Wait for successful completion of the installation process.
* '''Installs only selected components'''<br /> Try to use Deselect All button, and select a few additional components. Component selector should works sanely. After installation is done, only selected components should be in the target folder.<br /><font color="blue">'''<span class="caps">RTA</span>:'''</font> <span class="caps">RTA</span> verifies that only selected components are installed.
* '''Shortcuts from last page of installer works'''<br /> Select/deselect two check boxes (to run Qt Creator and see notes) on the final screen.
* '''Correctly creates Desktop shortcuts'''
* '''Correctly creates Start Menu shortcuts'''<br /> Check if Qt Creator can be found by the OS (start menu, launcher, dash home, etc.). Check the icon.
* '''Sets correct environment variables for shell'''
* '''Package shows up in Control Panel/Pkg. Manager'''
* '''Patching of files is done correctly'''
* '''Cancel button is available'''<br /> Use cancel button. Check if there are no changes done by the installer when the installation is interrupted.<br /><font color="blue">'''<span class="caps">RTA</span>:'''</font> <span class="caps">RTA</span> verifies that the installation can be cancelled. Note: <span class="caps">RTA</span> does not support cancelling the installing after the ‘install’ button is pressed i.e. while the installing of the components is ongoing.
* '''Cleans up after install failure'''
* '''Cleanly fails on insufficient disk space'''
==Uninstaller checks==
* '''Uninstalls properly'''<br /><font color="blue">'''<span class="caps">RTA</span>:'''</font> <span class="caps">RTA</span> verifies that uninstalling is done properly.
* '''Removes installed files properly'''<br /><font color="blue">'''<span class="caps">RTA</span>:'''</font> <span class="caps">RTA</span> verifies that all installed files are removed.
* '''Removes empty directories'''<br /><font color="blue">'''<span class="caps">RTA</span>:'''</font> <span class="caps">RTA</span> verifies that the installation directory is removed.
* '''Removes registry keys'''
* '''Reverses any other changes'''
==Both source/binary package checks==
* '''License is correct'''<br /><font color="blue">'''<span class="caps">RTA</span>:'''</font> <span class="caps">RTA</span> verifies that proper license files are found from $QTDIR\Licenses directory.
* '''Assistant works correctly'''<br /> Run Assistant (e.g. ./5.0.2/gcc_64/bin/assistant). Check the basic functionality, browse through, search for documentation.<br /><font color="blue">'''<span class="caps">RTA</span>:'''</font> <span class="caps">RTA</span> verifies that Assistant can be opened &amp; closed, browsing and searching works.
* '''Designer works correctly'''
* '''Qt Creator works correctly'''<br /> Qt Creator can be tested using the following test cases: [[:Category:Tools::QtCreator::QualityAssurance|Qt Creator Tests:]] ''[qt.io]''
* '''Demos and examples launch without crashing'''<br /><font color="blue">'''Each example'''</font> should be run with no error.<br /><font color="blue">'''<span class="caps">RTA</span>:'''</font> <span class="caps">RTA</span> launches and closes each Qt example (except console examples).
* '''Demos and examples function acceptably'''
'''<span class="caps">RTA</span>:''' Sanity level testing; see the QtModule sections below.
* '''Demos and examples can be rebuilt'''<br /><font color="blue">'''<span class="caps">RTA</span>:'''</font> <span class="caps">RTA</span> compiles the examples and demos from /Src and /bin directories.
* '''External apps build (Qt Creator and other applications)'''<br /> Create your own application, compile it and run.
* '''“<span class="caps">DLL</span> Swapping” work on an application'''
* '''<span class="caps">GUI</span> stress-testing works ok'''<br /> Try to perform many keystrokes, mouse clicks, etc.
* '''Audio/Video w/QtMultimedia works'''
* '''Raster engine works'''<br /> Run ./qtbase/examples/gui/rasterwindow example.
* '''Image formats work'''
* '''GraphicsView works'''
* '''OpenGL works'''<br /> Run ./qtbase/examples/gui/openglwindow example.<br /> This is also checked by most of the other examples.
* '''Printing works'''
* '''<span class="caps">QML</span> 1 apps work (qmlviewer)'''<br /><font color="blue">'''<span class="caps">RTA</span>:'''</font> <span class="caps">RTA</span> runs few QtQML1 examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.
* '''<span class="caps">QML</span> 2 apps work (qmlscene)'''<br /><font color="blue">'''<span class="caps">RTA</span>:'''</font> <span class="caps">RTA</span> runs few QtQML2 examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.
* '''QtNetwork works'''<br /><font color="blue">'''<span class="caps">RTA</span>:'''</font> <span class="caps">RTA</span> runs few QtNetwork examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.
* '''QtSql works'''<br /><font color="blue">'''<span class="caps">RTA</span>:'''</font> <span class="caps">RTA</span> runs few QtSql examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.
* '''QtSvg works'''<br /><font color="blue">'''<span class="caps">RTA</span>:'''</font> <span class="caps">RTA</span> runs few QtSvg examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.
* '''QtXml works'''<br /><font color="blue">'''<span class="caps">RTA</span>:'''</font> <span class="caps">RTA</span> runs few QtXML examples and verifies that some detailed level Qt functionality can be used. They are launched via the command prompt i.e. not from the Qt Creator.
* '''QtWebEngine works'''
* '''QtWebKit 1 works'''
* '''QtWebkit 2 works'''
* '''Qt Script 2 works'''<br /> Run ./qtscript/examples/script/qstetrix example. Use Debug option. context2d can be also run.<br /><font color="blue">'''<span class="caps">RTA</span>:'''</font> <span class="caps">RTA</span> runs few QtScript examples and verifies that some detailed level Qt functionality can be used. Example ./qtscript/examples/script/qstetrix is included into the testing. They are launched via the command prompt i.e. not from the Qt Creator.
==Qt Creator Checks==
* '''Right Qt Creator version is installed'''<br /> Check Qt Creator version in About dialog.
<font color="blue">'''<span class="caps">RTA</span>:'''</font> Linux, Mac: Qt Creator version is verified from the About dialog.<br /><font color="red">'''<span class="caps">RTA</span>:'''</font> Windows: not yet part of <span class="caps">RTA</span> (some testing tool issues; work ongoing to solve this). Manual testing required.
* [[:Category:Tools::QtCreator::QualityAssurance|Qt Creator Tests:]] ''[qt.io]''
<font color="blue">'''<span class="caps">RTA</span>:'''</font> Linux, Mac: Qt Creator is opened and closed in <span class="caps">RTA</span> tests. <br /><font color="red">'''<span class="caps">RTA</span>:'''</font> Windows: Qt Creator open &amp; close not yet covered by <span class="caps">RTA</span> (some testing tool issues; work ongoing to solve this). In general, other test cases except open&amp;close are not yet part of <span class="caps">RTA</span>. Manual testing required.
==CMake Checks==
Ensure that the binary package was installed with the ‘source components’ checked.
On Linux, install cmake from a repository (at least CMake version 2.8.3) and copy a script like this to your Qt installation and run it:
On Windows with msvc use a bat script instead (download cmake from http://www.cmake.org/cmake/resources/software.html ):
If testing with MinGW on Windows, use a script like this;
On all platforms, the result should be output showing the cmake tests running and passing. Please report failures to the releasing mailing list.
=Links=
* [[Building-Qt-5-from-Git|Building Qt 5 from Git]] ''[qt.io]''
* [http://www.qtcentre.org/wiki/index.php?title=Building_static_Qt_on_Windows Building static Qt on Windows] ''[qtcentre.org]''
* [http://qt.io/faq/answer/what_is_the_usage_of_the_qtlibinfix_and_qtnamespace_configuration_options What is the usage of the qtlibinfix and qtnamespace configuration options?] ''[qt.io]''
* [https://bugreports.qt.io/secure/Dashboard.jspa Bugtracker] ''[bugreports.qt.io]''
===Categories:===
* [[:Category:Release|Release]]
** [[:Category:Release::Testing|Testing]]

Revision as of 07:45, 24 February 2015

Retrieved from "https://wiki.qt.io/index.php?title=Sanity-Test-Guidelines&oldid=3908"