CMake Port/Development Notes

2019

07. November 2019

- qtbase benchmarks ported
- Some additional auto tests in qtbase (gui/util) were ported
- WIP on making standalone tests work better
- WIP on fixing some android architecture test madness regarding sse instruction sets

31. October 2019

- Ported qtx11extras
- Ported some more qtbase tests
- qtremoteobjects still WIP

24. October 2019

- Android is now built on Coin, tests are not currently executed though, just built
- Ported qtbase tests/auto/plugin tests
- Added support for jar file handing in pro2cmake
- make "test" almost works for Android (run androiddeployqt)
- Some fixes regarding c++ standard done in QtBuild.cmake

10. October 2019

- WIP building Android in Coin
- Finished porting qtquickcontrols2
- WIP porting rest of the qtbase tests and examples
- pro2cmake improved to handle examples better

03. October 2019

- Ported qtwinextras, qtknx
- WIP porting qtconnectivitly, qtremoteobjects
- Various fixes in pro2cmake
- WIP on getting Android building in Coin
- Various missing qtbase tests ported

26. September 2019

- A CMake Workshop took place
- Discussed many things in the workshop (API naming, providing information json files when building qt, various unhandled corner cases in the scripts)
- Ported qtscxml, qtwebchannel
- Started porting qtx11extras
- Ported a few more of the missing qtbase plugins
- Added support for configuring and building out-of-line config.tests
- Succeeded in building Qt Creator with CMake against a Qt6 built with CMake
- Various fixes in pro2cmake (better handling of requires() clauses in projects)
- Improved some of the qtdeclarative CMake public API
- Aligned handling of static plugins in Qt6 + CMake similar to how it is done in Qt 5.14 + CMake (API should be compatible now)

19. September 2019

- An internal CMake Field trip took place
- Various fixes done as a result of the field trip both in pro2cmake and CMake build API
- Ported qtwebsockets, qtcoap, qtserialport
- Started porting other repos: qtserialbus, qtwayland, qtknx, qtspeech, qtconnectivity, qtwinextras, qtscxml, qtwebchannel
- qtdeclarative is almost done (examples, tests, public api)

12. September 2019

- Fixes to pro2cmake to handle top level repo projects, tests top level projects and examples top level projects better
- Various fixes to prepare for the CMake field trip
- More qtbase tool tests ported
- Added precompile header support to the CMake port (requires CMake 3.16+)

05. September 2019

- Coin can now build repositories other than qtbase, currently successfully building qtbase, qtdeclarative, qtsvg, qtnetworkauth, qtimageformats.  
- qtquickcontrols2, qttools, qtmqtt are next on the list of building in Coin.
- CMake public api for creating qml plugins almost done. 
- Doing research work on the mixing qmake and cmake usecase. ( final goal - build qtwebengine with qmake against qtbase built with cmake). 
- Ported mysql, oracle, db2 sql driver plugins. 

29. August 2019

- Ported qtdesigner and qdoc from qttools 
- Started research on how hard will it be to generate required qmake files to allow building modules with a qmake which was built with CMake 
- Ongoing work on looking into latest Qt For Android changes, to make it work with latest 5.14 codebase merged into wip/cmake 
- Some investigation into WebAssembly support (qtbase is built without tests / and examples on local machine, not everything upstreamed yet)  
- work in progress patch for qtbase xml tests, they don't pass yet, looking into solution 
- Ongoing work on getting Coin yaml instructions for CMake working to build repositories other than qtbase 

22. August 2019

- Ongoing work on porting qt/qttools (designer almost ported) 
- Ongoing work on merging dependencies.yaml with correct SHA1s into all wip/qt6 branches 
- Some fixes in Qt5 CMake integration
- Ongoing work on getting some of the CMake private API to be public (like resource handling) 
- Ongoing work on porting all examples in qtdeclarative (requires changes in pro2cmake as well) 
- Ongoing progress on getting Coin to build repositories other than qtbase 
- More work on qtbase tests

15. August 2019

- qt5.git wip/qt6 is ready and green
- Merges from wip/qt6 to wip/cmake of ported repositories done
- 99% of qtdeclarative tests have been ported
- Some small fixes regarding warnings when building qtsvg, qtimageformats

08. August 2019

- Lots of fixes regarding Qml / Qt Quick support in the build system
- Lots of fixes regarding the Qt resources system build integration
- Lots of fixes in the pro2cmake conversion script
- Implemented proper support for qmake's functionality of removing subdirectories in CMake land, specifically in the pro2cmake.py script
- Initial port of qtgraphicaleffects
- Ongoing work on porting tests for qtdeclarative
- wip/qt6 is almost ready (missing webengine and webview builds)

01. August 2019

- Lots of work happening regarding properly building qml modules
- Initial work done on porting qtquickcontrols2
- Initial work done on getting the Qt Quick Compiler (qmlcachegen) to be used when building qtdeclarative / qtquickcontrols2 
- Ported the qml plugins / modules for qtdeclarative, confirmed that an example which uses qtdeclarative and QML works
- Implemented proof of concept for building universal (device and simulator) libraries for iOS, confirmed working with qtbase, qtsvg, qtdeclarative, but the work requires upstream changes in CMake and vcpkg (which is not merged yet)
- Merged support for automatic importing of static plugins in user applications when using a Qt static build
- Implemented proper support for default QPA plugin behaviour when using static Qt builds
- Looooots of work trying to get the wip/qt6 branches of all repositories into shape and getting the qt5.git wip/qt6 branch be up-to-date
- Some small fixes done regarding static builds, static plugins, cross-compilation and non-prefix builds
- Fixed a small bug in the pro2cmake conversion script to handle APPLE conditions correctly
- Fixed a bug in pro2cmake to correctly recursively expand qmake variables
- Ported more of the QtNework tests
- Initial work started on framework builds

Other interesting news:

- Qt Creator master branch learned to use the new CMake File API, which allows nicer integration with Qt Creator's project explorer
- Upstream work being done by Qt developers in CMake to get official CMake precompiled header support (it's almost done) and Unity builds (WIP)
- CMake precompiled header support confirmed to reduce build times of Qt Creator when built with CMake
- Finished and polished the implementation for automatic importing of Qt plugins in Qt applications built with CMake and Qt 5.x (awaits review)
- Implemented support / proof of concept for automatic importing of QML plugins in Qt applications built with CMake and Qt 5.x (awaits review)

04. July 2019

- Most CI code to support CMake has been done and merged to master, but needs to be deployed to production Coin, and then we would need to start a multi step process to make CMake and qmake builds coexist
- Lots of work done to support Android builds, building Qt libraries and applications using Qt libraries has been tested to work on armv7 devices and emulator
- Some preliminary work done to provide a convenience toolchain for building other modules and applications, which is especially relevant for Android / embedded Linux
- Lots of fixes for embedded Linux cross-builds regarding Qt Tools and packages
- Added ICC specific fixes to build on CI
- Added support for vcpkg toolchain chain loading in the Qt generated toolchain file

The rough steps that were considered for preparing CMake and qmake co-building in CI are:

- Update qt5 repo wip/qt6 branch to contain UseLegacyInstructions flag for all qmake configs, this is just a preparation step and everything should be built as before
- Merge all wip/cmake branches into wip/qt6 branches of all submodules (qmake configs still build as before, no new cmake configs are built yet, but this does bring in the yaml files from the wip/cmake branch) 
- We need to update qt5 repo wip/qt6 branch to contain all updated SHA1s of wip/qt6 repos (this will probably be the biggest issue, possible solutions is to wait for the pinning bot, or use direct pushes)
- Activate the cmake based configurations in qt5 wip/qt6 by duplicating the qmake configurations, and just removing the UseLegacyInstructions flag, but only for specific platform configurations that are known to be built in the wip/cmake branch 

13. June 2019

- 2 day workshop happend on 4th and 5th of June
- More work done on CI side to make Windows and macOS early warning system work
- qtbase dev was merged into wip/qt6, in preparation of merging wip/qt6 into wip/cmake
- More work done on porting qtdeclarative, libraries and tools are built, started working on qml plugins
- Ported WinMain for Windows gui applications
- Work started on Android support, qtbase already builds, the android support library is built, and necessary bits for deployment (generation of json file) are also available
- Added basic support for compiling applications using a qmake built with CMake
- Added warnings_are_errors support
- Fixed test data to be found properly, so now more tests should pass
- Implemented standalone tests building (needed for CI)
- Implemented standalone examples building (needed for end-users and Qt Creator)
- Ported QtNetworkAuth (so now we have qtbase, parts of qtdeclarative, qtsvg, qtimageformats, qtnetworkauth)
- Fixed some Windows issues due to bit rot
- Implemented initial support for automatic registration of Qt plugins to modules, and started work on automatic inclusion of plugins in static builds (patch in review)
- Fixed some xcb issues
- Implemented proper exception support
- Fixed OpenGL related issues
- Started work on implementing super builds (qt5 build), patch in review
- Fixed WinMain to be automatically linked into GUI applications

23. May 2019

- Coin now tests every single patch pushed to Gerrit to the qtbase wip/cmake branch on a single Linux Configuration, and posts the results on the Gerrit change (not enforcing), work is being done to test on Windows and macOS as well
- Most of qtimageformats has been ported
- Tests are now buildable with an installed Qt (useful for Coin)
- Implemented bundled double-conversion building (so using a package manager to install it is not strictly necessary anymore)
- Fixed hard requirement on needing Widgets when building any other module (was needed for qtimageformats)
- Fixed qmake parser to handle qmake syntax better in pro2cmake.py (doesn't crash anymore when trying to run it on qtdeclarative files )
- Continued investigation into building documentation and translation files for Qt Creator with CMake, once done the work would be forwarded to the Qt CMake Port 
- Work is being done on test data installation, so that tests that use test data can now pass

16. May 2019

- Implemented non-prefix builds (enabled with developer-build)
- Ported a part of qtimageformats
- Side work finished on porting Qt Creator to build with cmake, building docs for Qt Creator might be used for qtbase and friends
- Configure script changes merged to allow passing arbitrary cmake flags
- Continued looking into .pri file generation
- Various small fixes

09. May 2019

- Implemented script to keep manual modifications when regenerating projects
- Fixed some issues regarding DBus 
- Started looking into non-prefix builds (developer-build)
- Continued work from workshop on library mapping,
- Modified conversion script to support standalone example building
- Fixed various Linux package issues
- Regenerated a bunch of subprojects in src/
- More work on configure argument mapping
- Started looking into .pri file generation for built modules (so that qmake can use them while building apps or other qt modules)
- Continued working on the version-less targets
- Continued work on static plugins auto importing

03. May 2019 Workshop day 2

Things implemented during workshop

1. Final touches on packages providing tools
2. Final touches on keeping track which dependencies need to be found while consuming Qt in a CMake application
3. Fix current branch to be buildable with released CMake 3.14.x
4. Added a few missing xcb platform plugins
5. Implemented building qtsvg against qtbase
6. Support building qtbase & qtsvg as static libs
7. Fixed building offscreen platform plugin
8. Improved GLX detection and GL integration
9. Improved convertor script and regenerated configure.cmake files to keep track of more 3rd party dependencies
10. Updated cmake/README
11. Fixed finding host tools during cross compiling
12. Implemented exporting of major-version-less Qt targets, to facilitate easier upgrading of CMake applications from Qt5 to Qt6
13. Improved mapping of libraries from qmake to CMake land
14. Improved handling of _nolink targets where only include headers have to be propagated (Vulkan, OpenSSL)
15. Investigated the possibility of providing a user friendly Qt + CMake toolchain file for CMake applications to consume
16. Initial fixes for building qtbase with MinGW
17. Started work on building examples as standalone projects (not part of qtbase build)
18. Tested a few successful qtbase builds on a private Coin instance
19. Started work on handling static plugins in a consuming CMake application (when using a static qtbase build)

02. May 2019 Workshop day 1

Discussion points

1. Summary on the current state of CI builds of Qt-CMake-port
2. What changes have been made since last workshop
3. Be able to build applications against Qt5 or Qt6 without doing a lot of search and replaces in project files
4. What branch we want to target with our CMake changes (wip/cmake or qt5/dev or qt6/dev)
5. Time line on when we think we could merge the CMake changes into the mainline branch (Milestone 1 probably)
6. Depending on which branch we merge, decide If we need to keep qmake build of Qt working – if merged to qtbase/dev, then we need to keep qmake anyway
7. The convenience of using CMake to build applications instead of qmake (simplify what the user has to specify on the CMake command line), if we can accomplish that easily
8. What package manager we want to use for 3rd party packages (vcpkg, Conan, both, something else?)
9. What we want to do with qmake (move into different repo, keep it in qtbase?), we need the mkspec files currently which are provided in qtbase/mkspecs.
10. Configure script for CMake (do we want it? What do we want it to do?)
11. How do we want to keep up to date with qtbase/dev (regeneration of project files)
12. Examples and tests (building outside of a Qt build)

CI CMake State of the Union

Qtbase can be built currently with dirty hacks on the 3 major configs (Windows, Linux, macOS)

Implementation to allow choosing between building using CMake and QMake Qt builds properly in Coin (using yaml files in project repositories). Allows specifying build steps and options.

Tests are currently built as part of the build VM.

Timeline is 1-2 months to get the proper Coin stuff in.

What changes have been made since last workshop

We decided to get rid of qrc files, we changed instead to generate CMake rules with the pro2cmake.py convertor, which reads the qrc files and then generates a file list of files that were inside the qrc file, which are then embedded in the CMakeLists.txt. We need to do it for all modules.

TODO: Backport the qrc file conversion to qt5.

Qt has public dependencies. In static builds of Qt, private dependencies become public. We need to propagate those inside the generated QtCoreConfig.cmake files, so that when an application does find_package(Qt5Core), it also tries to find Core's dependencies. We found a way to do that in a nicer way, intead of tracking the dependencies manually (as it is done currently in KDE).

Lots and lots of improvement in the pro2cmake and configurejson2cmake convertor scripts: qrc handling, path handling, simd sources, no_link targets (vulkan, openssl), public and private dependencies handling.

As a result we merged qt5/dev into wip/cmake, and regenerated the files. This got us much closer to the current state of qtbase/dev.

Sample of special casing:

add_qt_library(Foo  
    SOURCES  
        bla.cpp foo.cpp 
    LIBRARIES  
        ZLIB::ZLIB # special case 
        # FreeType special case : remove this 
) 

We now generate Config files for each module's tools, for example Qt5CoreTools exports rcc,moc tools,

Qt5WidgetsTools exports uic tool. These packages can be used to cross-compile Qt (for Yocto).

We also were able to do a Yocto cross build.

Configure script state

It now understands –D CMake arguments and forwards them to CMake, and also qmake --foo commands, but the –foo commands are currently not converted to CMake style –D arguments, so it's just for qmake Qt builds.

It currently does not support in-source builds, and instead creates build directories and invokes CMake there.

Windows configure script has not been changed yet.

The conigure script is not strictly necessary for the CI now (which was the initial reason why we wanted configure to map to CMake options) because of the YAML changes.

It's still useful to provide the convenience for the user (so existing configure lines don't berak), so the question now is when should we do it, and what mechanism (duplicate mappings in .sh and .bat files, unified CMake script, something else?)

Be able to build applications against Qt5 or Qt6 without doing a lot of search and replaces in project files

Allow mixing Qt5 and Qt6 packages in a project, to allow partial porting of projects.

# Usecase: application wants to mix both Qt5 and Qt6, to allow gradual porting 
set(QT_CREATE_VERSIONLESS_TARGETS OFF) 
find_package(Qt5 COMPONENTS Core Gui Widgets) # Creates only Qt5::Core 
find_package(Qt6 COMPONENTS Core Gui Widgets) # Creates only Qt6::Core 

# Usecase: application doesn't mix Qt5 and Qt6, but allows to fully switch to link against either Qt5 or Qt6 
set(MY_APP_USE_QT6 TRUE) # <- set at command line by application developer 
set(MY_APP_QT_MAJOR_VERSION 6) # <- set at command line by application developer
# set(QT_CREATE_VERSIONLESS_TARGETS ON) <- Default, doesn't need to be set 
if(MY_APP_USE_QT6) 
   find_package(Qt6 COMPONENTS Core Gui Widgets) # Creates Qt6::Core, Qt::Core 
else() 
   find_package(Qt5 COMPONENTS Core Gui Widgets) # Creates Qt5::Core, Qt::Core 
endif() 

Or you could use

find_package(Qt${MY_APP_QT_MAJOR_VERSION} COMPONENTS Core Gui Widgets) # Creates Qt6::Core, Qt::Core 

What branch do we want to use for the CMake port

Make a recommendation to merge to qtbase/dev on mailing list, and make it blocking, so either CMake or qmake changes will both block integrations if build or test fails.

Pros merging into qtbase/dev

• Supporting two build system will have to be done anyway, regardless of branch, so might as well to do it in dev: because it spreads knowledge (more exposure to CMake port), that we distribute the work of maintaining the CMake changes.
• Regardless of the branch it goes into, somebody would still have to maintain two build systems, either in dev or in wip/qt6, the question is just where the changes will have to be made. If it's done in wip/qt6 -> less people can do the work (because not many people are currently involved in that branch).

Cons merging into qtbase/dev

• If a Coin Cmake config fails, it will fail the qmake one as well
• More load on the CI
• People need to learn a new build system ( but let's be real, they'll have to do it at some point anyway )

Time line on when we think we could merge the CMake changes into the mainline branch: To be discussed, but probably not before the yaml Coin change goes into production, which would take 1-2-3 months, so it depends on Coin production update.

The convenience of using CMake to build applications instead of qmake (simplify what the user has to specify on the CMake command line), if we can accomplish that easily

Provide a toolchain file, that would ease use for CMake projects, make it as easy as using qmake for a project.

This includes passing along things from configuration time like: mkspec, address sanitizer flags, original toolchain file which was used while building Qt, the public dependency Find modules (ZLIB, Atomic), build type (debug vs release) on windows, because of different MSVC runtimes.

What package manager we want to use for 3rd party packages (vcpkg, Conan, both, something else?)

Qt wants a package manager for 2 reasons:

Building 3rd party packages (to get rid of src/3rdparty)

To replace IFW with something better / nicer

The consensus is that for replacing IFW, Conan is the better choice.

For acquiring 3rd party packages, vcpkg is a much nicer solution, because Conan has some issues with package name and target name generations

(Conan generates all lower case names, while vcpkg and upstream CMake uses upper case or mixed case).

Upstream Conan issues:

https://github.com/conan-io/conan/issues/4430

https://github.com/conan-io/conan/issues/4460

https://github.com/conan-io/conan/issues/4597

https://github.com/conan-io/conan/issues/3400

What we want to do with qmake (move into different repo, keep it in qtbase?), we need the mkspec files currently which are provided in qtbase/mkspecs

Three action points:

We need to clean up the mkspecs to be integrated with CMake (generate the platform headers) (timeline is "has to happen before 6.0")

We need to make qmake compile against Qt without ifdefs (timeline is "has to happen before 6.0")

Consider moving qmake out of qtbase (timeline is later than release of qt 6.0)

Pros for keeping qmake in qtbase

Coin would test if qmake can link against QtBase within a qtbase test run, which allows us to find issues with qmake ASAP as long as we support it.

Cons

Psychological argument that qmake is still around

Coin adjustments will be needed, if qmake is in a different repo

How do we want to keep up to date with qtbase/dev (regeneration of project files)

Depend on Coin to tell us if something breaks upon a dev -> wip/cmake merge.

Proposal to make merges every two weeks, to keep up to date with new changes.

When doing a merge, do a git diff, to see if any .pro and .pri and configure.json files were touched, which is a good indicator that some CMakeLists.txt files might need to be regenerated.

Examples and tests (building outside of a Qt build)

Tests.

First workable solution would be to build tests together with Qt in Coin, and then just copy all the generated and binary files under qtbase/tests to a different machine, keeping the exact same file paths.

Examples.

Rewrite pro2cmake.py generation script to generate cmake code for examples which would be standalone and not using QtBuild.cmake functionality. This implies not using add_qt_executable, and using regular add_executable() and macros that are exported (for dbus handling, rcc handling, etc).

25. April 2019

- First merge in a while was done from qt5/dev into wip/cmake and as a result regenerated important CMakeLists.txt files.
- Fixed Windows build after qt5/dev -> wip/cmake merge and file regeneration
- Improved the exporting Tools CMake Targets patch
- Implemented 3rd party library dependency propagation when exporting module config files (packages like ZLIB or double-conversion)
- Implemented dependency propagation for a static build of Qt (targets like tinycbor, qtharfbuzz, simd targets, qtcore_qobject)
- A static build of qtbase now almost works
- Fixed more path handling issues in the pro2cmake convertor script,
- Fixed dependency relationships between Main and Private modules
- Implemented proper generation for _nolink libraries (think Vulkan and OpenSSL) 
- CMake build succeeded with Coin custom instructions (yaml files) patch, will make it easier to tweak CMake build instructions, not merged yet
- More work on the configure script, it can now build with both qmake or cmake, but no command line feature mapping yet

11. April 2019

- qtbase can now be built on Windows 10 in a private Coin instance,
- Research ongoing on building qtsvg against an installed qtbase, some issues already discovered
- Research into cross compiling to iOS by leveraging latest CMake 3.14 iOS support (which is still rudimentary)
- Investigated Conan, hunter and vcpkg in regards to providing 3rd party packages for iOS
- Ongoing work on providing packages and targets for module tools (CoreTools -> moc, rcc, WidgetTools -> uic, etc) which can then be used during cross building
- Fixed pro2cmake generation script to handle .pri includes and files better (fewer things that are not found),
- Added support to the generation script for qmake style no_link libraries (openssl, vulkan),
- Continuing work on merging qt5 dev branch to wip/cmake 
- Worked on infrastructure in Coin to allow building projects both with qmake and cmake (or anything else really), a prototype is available but still needs some work
- Continuing work on the configure script, and the mapping between configure features and cmake features
- Discussion ongoing on how to handle building and running tests in Coin

04. April 2019

- Work done on using a yocto generated sysroot to cross-build qtbase to an embedded device (some qtbase patches pending review, some patches still need to be upstreamed to the meta-b2qt repo)
- Regenerated many CMakeLists.txt files with the pro2cmake script
- WIP fixing the conversion script to handle .pri inclusion paths better
- WIP fixing the Windows provisioning to build the Qt CMake port on Windows
- Tested building of qtbase on macOS and Linux on a private instance of Coin (succeeds)
- WIP fixing the conversion script to deal with qmake's "SUBDIRS -=" functionality, which is not supported natively by CMake
- Progress on changing the qtbase configure script to use CMake to build Qt (rudimentary support, no feature mapping yet)
- Various small fixes to omissions in regenerated CMakeLists.txt files

21. March 2019

- Improved pro2cmake script to regenerate Core, Network, Gui, Widgets modules, to have smaller diffs in anticipation of a dev -> wip/cmake merge
- Fixed windows build (qpa plugin, various small conditions and other tiny things)
- Fixed automoc and autorcc not to hang due to incorrect target dependencies (upstream patch)
- Fixed certain incorrect target dependencies
- Improved pro2cmake script to parse which files need to be processed by rcc, instead of using rcc for that
- Worked on provisioning of latest CMake binaries on macOS and Linux
- Worked on getting private instances of Coin to build the CMake port on macOS and Linux
- Initial work on getting the configure script to invoke CMake instead of qmake for building qtbase