SUBDIRS - handling dependencies: Difference between revisions

From Qt Wiki
Jump to navigation Jump to search
(Added a warning that CONFIG+=ordered is discouraged)
 
(26 intermediate revisions by 7 users not shown)
Line 1: Line 1:
[[Category:Tools::qmake]]
== Introduction ==
== Introduction ==


Line 9: Line 10:
The following text uses a small example, consisting of an application: app and two libraries: lib and lib2.
The following text uses a small example, consisting of an application: app and two libraries: lib and lib2.
The application app uses features of the libraries lib and lib2 while the libraries are in no direct relation.
The application app uses features of the libraries lib and lib2 while the libraries are in no direct relation.
So, to have a running application the toolchain must create the lib.lib and lib2.lib files and finally create the app.exe by linking the lib.lib and lib2.lib to the object files of the app components. (The following Text uses the Windows file name notations of the objects)
So, to have a running application the toolchain must create the lib.lib and lib2.lib files and finally create the app.exe by linking the lib.lib and lib2.lib to the object files of the app components. (The following Text uses the Windows file name notations of the objects.)


When the software project is in an early design and implementation phase, the libraries might change as often as the application using the libraries. In the Qt domain, each of the three topics: lib, lib2 and app will be handled bye one project each. Typically there will be three folders: lib, lib2 and app, each containing a specific .pro file, describing how to handle each project (which source files, compile parameters, ...). The profiles typically will be named after the folders they live in: lib.pro, lib2.pro and app.pro. For this example all three projects will live in one folder: src.
When the software project is in an early design and implementation phase, the libraries might change as often as the application using the libraries. In the Qt domain, each of the three topics: lib, lib2 and app will be handled by one project each. Typically there will be three folders: lib, lib2 and app, each containing a specific .pro file, describing how to handle each project (which source files, compile parameters, ...). The profiles typically will be named after the folders they live in: lib.pro, lib2.pro and app.pro. For this example all three projects will live in one folder: src.


On the file system this can be seen as following hierarchy:
On the file system this can be seen as following hierarchy:
  <nowiki>
  <nowiki>
/src
  /src
|--- app
  |--- app
|    |--- app.pro
  |    |--- app.pro
|    `--- ... (source files of app)
  |    `--- ... (source files of app)
|--- lib
  |--- lib
|    |--- lib.pro
  |    |--- lib.pro
|    `--- ... (source files of lib)
  |    `--- ... (source files of lib)
`--- lib2
  `--- lib2
      |--- lib2.pro
        |--- lib2.pro
      `--- ... (source files of lib2)
        `--- ... (source files of lib2)
</nowiki>
  </nowiki>


Each of this projects could for example be edited separately in three Microsoft Visual Studio 2010 IDEs (MSVS2010) or handled as three separate projects in one QtCreator. But this handling of the projects would mean to compile each by hand separately.
Each of this projects could for example be edited separately in three Microsoft Visual Studio 2010 IDEs (MSVS2010) or handled as three separate projects in one QtCreator. But this handling of the projects would mean to compile each by hand separately.
Line 34: Line 35:
To handle this relation automatically qmake has a template type: subdirs ([http://doc.qt.io/qt-5/qmake-variable-reference.html#template template]).  
To handle this relation automatically qmake has a template type: subdirs ([http://doc.qt.io/qt-5/qmake-variable-reference.html#template template]).  


The basic idea of this template type is, to list all projects that belong to a kind of meta project.
The basic idea of this template type is to list all projects that belong to a kind of meta project.
A new .pro file is created for that meta project, that simply consists of the qmake system variable [http://doc.qt.io/qt-5/qmake-variable-reference.html#subdirs SUBDIRS]. This variable is filled with the names of the project folders by giving the
A new .pro file is created for that meta project that simply consists of the qmake system variable [http://doc.qt.io/qt-5/qmake-variable-reference.html#subdirs SUBDIRS]. This variable is filled with the names of the project folders by giving the
relative path to where the meta .pro file lies. In the following text this meta profile is names subdirs.pro.
relative path to where the meta .pro file lies. In the following text this meta profile is named subdirs.pro.
For a better overview in the file hierarchy a new parent folder is introduced, named after the software project that is created:
For a better overview in the file hierarchy a new parent folder is introduced, named after the software project that is created:
FooApp.
FooApp.


  <nowiki>
  <nowiki>
/FooApp
  /FooApp
|--- subdirs.pro
  |--- subdirs.pro
`--- src
  `--- src
      |
        |
      |--- app
        |--- app
      |    |--- app.pro
        |    |--- app.pro
      |    `--- ... (source files of app)
        |    `--- ... (source files of app)
      |--- lib
        |--- lib
      |    |--- lib.pro
        |    |--- lib.pro
      |    `--- ... (source files of lib)
        |    `--- ... (source files of lib)
      `--- lib2
        `--- lib2
          |--- lib2.pro
            |--- lib2.pro
          `--- ... (source files of lib2)
            `--- ... (source files of lib2)
</nowiki>
  </nowiki>
The meta profile subdirs.pro is on the same level as the src folder.
The meta profile subdirs.pro is on the same level as the src folder.


The content of the subdirs.pro is
The content of the subdirs.pro is
  <nowiki>
   
template = subdirs
  TEMPLATE = subdirs
 
 
SUBDIRS = \
  SUBDIRS = \
          src/app \  # relative paths
            src/app \  # relative paths
          src/lib \
            src/lib \
          src/lib2
            src/lib2
</nowiki>
 


The subdirs.pro file can be opened by QtCreator. In QtCreator there will be just one project with child elements, the sub-project. The advantage is now that when compiling this meta project the sub-projects are compiled automatically. Asking for a qmake run within QtCreator on this meta project will also recurse into the sub-projects calling qmake for each.
The subdirs.pro file can be opened by QtCreator. In QtCreator there will be just one project with child elements: the sub-project. The advantage now is that when compiling this meta project the sub-projects are compiled automatically. Asking for a qmake run within QtCreator on this meta project will also recurse into the sub-projects calling qmake for each.


To make this meta project available for MSVC2010, qmake must be called on the command line in the folder FooApp:
To make this meta project available for MSVC2010, qmake must be called on the command line in the folder FooApp:
Line 74: Line 75:
This will generate a solution file with the name subdirs.sln. When opening this file in MSVC2010 a solution will be opened that contain the three project: app, lib, and lib2. A compile on the solution will call compile for each of the contained projects.
This will generate a solution file with the name subdirs.sln. When opening this file in MSVC2010 a solution will be opened that contain the three project: app, lib, and lib2. A compile on the solution will call compile for each of the contained projects.


'''Attention:''' The order in with the subproject are compiled is '''not''' defined by the SUBDIRS content. And this could lead to a wrong processing order. So, the app project could not be build if it is processed first; it would miss the libraries lib and lib2.
'''Attention:''' The order in which the subprojects are compiled is '''not''' defined by the SUBDIRS content. This could lead to a wrong processing order. So, the app project could not be built if it is processed first; it would miss the libraries lib and lib2.
One could assume that the build order is given by the order of the entities in the SUBDIRS variable (so example given above would be wrong). But many modern compile tools offer parallel compilation. If the order of the entity would be forcing the build order, then parallel compilation on the projects level would not be possible. Per default the sub-projects are compiled in any order the used build-systems logic prefers.
One could assume that the build order is given by the order of the entities in the SUBDIRS variable (so example given above would be wrong). But many modern compile tools offer parallel compilation. If the order of the entity would be forcing the build order, then parallel compilation on the projects level would not be possible. Per default the sub-projects are compiled in any order the used build-systems logic prefers.


=== Defining Project Dependencies: .depends attributes ===
=== Defining Project Dependencies: .depends attributes ===


To support parallel processing of the projects on modern computers, an additional information must be given: Which project must be finished before another project can be processed.
'''Attention Visual Studio''' This method is very fragile implemented in qmake when generating the Windows Visual Studio files (.sln and .vcxproj).
Qt 4.8.x does not support it at all and Qt 5.4.1 must have special setting in the sub projects pro files.
A bug was filled in [https://bugreports.qt.io/browse/QTBUG-45706 QTBUG-45706].
 
To support parallel processing of the projects on modern computers, additional information must be given: Which project must be finished before another project can be processed.


In this example we had noted that app can only be build, when lib and lib2 are build. There are two ways to tell qmake this dependency.
In the running example, we have noted that app can only be built when both lib and lib2 are built.


By telling the dependency explicitly in the meta project file, qmake can build processing instructions that allow a high degree of parallelism.
By specifying the dependency explicitly in the meta project file, qmake can build processing instructions that allow a high degree of parallelism.


  <nowiki>
   
  template = subdirs
  TEMPLATE = subdirs
 
  SUBDIRS = \
            lib2 \  # sub-project names
            lib \
            app
 
  # where to find the sub projects - give the folders
  lib2.subdir = src/lib2
  lib.subdir  = src/lib
  app.subdir  = src/app
 
  # what subproject depends on others
  app.depends = lib lib2
 


SUBDIRS = \
The important addition to the file is the line <tt>app.depends = lib lib2</tt>. Setting the .depends attribute of the project app tells qmake that app can only be built if lib and lib2 is processed.
          lib2 \  # sub-project names
          lib \
          app


# where to find the sub projects - give the folders
This allows qmake to build processing instructions that can be executed in a high degree of parallelism. For example lib and lib2 can be built in parallel, while app is held back until lib and lib2 processing is finished.
lib2.subdir = src/lib2
lib.subdir  = src/lib
app.subdir  = src/app


# what subproject depends on others
In addition there is an implicit introduction of projects names. In SUBDIRS the project names are given (not the path anymore). The path to the project directory is set below by setting the .subdir attribute to the relative path where the project folder is located.
app.depends = lib lib2
</nowiki>


=== Defining a processing sequence: CONFIG += ordered ===
=== Defining a processing sequence: CONFIG += ordered ===
'''Attention:''' Using CONFIG += ordered is ''not'' recommended as it hinders effective multicore builds. Using .depends is the correct way as written in this [[ https://blog.rburchell.com/2013/10/every-time-you-configordered-kitten-dies.html|blog]].
This section is kept only as there may be a bug when creating Visual Studio projects and specifying dependencies may otherwise not work.
'''Attention Visual Studio''': This feature is not supported under Qt 4.8.x ([http://doc.qt.io/qt-4.8/qmake-variable-reference.html#ordered-targets-and-visual-studio-solution-files Qt Docs 4.8]). Instead an automatic dependency detection scheme is used in the Microsoft Visual Studio file generation (.sln and .vcxproj). In the Qt5 documentation, the section describing the automatism is deleted! A small test showed that the ordered option as show here does still not work for Qt 5.4.1.
A bug report was filled in [https://bugreports.qt.io/browse/QTBUG-45706 QTBUG-45706].


With this option the processing of the sub-project can be set to follow the order in which they are listed in the SUBDIRS variable.
With this option the processing of the sub-project can be set to follow the order in which they are listed in the SUBDIRS variable.
  <nowiki>
   
template = subdirs
  TEMPLATE = subdirs
 
 
SUBDIRS = \
  SUBDIRS = \
          src/lib2\   
            src/lib2\   
          src/lib \
            src/lib \
          src/app
            src/app
 
  # build the project sequentially as listed in SUBDIRS !
  CONFIG += ordered
 
By having the listing order: src/lib2, src/lib, src/app the build tool will follow that order and build the project one at a time.
Due to processing in this strict project order the build dependencies are implicitly fulfilled, as long as the listing order is chosen correctly.


# build the project sequentially as listed in SUBDIRS !
But processing "one at a time" eliminates the possibility to process independent projects in parallel. However each projects might process in parallel in its local context - e.g. the .cpp files within lib2 might get compiled in parallel.
CONFIG += ordered
</nowiki>
By having the listing order: src/lib2, src/lib, src/app the build tool will follow that order and build each project after the other. This eliminates the possibility to process independent projects in parallel. However each projects might process in parallel in its local context.


=== Background on Windows Build Systems ===
=== Background on Windows Build Systems ===
There are different ways to compile a project (or meta project) under windows.
There are different ways to process a project (or meta project) under windows.


==== Makefiles ====
==== Makefiles ====
There are command line tools to compile project with the help of a compile rule file: a Makefile.
There are command line tools to compile project with the help of a compile rule file: a Makefile ([http://en.wikipedia.org/wiki/Makefile wikipedia]).


For a meta-project qmake must be called with the recursive flag in the FooApp folder, the one with the subdirs.pro file:
For a meta-project, qmake must be called with the recursive option in the FooApp folder, the one with the subdirs.pro file:
  qmake -r
  qmake -r
This will generate a Makefile in the same directory. In addition qmake will process all the subdirs given in the SUBDIRS variable and create Makefiles in each of the sub-project folders, containing rules to create the sub-project.
This will generate a Makefile in the same directory. In addition qmake will process all the subdirs given in the SUBDIRS variable and create Makefiles in each of the sub-project folders, containing rules to create the sub-project.
  <nowiki>
  <nowiki>
/FooApp
  /FooApp
|--- Makefile
  |--- Makefile
|--- subdirs.pro
  |--- subdirs.pro
`--- src
  `--- src
      |
        |
      |--- app
        |--- app
      |    |--- app.pro
        |    |--- app.pro
      |    |--- Makefile
        |    |--- Makefile
      |    |--- Makefile.Debug
        |    |--- Makefile.Debug
      |    |--- Makefile.Release
        |    |--- Makefile.Release
      |    `--- ... (source files of app)
        |    `--- ... (source files of app)
      |--- lib
        |--- lib
      |    |--- lib.pro
        |    |--- lib.pro
      |    |--- Makefile
        |    |--- Makefile
      |    |--- Makefile.Debug
        |    |--- Makefile.Debug
      |    |--- Makefile.Release
        |    |--- Makefile.Release
      |    `--- ... (source files of lib)
        |    `--- ... (source files of lib)
      `--- lib2
        `--- lib2
          |--- lib2.pro
            |--- lib2.pro
          |--- Makefile
            |--- Makefile
          |--- Makefile.Debug
            |--- Makefile.Debug
          |--- Makefile.Release
            |--- Makefile.Release
          `--- ... (source files of lib2)
            `--- ... (source files of lib2)
</nowiki>
  </nowiki>


===== nmake =====
===== nmake =====
Line 160: Line 181:
will find the Makefile and will process it. As this Makefile is one for a meta-project, it contains rules to process the Makefiles of the sub-project.
will find the Makefile and will process it. As this Makefile is one for a meta-project, it contains rules to process the Makefiles of the sub-project.


nmake has no feature to call compile or link in parallel. An option -j n as known with Unix make or gnumake does not exist. So nmake start compiles file after file and probably the listing order of the sub-projects within the SUBDIRS variable will be used. But this is nothing to rely on.
nmake has no feature to call compile or link in parallel. An option "-j <n>" as known with Unix make or gnumake does not exist. So nmake start project after project and in each project compiles file after file. Probably the listing order of the sub-projects within the SUBDIRS variable will be used. But this is nothing to rely on! How to determine the processing order sufficiently is shown in the chapters above.
 


===== jom =====
===== jom =====
The missing parallel execution feature of nmake is inefficient on modern computer architectures. With CPUs having many kernels for parallelism and fast storage units, a modern computer is bored by using nmake and project compilation takes longer as it could take. This lead to the creation of a new make tool: [[Jom]]. Jom is basically a nmake clone extended with the '/J <n>' option to compile in parallel. To compile a meta projects Makefile the call has to be
The missing parallel execution feature of nmake is inefficient on modern computer architectures. With CPUs having many kernels for parallelism and fast storage units, a modern computer is bored by using nmake and project compilation takes longer as it could take. This lead to the creation of a new make tool: [[Jom]]. jom is basically an nmake clone extended with the '/J <n>' option to compile in parallel. To compile a meta projects Makefile the call has to be
  jom
  jom
in the FooApp folder. Jom will recurse automatically in the sub-project folders and use the Makefile of the sub-folders. But all in a parallel fashion.
in the FooApp folder. jom will recurse automatically in the sub-project folders and use the Makefile of the sub-folders, but all in a parallel fashion.
To see the dependency graph that jom processes, the command
To see the dependency graph that jom processes, the command
  jom /DUMPGRAPH
  jom /DUMPGRAPH
can be called.
can be called.
For the insufficient example
For the insufficient and wrongly ordered example
  <nowiki>
   
template = subdirs
  TEMPLATE = subdirs
 
 
SUBDIRS = \
  SUBDIRS = \
          src/app \  # relative paths
            src/app \  # relative paths
          src/lib \
            src/lib \
          src/lib2
            src/lib2
</nowiki>
 
the output of jom is
the output of jom is
  <nowiki>
  <nowiki>
first
  first
make_first
  make_first
  sub-src-app-make_first
    sub-src-app-make_first
  FORCE
    FORCE
  sub-src-lib-make_first
    sub-src-lib-make_first
  FORCE
    FORCE
  sub-src-lib2-make_first
    sub-src-lib2-make_first
  FORCE
    FORCE
  FORCE
    FORCE
</nowiki>
  </nowiki>
All three sub-src-xxx-make_first are on the same level and they have no dependency under it (aside from FORCE).
All three sub-src-xxx-make_first are on the same level and they have no dependency under it (aside from FORCE).
So all is build in parallel, if the number of Kernels is high enough.
So ''all'' is build in parallel if the number of kernels is high enough.  
The parallel processing might result in a build error due to race conditions: The linker of the app project could be started even though the necessary libraries are not build already!


For an example with dependency given explicitly:
For an example with dependency given explicitly:
  <nowiki>
   
template = subdirs
  TEMPLATE = subdirs
 
 
SUBDIRS = \
  SUBDIRS = \
          lib2 \  # sub-project names
            lib2 \  # sub-project names
          lib  \
            lib  \
          app
            app
 
 
# where to find the sub projects - give the folders
  # where to find the sub projects - give the folders
lib2.subdir = src/lib2
  lib2.subdir = src/lib2
lib.subdir  = src/lib
  lib.subdir  = src/lib
app.subdir  = src/app
  app.subdir  = src/app
 
 
# what subproject depends on others
  # what subproject depends on others
app.depends = lib lib2
  app.depends = lib lib2
</nowiki>
 
jom output for the graph:
jom output for the graph:
  <nowiki>
  <nowiki>
first
  first
make_first
  make_first
  sub-src-lib-make_first
    sub-src-lib-make_first
  FORCE
    FORCE
  sub-src-lib2-make_first
    sub-src-lib2-make_first
  FORCE
    FORCE
  sub-src-app-make_first
    sub-src-app-make_first
  sub-src-lib-make_first
    sub-src-lib-make_first
    FORCE
      FORCE
  sub-src-lib2-make_first
    sub-src-lib2-make_first
      FORCE
    FORCE
     FORCE
     FORCE
   FORCE
   </nowiki>
  FORCE
Here sub-src-app-make_first depends on sub-src-lib-make_first and sub-src-lib2-make_first. So lib and lib2 must first be built before app can be processed. Here lib and lib2 can even be processed in parallel, they have no dependencies.
</nowiki>
Here sub-src-app-make_first depends on sub-src-lib-make_first and sub-src-lib2-make_first. So lib and lib2 must first be build before app can be processed. But lib and lib2 can be processed in parallel, they have no dependencies.


Finally for the sequential case
Finally for the sequential case
  TEMPLATE = subdirs
 
  SUBDIRS = \
            src/lib2\ 
            src/lib \
            src/app
 
  # build the project sequentially as listed in SUBDIRS !
  CONFIG += ordered
 
jom's output for the graph is:
  <nowiki>
  <nowiki>
template = subdirs
   first
 
   make_first
SUBDIRS = \
          src/lib2\    
          src/lib \
          src/app
 
# build the project sequentially as listed in SUBDIRS !
CONFIG += ordered
</nowiki>
jum output for the graph is:
<nowiki>
first
make_first
  sub-src-lib2-make_first-ordered
  FORCE
  sub-src-lib-make_first-ordered
  sub-src-lib2-make_first-ordered
    FORCE
  FORCE
  sub-src-app-make_first-ordered
   sub-src-lib-make_first-ordered
     sub-src-lib2-make_first-ordered
     sub-src-lib2-make_first-ordered
    FORCE
    sub-src-lib-make_first-ordered
    sub-src-lib2-make_first-ordered
      FORCE
    FORCE
    sub-src-app-make_first-ordered
    sub-src-lib-make_first-ordered
      sub-src-lib2-make_first-ordered
      FORCE
      FORCE
     FORCE
     FORCE
     FORCE
     FORCE
   FORCE
   </nowiki>
  FORCE
So app can be built if lib was build and lib can be built if lib2 was build: first lib2, then lib, then app. No parallelism on the project level is possible.
</nowiki>
So app can be build, if lib was build and lib can be build if lib2 was build: first lib2, then lib, then app. No parallelism on the project level is possible.


With
With
Line 265: Line 286:
ten parallel compiles will be started if enough independent compiles are found.
ten parallel compiles will be started if enough independent compiles are found.


As there are no dependencies recognized in the simple SUBDIRS example we gave, the parallel processing might result in a build error due to race conditions: The linker of the app project could be started even though the necessary libraries are not build already!
==== .sln files / .vcxproj files ====
 
The standard Microsoft Visual Studio toolchain does not use Makefiles. The toolchain needs a solution file (.sln) and one ore more project files (.vcxproj). The solution is again a kind of meta project collecting the project a solution consists of.
These files can be processed by the Microsoft Visual Studio C++ IDE (MSVC2010) or with the msbuild tool as a command line tool.
 
qmake can also generate these files, to make it possible to use this IDE. To have this generated
<nowiki>
  qmake -tp vc -r
  </nowiki>
must be called in the FooApp folder containing the meta project subdirs.pro project file. qmake will create a .sln file in the same FooApp folder. qmake will also go into the SUBDIRS project folders and generate .vcxproj files that are refered to in the .sln file. In MSVS2010 IDE the .sln file can be opened a a listing of subdir projects will be listed. For the running example the file would be names: subdirs.sln, app.vcxproj, lib.vcxproj, and lib2.vcxproj.
 
With the explicit dependency giving subdir.pro file
  TEMPLATE = subdirs
 
  SUBDIRS = \
            lib2 \  # sub-project names
            lib  \
            app
 
  # where to find the sub projects - give the folders
  lib2.subdir = src/lib2
  lib.subdir  = src/lib
  app.subdir  = src/app
 
  # what subproject depends on others
  app.depends = lib lib2
 
solution and project files will be generated that reflect these dependencies.
 
===== Microsoft Visual Studio IDE =====
 
Within the IDE the solution properties contain a section "Common Properties" and in this a subsection "Project Dependencies". This subsection has a combo box to choose a project. All project this chosen on depends on are checked in the listing below.


==== .sln files / .vcxproj files ====
[[File:MSVC2010 dependency property1.PNG|frame|center|app depends on lib and lib2]]
 
This dependency constellation can also be checked in the .sln file.
<nowiki>
  Microsoft Visual Studio Solution File, Format Version 11.00
  # Visual Studio 2010
  Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "lib", "C:/Users/Michael/QtProgramme/subdirs_test/src/lib\lib.vcxproj", "{BBD7705B-EE58-39F5-9DA4-17D53DE9F205}"
  EndProject
  Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "lib2", "C:/Users/Michael/QtProgramme/subdirs_test/src/lib2\lib2.vcxproj", "{E2C38A45-30F3-3D3D-9D94-91EB9244EAF6}"
  EndProject
  Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "app", "C:/Users/Michael/QtProgramme/subdirs_test/src/app\app.vcxproj", "{124A643E-E033-36E6-AA41-AD3E268A86DE}"
  ProjectSection(ProjectDependencies) = postProject
  {BBD7705B-EE58-39F5-9DA4-17D53DE9F205} = {BBD7705B-EE58-39F5-9DA4-17D53DE9F205}
  {E2C38A45-30F3-3D3D-9D94-91EB9244EAF6} = {E2C38A45-30F3-3D3D-9D94-91EB9244EAF6}
  EndProjectSection
  EndProject
  Global
  GlobalSection(SolutionConfigurationPlatforms) = preSolution
  Debug|Win32 = Debug|Win32
  Release|Win32 = Release|Win32
  EndGlobalSection
  GlobalSection(ProjectConfigurationPlatforms) = postSolution
  {BBD7705B-EE58-39F5-9DA4-17D53DE9F205}.Debug|Win32.ActiveCfg = Debug|Win32
  {BBD7705B-EE58-39F5-9DA4-17D53DE9F205}.Debug|Win32.Build.0 = Debug|Win32
  {BBD7705B-EE58-39F5-9DA4-17D53DE9F205}.Release|Win32.ActiveCfg = Release|Win32
  {BBD7705B-EE58-39F5-9DA4-17D53DE9F205}.Release|Win32.Build.0 = Release|Win32
  {E2C38A45-30F3-3D3D-9D94-91EB9244EAF6}.Debug|Win32.ActiveCfg = Debug|Win32
  {E2C38A45-30F3-3D3D-9D94-91EB9244EAF6}.Debug|Win32.Build.0 = Debug|Win32
  {E2C38A45-30F3-3D3D-9D94-91EB9244EAF6}.Release|Win32.ActiveCfg = Release|Win32
  {E2C38A45-30F3-3D3D-9D94-91EB9244EAF6}.Release|Win32.Build.0 = Release|Win32
  {124A643E-E033-36E6-AA41-AD3E268A86DE}.Debug|Win32.ActiveCfg = Debug|Win32
  {124A643E-E033-36E6-AA41-AD3E268A86DE}.Debug|Win32.Build.0 = Debug|Win32
  {124A643E-E033-36E6-AA41-AD3E268A86DE}.Release|Win32.ActiveCfg = Release|Win32
  {124A643E-E033-36E6-AA41-AD3E268A86DE}.Release|Win32.Build.0 = Release|Win32
  EndGlobalSection
  GlobalSection(ExtensibilityGlobals) = postSolution
  EndGlobalSection
  GlobalSection(ExtensibilityAddIns) = postSolution
  EndGlobalSection
  EndGlobal
  </nowiki>
In the project app there is a section to describe the dependency.
<nowiki>
  ProjectSection(ProjectDependencies) = postProject
  {BBD7705B-EE58-39F5-9DA4-17D53DE9F205} = {BBD7705B-EE58-39F5-9DA4-17D53DE9F205}
  {E2C38A45-30F3-3D3D-9D94-91EB9244EAF6} = {E2C38A45-30F3-3D3D-9D94-91EB9244EAF6}
  EndProjectSection
  </nowiki>
 
Such a solution can be built from within the IDE. Within the IDE the projects are processed in parallel, respecting the dependencies.
 
In the build output window there will be something like
<nowiki>
  1>------ Rebuild All started: Project: lib2, Configuration: Debug Win32 ------
  2>------ Rebuild All started: Project: lib, Configuration: Debug Win32 ------
  1>  lib2.cpp
  2>  lib.cpp
  1>    Creating library debug\\lib2d.lib and object debug\\lib2d.exp
  2>    Creating library debug\\libd.lib and object debug\\libd.exp
  1>  lib2.vcxproj -> C:\Users\Michael\QtProgramme\subdirs_test\src\lib2\debug\lib2d.dll
  2>  lib.vcxproj -> C:\Users\Michael\QtProgramme\subdirs_test\src\lib\debug\libd.dll
  3>------ Rebuild All started: Project: app, Configuration: Debug Win32 ------
  3>  main.cpp
  3>  app.vcxproj -> C:\Users\Michael\QtProgramme\subdirs_test\src\app\debug\app.exe
  </nowiki>
The parallelism in processing lib and lib2 can be seen very well. After lib and lib2 finishes processing of app starts.
 
===== msbuild =====
 
From the command line the tool msbuild can be used to process the solution. In the FooApp folder, where the .sln file was generated, a call
<nowiki>
  msbuild
  </nowiki>
will process the solution file.
 
By default msbuild does not process in any parallel fashion. But similarly to joms "/J <n>" option msbuild has the "/m:<n>" option. When calling
<nowiki>
  msbuild /m:4
  </nowiki>
maximal four cores of the computer will be used to parallelize the processing of the solution.
An output from such a call might look like this for the running example
<nowiki>
  Microsoft (R)-Buildmodul, Version 4.0.30319.18408
  [Microsoft .NET Framework, Version 4.0.30319.18408]
  Copyright (C) Microsoft Corporation 2007. Alle Rechte vorbehalten.
 
  Der Buildvorgang wurde am 19.04.2015 21:37:32 gestartet.
      1>Projekt "c:\Users\Michael\QtProgramme\subdirs_test\subdirs_test.sln" auf Knoten "1" (Standardziele).
      1>ValidateSolutionConfiguration:
          Die Projektmappenkonfiguration "Debug|Win32" wird erstellt.
      1>Das Projekt "c:\Users\Michael\QtProgramme\subdirs_test\subdirs_test.sln" (1) erstellt "c:\Users\Michael\QtProgramme\subdirs_test\src\lib2\lib2.vcxproj" (3) auf Knoten "2" (Standardziele).
      3>InitializeBuildStatus:
          "debug\lib2.unsuccessfulbuild" wird erstellt, da "AlwaysCreate" angegeben wurde.
        ClCompile:
          c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\CL.exe /c /I. /IC:\Qt_online\5.4\msvc2010_opengl\include /IC:\Qt_online\5.4\msvc2010_opengl\include\QtCore /Idebug /I"C:\Qt_online\5.4\msvc2010_opengl\mkspecs\win32-msvc2010" /Zi /nologo /W3 /WX- /Od /Oy- /D _WINDOWS /D UNICODE /D WIN32 /D LIB2_LIBRARY /D QT_CORE_LIB /D _WINDLL /Gm- /EHsc /MDd /GS /fp:precise /Zc:wchar_t /Zc:forScope /GR /Fo"debug\\" /Fd"debug\vc100.pdb" /Gd /TP /analyze- /errorReport:queue lib2.cpp -Zm200 -w34100 -w34189
      1>Das Projekt "c:\Users\Michael\QtProgramme\subdirs_test\subdirs_test.sln" (1) erstellt "c:\Users\Michael\QtProgramme\subdirs_test\src\lib\lib.vcxproj" (2) auf Knoten "1" (Standardziele).
      2>InitializeBuildStatus:
          "debug\lib.unsuccessfulbuild" wird erstellt, da "AlwaysCreate" angegeben wurde.
      3>ClCompile:
          lib2.cpp
      2>ClCompile:
          c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\CL.exe /c /I. /IC:\Qt_online\5.4\msvc2010_opengl\include /IC:\Qt_online\5.4\msvc2010_opengl\include\QtCore /Idebug /I"C:\Qt_online\5.4\msvc2010_opengl\mkspecs\win32-msvc2010" /Zi /nologo /W3 /WX- /Od /Oy- /D _WINDOWS /D UNICODE /D WIN32 /D LIB_LIBRARY /D QT_CORE_LIB /D _WINDLL /Gm- /EHsc /MDd /GS /fp:precise /Zc:wchar_t /Zc:forScope /GR /Fo"debug\\" /Fd"debug\vc100.pdb" /Gd /TP /analyze- /errorReport:queue lib.cpp -Zm200 -w34100 -w34189
          lib.cpp
      3>Link:
          c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\link.exe /ERRORREPORT:QUEUE /OUT:"debug\\lib2d.dll" /NOLOGO /LIBPATH:C:\Qt_online\5.4\msvc2010_opengl\lib C:\Qt_online\5.4\msvc2010_opengl\lib\Qt5Cored.lib kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /MANIFEST /ManifestFile:"debug\lib2d.dll.intermediate.manifest" /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /DEBUG /PDB:"c:\Users\Michael\QtProgramme\subdirs_test\src\lib2\debug\lib2d.pdb" /SUBSYSTEM:WINDOWS /TLBID:1 /DYNAMICBASE /NXCOMPAT /MACHINE:X86 /DLL debug\lib2.obj
              Creating library debug\\lib2d.lib and object debug\\lib2d.exp
          lib2.vcxproj -> c:\Users\Michael\QtProgramme\subdirs_test\src\lib2\debug\lib2d.dll
        Manifest:
          C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\bin\mt.exe /nologo /verbose /outputresource:"debug\\lib2d.dll;#2" /manifest debug\lib2d.dll.intermediate.manifest
      2>Link:
          c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\link.exe /ERRORREPORT:QUEUE /OUT:"debug\\libd.dll" /NOLOGO /LIBPATH:C:\Qt_online\5.4\msvc2010_opengl\lib C:\Qt_online\5.4\msvc2010_opengl\lib\Qt5Cored.lib kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /MANIFEST /ManifestFile:"debug\libd.dll.intermediate.manifest" /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /DEBUG /PDB:"c:\Users\Michael\QtProgramme\subdirs_test\src\lib\debug\libd.pdb" /SUBSYSTEM:WINDOWS /TLBID:1 /DYNAMICBASE /NXCOMPAT /MACHINE:X86 /DLL debug\lib.obj
      3>FinalizeBuildStatus:
          Die Datei "debug\lib2.unsuccessfulbuild" wird gel"scht.
          Aktualisieren des Timestamps von "debug\lib2.lastbuildstate".
      3>Die Erstellung von Projekt "c:\Users\Michael\QtProgramme\subdirs_test\src\lib2\lib2.vcxproj" ist abgeschlossen (Standardziele).
      2>Link:
              Creating library debug\\libd.lib and object debug\\libd.exp
          lib.vcxproj -> c:\Users\Michael\QtProgramme\subdirs_test\src\lib\debug\libd.dll
        Manifest:
          C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\bin\mt.exe /nologo /verbose /outputresource:"debug\\libd.dll;#2" /manifest debug\libd.dll.intermediate.manifest
        FinalizeBuildStatus:
          Die Datei "debug\lib.unsuccessfulbuild" wird gel"scht.
          Aktualisieren des Timestamps von "debug\lib.lastbuildstate".
      2>Die Erstellung von Projekt "c:\Users\Michael\QtProgramme\subdirs_test\src\lib\lib.vcxproj" ist abgeschlossen (Standardziele).
      1>Das Projekt "c:\Users\Michael\QtProgramme\subdirs_test\subdirs_test.sln" (1) erstellt "c:\Users\Michael\QtProgramme\subdirs_test\src\app\app.vcxproj.metaproj" (4) auf Knoten "1" (Standardziele).
      4>Das Projekt "c:\Users\Michael\QtProgramme\subdirs_test\src\app\app.vcxproj.metaproj" (4) erstellt "c:\Users\Michael\QtProgramme\subdirs_test\src\app\app.vcxproj" (5) auf Knoten "2" (Standardziele).
      5>InitializeBuildStatus:
          "debug\app.unsuccessfulbuild" wird erstellt, da "AlwaysCreate" angegeben wurde.
        ClCompile:
          c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\CL.exe /c /I. /I..\lib /IC:\Qt_online\5.4\msvc2010_opengl\include /IC:\Qt_online\5.4\msvc2010_opengl\include\QtCore /Idebug /I"C:\Qt_online\5.4\msvc2010_opengl\mkspecs\win32-msvc2010" /Zi /nologo /W3 /WX- /Od /Oy- /D _CONSOLE /D UNICODE /D WIN32 /D QT_CORE_LIB /Gm- /EHsc /MDd /GS /fp:precise /Zc:wchar_t /Zc:forScope /GR /Fo"debug\\" /Fd"debug\vc100.pdb" /Gd /TP /analyze- /errorReport:queue main.cpp -Zm200 -w34100 -w34189
          main.cpp
        Link:
          c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\link.exe /ERRORREPORT:QUEUE /OUT:"debug\\app.exe" /NOLOGO /LIBPATH:C:\Users\Michael\QtProgramme\subdirs_test\src\app\..\lib\debug\ /LIBPATH:C:\Users\Michael\QtProgramme\subdirs_test\src\app\..\lib2\debug\ /LIBPATH:C:\Qt_online\5.4\msvc2010_opengl\lib C:\Users\Michael\QtProgramme\subdirs_test\src\app\..\lib\debug\\libd.lib C:\Users\Michael\QtProgramme\subdirs_test\src\app\..\lib2\debug\\lib2d.lib C:\Qt_online\5.4\msvc2010_opengl\lib\Qt5Cored.lib kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /MANIFEST /ManifestFile:"debug\app.exe.intermediate.manifest" /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /DEBUG /PDB:"c:\Users\Michael\QtProgramme\subdirs_test\src\app\debug\app.pdb" /SUBSYSTEM:CONSOLE /TLBID:1 /DYNAMICBASE /NXCOMPAT /MACHINE:X86 debug\main.obj "/MANIFESTDEPENDENCY:type='win32' name='Microsoft.Windows.Common-Controls' version='6.0.0.0' publicKeyToken='6595b64144ccf1df' language='*' processorArchitecture='*'"
          app.vcxproj -> c:\Users\Michael\QtProgramme\subdirs_test\src\app\debug\app.exe
        Manifest:
          C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\bin\mt.exe /nologo /verbose /outputresource:"debug\\app.exe;#1" /manifest debug\app.exe.intermediate.manifest
        FinalizeBuildStatus:
          Die Datei "debug\app.unsuccessfulbuild" wird gel"scht.
          Aktualisieren des Timestamps von "debug\app.lastbuildstate".
      5>Die Erstellung von Projekt "c:\Users\Michael\QtProgramme\subdirs_test\src\app\app.vcxproj" ist abgeschlossen (Standardziele).
      4>Die Erstellung von Projekt "c:\Users\Michael\QtProgramme\subdirs_test\src\app\app.vcxproj.metaproj" ist abgeschlossen (Standardziele).
      1>Die Erstellung von Projekt "c:\Users\Michael\QtProgramme\subdirs_test\subdirs_test.sln" ist abgeschlossen (Standardziele).
 
  Der Buildvorgang wurde erfolgreich ausgefhrt.
      0 Warnung(en)
      0 Fehler
 
  Verstrichene Zeit 00:00:02.23
  </nowiki>
 
 
 
msbuild has many [https://msdn.microsoft.com/en-us/library/ms164311%28v=vs.100%29.aspx command line options].
For example, to let msbuild construct the release version the command
<nowiki>
  msbuild /p:Configuration=Release
  </nowiki>
must be called.


The standard Microsoft Visual Studio toolchain has another method to define the operation that has to be done to compile a project or meta project. The .sln solution files and the .vcxproj project files are used. These files can be processed by the MSVS2010 IDE or with the msbuild tool as a command line tool.


to be continued....
to be continued....


--[[User:Moellney|Moellney]] ([[User talk:Moellney|talk]]) 22:30, 18 April 2015 (UTC)
--[[User:Moellney|Moellney]] ([[User talk:Moellney|talk]]) 21:48, 20 April 2015 (UTC)

Latest revision as of 09:16, 13 July 2019

Introduction

It is common for larger software projects to separate its components into different topics. There might be components the are relevant to the GUI of a software, other might be in the topic of handling geometrical processing, others are in the topic of generating a report.

Typically the software components belonging to a topic are combined into a software library (Wikipedia). In Windows these libraries are often found as .lib or .dll files; under Linux there are .a and .so files; under MacOS X they are called .a or .dylib files.

Finally these libraries are linked to an executable application.

The following text uses a small example, consisting of an application: app and two libraries: lib and lib2. The application app uses features of the libraries lib and lib2 while the libraries are in no direct relation. So, to have a running application the toolchain must create the lib.lib and lib2.lib files and finally create the app.exe by linking the lib.lib and lib2.lib to the object files of the app components. (The following Text uses the Windows file name notations of the objects.)

When the software project is in an early design and implementation phase, the libraries might change as often as the application using the libraries. In the Qt domain, each of the three topics: lib, lib2 and app will be handled by one project each. Typically there will be three folders: lib, lib2 and app, each containing a specific .pro file, describing how to handle each project (which source files, compile parameters, ...). The profiles typically will be named after the folders they live in: lib.pro, lib2.pro and app.pro. For this example all three projects will live in one folder: src.

On the file system this can be seen as following hierarchy: 

   /src
   |--- app
   |    |--- app.pro
   |    `--- ... (source files of app)
   |--- lib
   |    |--- lib.pro
   |    `--- ... (source files of lib)
   `--- lib2
        |--- lib2.pro
        `--- ... (source files of lib2)

Each of this projects could for example be edited separately in three Microsoft Visual Studio 2010 IDEs (MSVS2010) or handled as three separate projects in one QtCreator. But this handling of the projects would mean to compile each by hand separately. It gets tedious when taking into account the dependencies between the three projects: the libs must be compiled before the app can be created!

template: subdirs

To handle this relation automatically qmake has a template type: subdirs (template).

The basic idea of this template type is to list all projects that belong to a kind of meta project. A new .pro file is created for that meta project that simply consists of the qmake system variable SUBDIRS. This variable is filled with the names of the project folders by giving the relative path to where the meta .pro file lies. In the following text this meta profile is named subdirs.pro. For a better overview in the file hierarchy a new parent folder is introduced, named after the software project that is created: FooApp. 

   /FooApp
   |--- subdirs.pro
   `--- src
        |
        |--- app
        |    |--- app.pro
        |    `--- ... (source files of app)
        |--- lib
        |    |--- lib.pro
        |    `--- ... (source files of lib)
        `--- lib2
             |--- lib2.pro
             `--- ... (source files of lib2)

The meta profile subdirs.pro is on the same level as the src folder.

The content of the subdirs.pro is 

  TEMPLATE = subdirs
 
  SUBDIRS = \
            src/app \   # relative paths
            src/lib \
            src/lib2

The subdirs.pro file can be opened by QtCreator. In QtCreator there will be just one project with child elements: the sub-project. The advantage now is that when compiling this meta project the sub-projects are compiled automatically. Asking for a qmake run within QtCreator on this meta project will also recurse into the sub-projects calling qmake for each.

To make this meta project available for MSVC2010, qmake must be called on the command line in the folder FooApp: 

qmake -tp vc -r

This will generate a solution file with the name subdirs.sln. When opening this file in MSVC2010 a solution will be opened that contain the three project: app, lib, and lib2. A compile on the solution will call compile for each of the contained projects.

Attention: The order in which the subprojects are compiled is not defined by the SUBDIRS content. This could lead to a wrong processing order. So, the app project could not be built if it is processed first; it would miss the libraries lib and lib2. One could assume that the build order is given by the order of the entities in the SUBDIRS variable (so example given above would be wrong). But many modern compile tools offer parallel compilation. If the order of the entity would be forcing the build order, then parallel compilation on the projects level would not be possible. Per default the sub-projects are compiled in any order the used build-systems logic prefers.

Defining Project Dependencies: .depends attributes

Attention Visual Studio This method is very fragile implemented in qmake when generating the Windows Visual Studio files (.sln and .vcxproj). Qt 4.8.x does not support it at all and Qt 5.4.1 must have special setting in the sub projects pro files. A bug was filled in QTBUG-45706.

To support parallel processing of the projects on modern computers, additional information must be given: Which project must be finished before another project can be processed.

In the running example, we have noted that app can only be built when both lib and lib2 are built.

By specifying the dependency explicitly in the meta project file, qmake can build processing instructions that allow a high degree of parallelism. 


  TEMPLATE = subdirs
 
  SUBDIRS = \
            lib2 \   # sub-project names
            lib  \
            app
 
  # where to find the sub projects - give the folders
  lib2.subdir = src/lib2
  lib.subdir  = src/lib
  app.subdir  = src/app
 
  # what subproject depends on others
  app.depends = lib lib2

The important addition to the file is the line app.depends = lib lib2. Setting the .depends attribute of the project app tells qmake that app can only be built if lib and lib2 is processed.

This allows qmake to build processing instructions that can be executed in a high degree of parallelism. For example lib and lib2 can be built in parallel, while app is held back until lib and lib2 processing is finished.

In addition there is an implicit introduction of projects names. In SUBDIRS the project names are given (not the path anymore). The path to the project directory is set below by setting the .subdir attribute to the relative path where the project folder is located.

Defining a processing sequence: CONFIG += ordered

Attention: Using CONFIG += ordered is not recommended as it hinders effective multicore builds. Using .depends is the correct way as written in this [[ https://blog.rburchell.com/2013/10/every-time-you-configordered-kitten-dies.html%7Cblog]].

This section is kept only as there may be a bug when creating Visual Studio projects and specifying dependencies may otherwise not work.

Attention Visual Studio: This feature is not supported under Qt 4.8.x (Qt Docs 4.8). Instead an automatic dependency detection scheme is used in the Microsoft Visual Studio file generation (.sln and .vcxproj). In the Qt5 documentation, the section describing the automatism is deleted! A small test showed that the ordered option as show here does still not work for Qt 5.4.1. A bug report was filled in QTBUG-45706.

With this option the processing of the sub-project can be set to follow the order in which they are listed in the SUBDIRS variable. 

  TEMPLATE = subdirs
 
  SUBDIRS = \
            src/lib2\   
            src/lib \
            src/app
 
  # build the project sequentially as listed in SUBDIRS !
  CONFIG += ordered

By having the listing order: src/lib2, src/lib, src/app the build tool will follow that order and build the project one at a time. Due to processing in this strict project order the build dependencies are implicitly fulfilled, as long as the listing order is chosen correctly.

But processing "one at a time" eliminates the possibility to process independent projects in parallel. However each projects might process in parallel in its local context - e.g. the .cpp files within lib2 might get compiled in parallel.

Background on Windows Build Systems

There are different ways to process a project (or meta project) under windows.

Makefiles

There are command line tools to compile project with the help of a compile rule file: a Makefile (wikipedia).

For a meta-project, qmake must be called with the recursive option in the FooApp folder, the one with the subdirs.pro file: 

qmake -r

This will generate a Makefile in the same directory. In addition qmake will process all the subdirs given in the SUBDIRS variable and create Makefiles in each of the sub-project folders, containing rules to create the sub-project. 

   /FooApp
   |--- Makefile
   |--- subdirs.pro
   `--- src
        |
        |--- app
        |    |--- app.pro
        |    |--- Makefile
        |    |--- Makefile.Debug
        |    |--- Makefile.Release
        |    `--- ... (source files of app)
        |--- lib
        |    |--- lib.pro
        |    |--- Makefile
        |    |--- Makefile.Debug
        |    |--- Makefile.Release
        |    `--- ... (source files of lib)
        `--- lib2
             |--- lib2.pro
             |--- Makefile
             |--- Makefile.Debug
             |--- Makefile.Release
             `--- ... (source files of lib2)
nmake

The windows nmake tool reads a Windows Makefile that was generated by qmake. nmake will understand the rules in the Makefile and will call the compiler and linker to build the libraries and application.

Calling nmake in the meta project folder FooApp via 

nmake

will find the Makefile and will process it. As this Makefile is one for a meta-project, it contains rules to process the Makefiles of the sub-project.

nmake has no feature to call compile or link in parallel. An option "-j <n>" as known with Unix make or gnumake does not exist. So nmake start project after project and in each project compiles file after file. Probably the listing order of the sub-projects within the SUBDIRS variable will be used. But this is nothing to rely on! How to determine the processing order sufficiently is shown in the chapters above.

jom

The missing parallel execution feature of nmake is inefficient on modern computer architectures. With CPUs having many kernels for parallelism and fast storage units, a modern computer is bored by using nmake and project compilation takes longer as it could take. This lead to the creation of a new make tool: Jom. jom is basically an nmake clone extended with the '/J <n>' option to compile in parallel. To compile a meta projects Makefile the call has to be 

jom

in the FooApp folder. jom will recurse automatically in the sub-project folders and use the Makefile of the sub-folders, but all in a parallel fashion. To see the dependency graph that jom processes, the command 

jom /DUMPGRAPH

can be called. For the insufficient and wrongly ordered example 

  TEMPLATE = subdirs
 
  SUBDIRS = \
            src/app \   # relative paths
            src/lib \
            src/lib2

the output of jom is 

  first
   make_first
    sub-src-app-make_first
     FORCE
    sub-src-lib-make_first
     FORCE
    sub-src-lib2-make_first
     FORCE
    FORCE

All three sub-src-xxx-make_first are on the same level and they have no dependency under it (aside from FORCE). So all is build in parallel if the number of kernels is high enough. The parallel processing might result in a build error due to race conditions: The linker of the app project could be started even though the necessary libraries are not build already!

For an example with dependency given explicitly: 

  TEMPLATE = subdirs
 
  SUBDIRS = \
            lib2 \   # sub-project names
            lib  \
            app
 
  # where to find the sub projects - give the folders
  lib2.subdir = src/lib2
  lib.subdir  = src/lib
  app.subdir  = src/app
 
  # what subproject depends on others
  app.depends = lib lib2

jom output for the graph: 

  first
   make_first
    sub-src-lib-make_first
     FORCE
    sub-src-lib2-make_first
     FORCE
    sub-src-app-make_first
     sub-src-lib-make_first
      FORCE
     sub-src-lib2-make_first
      FORCE
     FORCE
    FORCE

Here sub-src-app-make_first depends on sub-src-lib-make_first and sub-src-lib2-make_first. So lib and lib2 must first be built before app can be processed. Here lib and lib2 can even be processed in parallel, they have no dependencies.

Finally for the sequential case 

  TEMPLATE = subdirs
 
  SUBDIRS = \
            src/lib2\   
            src/lib \
            src/app
 
  # build the project sequentially as listed in SUBDIRS !
  CONFIG += ordered

jom's output for the graph is: 

  first
   make_first
    sub-src-lib2-make_first-ordered
     FORCE
    sub-src-lib-make_first-ordered
     sub-src-lib2-make_first-ordered
      FORCE
     FORCE
    sub-src-app-make_first-ordered
     sub-src-lib-make_first-ordered
      sub-src-lib2-make_first-ordered
       FORCE
      FORCE
     FORCE
    FORCE

So app can be built if lib was build and lib can be built if lib2 was build: first lib2, then lib, then app. No parallelism on the project level is possible.

With 

jom /J 10

ten parallel compiles will be started if enough independent compiles are found.

.sln files / .vcxproj files

The standard Microsoft Visual Studio toolchain does not use Makefiles. The toolchain needs a solution file (.sln) and one ore more project files (.vcxproj). The solution is again a kind of meta project collecting the project a solution consists of. These files can be processed by the Microsoft Visual Studio C++ IDE (MSVC2010) or with the msbuild tool as a command line tool.

qmake can also generate these files, to make it possible to use this IDE. To have this generated 

   qmake -tp vc -r

must be called in the FooApp folder containing the meta project subdirs.pro project file. qmake will create a .sln file in the same FooApp folder. qmake will also go into the SUBDIRS project folders and generate .vcxproj files that are refered to in the .sln file. In MSVS2010 IDE the .sln file can be opened a a listing of subdir projects will be listed. For the running example the file would be names: subdirs.sln, app.vcxproj, lib.vcxproj, and lib2.vcxproj.

With the explicit dependency giving subdir.pro file 

  TEMPLATE = subdirs
 
  SUBDIRS = \
            lib2 \   # sub-project names
            lib  \
            app
 
  # where to find the sub projects - give the folders
  lib2.subdir = src/lib2
  lib.subdir  = src/lib
  app.subdir  = src/app
 
  # what subproject depends on others
  app.depends = lib lib2

solution and project files will be generated that reflect these dependencies.

Microsoft Visual Studio IDE

Within the IDE the solution properties contain a section "Common Properties" and in this a subsection "Project Dependencies". This subsection has a combo box to choose a project. All project this chosen on depends on are checked in the listing below.

app depends on lib and lib2

This dependency constellation can also be checked in the .sln file. 

  Microsoft Visual Studio Solution File, Format Version 11.00
  # Visual Studio 2010
  Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "lib", "C:/Users/Michael/QtProgramme/subdirs_test/src/lib\lib.vcxproj", "{BBD7705B-EE58-39F5-9DA4-17D53DE9F205}"
  EndProject
  Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "lib2", "C:/Users/Michael/QtProgramme/subdirs_test/src/lib2\lib2.vcxproj", "{E2C38A45-30F3-3D3D-9D94-91EB9244EAF6}"
  EndProject
  Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "app", "C:/Users/Michael/QtProgramme/subdirs_test/src/app\app.vcxproj", "{124A643E-E033-36E6-AA41-AD3E268A86DE}"
  	ProjectSection(ProjectDependencies) = postProject
  		{BBD7705B-EE58-39F5-9DA4-17D53DE9F205} = {BBD7705B-EE58-39F5-9DA4-17D53DE9F205}
  		{E2C38A45-30F3-3D3D-9D94-91EB9244EAF6} = {E2C38A45-30F3-3D3D-9D94-91EB9244EAF6}
  	EndProjectSection
  EndProject
  Global
  	GlobalSection(SolutionConfigurationPlatforms) = preSolution
  		Debug|Win32 = Debug|Win32
  		Release|Win32 = Release|Win32
  	EndGlobalSection
  	GlobalSection(ProjectConfigurationPlatforms) = postSolution
  		{BBD7705B-EE58-39F5-9DA4-17D53DE9F205}.Debug|Win32.ActiveCfg = Debug|Win32
  		{BBD7705B-EE58-39F5-9DA4-17D53DE9F205}.Debug|Win32.Build.0 = Debug|Win32
  		{BBD7705B-EE58-39F5-9DA4-17D53DE9F205}.Release|Win32.ActiveCfg = Release|Win32
  		{BBD7705B-EE58-39F5-9DA4-17D53DE9F205}.Release|Win32.Build.0 = Release|Win32
  		{E2C38A45-30F3-3D3D-9D94-91EB9244EAF6}.Debug|Win32.ActiveCfg = Debug|Win32
  		{E2C38A45-30F3-3D3D-9D94-91EB9244EAF6}.Debug|Win32.Build.0 = Debug|Win32
  		{E2C38A45-30F3-3D3D-9D94-91EB9244EAF6}.Release|Win32.ActiveCfg = Release|Win32
  		{E2C38A45-30F3-3D3D-9D94-91EB9244EAF6}.Release|Win32.Build.0 = Release|Win32
  		{124A643E-E033-36E6-AA41-AD3E268A86DE}.Debug|Win32.ActiveCfg = Debug|Win32
  		{124A643E-E033-36E6-AA41-AD3E268A86DE}.Debug|Win32.Build.0 = Debug|Win32
  		{124A643E-E033-36E6-AA41-AD3E268A86DE}.Release|Win32.ActiveCfg = Release|Win32
  		{124A643E-E033-36E6-AA41-AD3E268A86DE}.Release|Win32.Build.0 = Release|Win32
  	EndGlobalSection
  	GlobalSection(ExtensibilityGlobals) = postSolution
  	EndGlobalSection
  	GlobalSection(ExtensibilityAddIns) = postSolution
  	EndGlobalSection
  EndGlobal

In the project app there is a section to describe the dependency. 

  ProjectSection(ProjectDependencies) = postProject
  		{BBD7705B-EE58-39F5-9DA4-17D53DE9F205} = {BBD7705B-EE58-39F5-9DA4-17D53DE9F205}
  		{E2C38A45-30F3-3D3D-9D94-91EB9244EAF6} = {E2C38A45-30F3-3D3D-9D94-91EB9244EAF6}
  	EndProjectSection

Such a solution can be built from within the IDE. Within the IDE the projects are processed in parallel, respecting the dependencies.

In the build output window there will be something like 

   1>------ Rebuild All started: Project: lib2, Configuration: Debug Win32 ------
   2>------ Rebuild All started: Project: lib, Configuration: Debug Win32 ------
   1>  lib2.cpp
   2>  lib.cpp
   1>     Creating library debug\\lib2d.lib and object debug\\lib2d.exp
   2>     Creating library debug\\libd.lib and object debug\\libd.exp
   1>  lib2.vcxproj -> C:\Users\Michael\QtProgramme\subdirs_test\src\lib2\debug\lib2d.dll
   2>  lib.vcxproj -> C:\Users\Michael\QtProgramme\subdirs_test\src\lib\debug\libd.dll
   3>------ Rebuild All started: Project: app, Configuration: Debug Win32 ------
   3>  main.cpp
   3>  app.vcxproj -> C:\Users\Michael\QtProgramme\subdirs_test\src\app\debug\app.exe

The parallelism in processing lib and lib2 can be seen very well. After lib and lib2 finishes processing of app starts.

msbuild

From the command line the tool msbuild can be used to process the solution. In the FooApp folder, where the .sln file was generated, a call 

   msbuild

will process the solution file.

By default msbuild does not process in any parallel fashion. But similarly to joms "/J <n>" option msbuild has the "/m:<n>" option. When calling 

   msbuild /m:4

maximal four cores of the computer will be used to parallelize the processing of the solution. An output from such a call might look like this for the running example 

  Microsoft (R)-Buildmodul, Version 4.0.30319.18408
  [Microsoft .NET Framework, Version 4.0.30319.18408]
  Copyright (C) Microsoft Corporation 2007. Alle Rechte vorbehalten.
  
  Der Buildvorgang wurde am 19.04.2015 21:37:32 gestartet.
       1>Projekt "c:\Users\Michael\QtProgramme\subdirs_test\subdirs_test.sln" auf Knoten "1" (Standardziele).
       1>ValidateSolutionConfiguration:
           Die Projektmappenkonfiguration "Debug|Win32" wird erstellt.
       1>Das Projekt "c:\Users\Michael\QtProgramme\subdirs_test\subdirs_test.sln" (1) erstellt "c:\Users\Michael\QtProgramme\subdirs_test\src\lib2\lib2.vcxproj" (3) auf Knoten "2" (Standardziele).
       3>InitializeBuildStatus:
           "debug\lib2.unsuccessfulbuild" wird erstellt, da "AlwaysCreate" angegeben wurde.
         ClCompile:
           c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\CL.exe /c /I. /IC:\Qt_online\5.4\msvc2010_opengl\include /IC:\Qt_online\5.4\msvc2010_opengl\include\QtCore /Idebug /I"C:\Qt_online\5.4\msvc2010_opengl\mkspecs\win32-msvc2010" /Zi /nologo /W3 /WX- /Od /Oy- /D _WINDOWS /D UNICODE /D WIN32 /D LIB2_LIBRARY /D QT_CORE_LIB /D _WINDLL /Gm- /EHsc /MDd /GS /fp:precise /Zc:wchar_t /Zc:forScope /GR /Fo"debug\\" /Fd"debug\vc100.pdb" /Gd /TP /analyze- /errorReport:queue lib2.cpp -Zm200 -w34100 -w34189 
       1>Das Projekt "c:\Users\Michael\QtProgramme\subdirs_test\subdirs_test.sln" (1) erstellt "c:\Users\Michael\QtProgramme\subdirs_test\src\lib\lib.vcxproj" (2) auf Knoten "1" (Standardziele).
       2>InitializeBuildStatus:
           "debug\lib.unsuccessfulbuild" wird erstellt, da "AlwaysCreate" angegeben wurde.
       3>ClCompile:
           lib2.cpp
       2>ClCompile:
           c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\CL.exe /c /I. /IC:\Qt_online\5.4\msvc2010_opengl\include /IC:\Qt_online\5.4\msvc2010_opengl\include\QtCore /Idebug /I"C:\Qt_online\5.4\msvc2010_opengl\mkspecs\win32-msvc2010" /Zi /nologo /W3 /WX- /Od /Oy- /D _WINDOWS /D UNICODE /D WIN32 /D LIB_LIBRARY /D QT_CORE_LIB /D _WINDLL /Gm- /EHsc /MDd /GS /fp:precise /Zc:wchar_t /Zc:forScope /GR /Fo"debug\\" /Fd"debug\vc100.pdb" /Gd /TP /analyze- /errorReport:queue lib.cpp -Zm200 -w34100 -w34189 
           lib.cpp
       3>Link:
           c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\link.exe /ERRORREPORT:QUEUE /OUT:"debug\\lib2d.dll" /NOLOGO /LIBPATH:C:\Qt_online\5.4\msvc2010_opengl\lib C:\Qt_online\5.4\msvc2010_opengl\lib\Qt5Cored.lib kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /MANIFEST /ManifestFile:"debug\lib2d.dll.intermediate.manifest" /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /DEBUG /PDB:"c:\Users\Michael\QtProgramme\subdirs_test\src\lib2\debug\lib2d.pdb" /SUBSYSTEM:WINDOWS /TLBID:1 /DYNAMICBASE /NXCOMPAT /MACHINE:X86 /DLL debug\lib2.obj
              Creating library debug\\lib2d.lib and object debug\\lib2d.exp
           lib2.vcxproj -> c:\Users\Michael\QtProgramme\subdirs_test\src\lib2\debug\lib2d.dll
         Manifest:
           C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\bin\mt.exe /nologo /verbose /outputresource:"debug\\lib2d.dll;#2" /manifest debug\lib2d.dll.intermediate.manifest
       2>Link:
           c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\link.exe /ERRORREPORT:QUEUE /OUT:"debug\\libd.dll" /NOLOGO /LIBPATH:C:\Qt_online\5.4\msvc2010_opengl\lib C:\Qt_online\5.4\msvc2010_opengl\lib\Qt5Cored.lib kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /MANIFEST /ManifestFile:"debug\libd.dll.intermediate.manifest" /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /DEBUG /PDB:"c:\Users\Michael\QtProgramme\subdirs_test\src\lib\debug\libd.pdb" /SUBSYSTEM:WINDOWS /TLBID:1 /DYNAMICBASE /NXCOMPAT /MACHINE:X86 /DLL debug\lib.obj
       3>FinalizeBuildStatus:
           Die Datei "debug\lib2.unsuccessfulbuild" wird gel"scht.
           Aktualisieren des Timestamps von "debug\lib2.lastbuildstate".
       3>Die Erstellung von Projekt "c:\Users\Michael\QtProgramme\subdirs_test\src\lib2\lib2.vcxproj" ist abgeschlossen (Standardziele).
       2>Link:
              Creating library debug\\libd.lib and object debug\\libd.exp
           lib.vcxproj -> c:\Users\Michael\QtProgramme\subdirs_test\src\lib\debug\libd.dll
         Manifest:
           C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\bin\mt.exe /nologo /verbose /outputresource:"debug\\libd.dll;#2" /manifest debug\libd.dll.intermediate.manifest
         FinalizeBuildStatus:
           Die Datei "debug\lib.unsuccessfulbuild" wird gel"scht.
           Aktualisieren des Timestamps von "debug\lib.lastbuildstate".
       2>Die Erstellung von Projekt "c:\Users\Michael\QtProgramme\subdirs_test\src\lib\lib.vcxproj" ist abgeschlossen (Standardziele).
       1>Das Projekt "c:\Users\Michael\QtProgramme\subdirs_test\subdirs_test.sln" (1) erstellt "c:\Users\Michael\QtProgramme\subdirs_test\src\app\app.vcxproj.metaproj" (4) auf Knoten "1" (Standardziele).
       4>Das Projekt "c:\Users\Michael\QtProgramme\subdirs_test\src\app\app.vcxproj.metaproj" (4) erstellt "c:\Users\Michael\QtProgramme\subdirs_test\src\app\app.vcxproj" (5) auf Knoten "2" (Standardziele).
       5>InitializeBuildStatus:
           "debug\app.unsuccessfulbuild" wird erstellt, da "AlwaysCreate" angegeben wurde.
         ClCompile:
           c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\CL.exe /c /I. /I..\lib /IC:\Qt_online\5.4\msvc2010_opengl\include /IC:\Qt_online\5.4\msvc2010_opengl\include\QtCore /Idebug /I"C:\Qt_online\5.4\msvc2010_opengl\mkspecs\win32-msvc2010" /Zi /nologo /W3 /WX- /Od /Oy- /D _CONSOLE /D UNICODE /D WIN32 /D QT_CORE_LIB /Gm- /EHsc /MDd /GS /fp:precise /Zc:wchar_t /Zc:forScope /GR /Fo"debug\\" /Fd"debug\vc100.pdb" /Gd /TP /analyze- /errorReport:queue main.cpp -Zm200 -w34100 -w34189 
           main.cpp
         Link:
           c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\link.exe /ERRORREPORT:QUEUE /OUT:"debug\\app.exe" /NOLOGO /LIBPATH:C:\Users\Michael\QtProgramme\subdirs_test\src\app\..\lib\debug\ /LIBPATH:C:\Users\Michael\QtProgramme\subdirs_test\src\app\..\lib2\debug\ /LIBPATH:C:\Qt_online\5.4\msvc2010_opengl\lib C:\Users\Michael\QtProgramme\subdirs_test\src\app\..\lib\debug\\libd.lib C:\Users\Michael\QtProgramme\subdirs_test\src\app\..\lib2\debug\\lib2d.lib C:\Qt_online\5.4\msvc2010_opengl\lib\Qt5Cored.lib kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /MANIFEST /ManifestFile:"debug\app.exe.intermediate.manifest" /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /DEBUG /PDB:"c:\Users\Michael\QtProgramme\subdirs_test\src\app\debug\app.pdb" /SUBSYSTEM:CONSOLE /TLBID:1 /DYNAMICBASE /NXCOMPAT /MACHINE:X86 debug\main.obj "/MANIFESTDEPENDENCY:type='win32' name='Microsoft.Windows.Common-Controls' version='6.0.0.0' publicKeyToken='6595b64144ccf1df' language='*' processorArchitecture='*'" 
           app.vcxproj -> c:\Users\Michael\QtProgramme\subdirs_test\src\app\debug\app.exe
         Manifest:
           C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\bin\mt.exe /nologo /verbose /outputresource:"debug\\app.exe;#1" /manifest debug\app.exe.intermediate.manifest
         FinalizeBuildStatus:
           Die Datei "debug\app.unsuccessfulbuild" wird gel"scht.
           Aktualisieren des Timestamps von "debug\app.lastbuildstate".
       5>Die Erstellung von Projekt "c:\Users\Michael\QtProgramme\subdirs_test\src\app\app.vcxproj" ist abgeschlossen (Standardziele).
       4>Die Erstellung von Projekt "c:\Users\Michael\QtProgramme\subdirs_test\src\app\app.vcxproj.metaproj" ist abgeschlossen (Standardziele).
       1>Die Erstellung von Projekt "c:\Users\Michael\QtProgramme\subdirs_test\subdirs_test.sln" ist abgeschlossen (Standardziele).
  
  Der Buildvorgang wurde erfolgreich ausgefhrt.
      0 Warnung(en)
      0 Fehler
  
  Verstrichene Zeit 00:00:02.23


msbuild has many command line options. For example, to let msbuild construct the release version the command 

   msbuild /p:Configuration=Release

must be called.


to be continued....

--Moellney (talk) 21:48, 20 April 2015 (UTC)

Retrieved from "https://wiki.qt.io/index.php?title=SUBDIRS_-_handling_dependencies&oldid=35827"