SUBDIRS - handling dependencies

Introduction

It is common for larger software projects to separate its components into different topics. There might be components the are relevant to the GUI of a software, other might be in the topic of handling geometrical processing, others are in the topic of generating a report.

Typically the software components belonging to a topic are combined into a software library (Wikipedia). In Windows these libraries are often found as .lib or .dll files; under Linux there are .a and .so files; under MacOS X they are called .a or .dylib files.

Finally these libraries are linked to an executable application.

The following text uses a small example, consisting of an application: app and two libraries: lib and lib2. The application app uses features of the libraries lib and lib2 while the libraries are in no direct relation. So, to have a running application the toolchain must create the lib.lib and lib2.lib files and finally create the app.exe by linking the lib.lib and lib2.lib to the object files of the app components. (The following Text uses the Windows file name notations of the objects)

When the software project is in an early design and implementation phase, the libraries might change as often as the application using the libraries. In the Qt domain, each of the three topics: lib, lib2 and app will be handled bye one project each. Typically there will be three folders: lib, lib2 and app, each containing a specific .pro file, describing how to handle each project (which source files, compile parameters, ...). The profiles typically will be named after the folders they live in: lib.pro, lib2.pro and app.pro. For this example all three projects will live in one folder: src.

On the file system this can be seen as following hierarchy:

 /src
 |--- app
 |    |--- app.pro
 |    `--- ... (source files of app)
 |--- lib
 |    |--- lib.pro
 |    `--- ... (source files of lib)
 `--- lib2
      |--- lib2.pro
      `--- ... (source files of lib2)
 

Each of this projects could for example be edited separately in three Microsoft Visual Studio 2010 IDEs (MSVS2010) or handled as three separate projects in one QtCreator. But this handling of the projects would mean to compile each by hand separately. It gets tedious when taking into account the dependencies between the three projects: the libs must be compiled before the app can be created!

template: subdirs

To handle this relation automatically qmake has a template type: subdirs (template).

The basic idea of this template type is, to list all projects that belong to a kind of meta project. A new .pro file is created for that meta project, that simply consists of the qmake system variable SUBDIRS. This variable is filled with the names of the project folders by giving the relative path to where the meta .pro file lies. In the following text this meta profile is names subdirs.pro. For a better overview in the file hierarchy a new parent folder is introduced, named after the software project that is created: FooApp.

 /FooApp
 |--- subdirs.pro
 `--- src
      |
      |--- app
      |    |--- app.pro
      |    `--- ... (source files of app)
      |--- lib
      |    |--- lib.pro
      |    `--- ... (source files of lib)
      `--- lib2
           |--- lib2.pro
           `--- ... (source files of lib2)
 

The meta profile subdirs.pro is on the same level as the src folder.

The content of the subdirs.pro is

 template = subdirs

 SUBDIRS = \
           src/app \   # relative paths
           src/lib \
           src/lib2
 

The subdirs.pro file can be opened by QtCreator. In QtCreator there will be just one project with child elements, the sub-project. The advantage is now that when compiling this meta project the sub-projects are compiled automatically. Asking for a qmake run within QtCreator on this meta project will also recurse into the sub-projects calling qmake for each.

To make this meta project available for MSVC2010, qmake must be called on the command line in the folder FooApp:

qmake -tp vc -r

This will generate a solution file with the name subdirs.sln. When opening this file in MSVC2010 a solution will be opened that contain the three project: app, lib, and lib2. A compile on the solution will call compile for each of the contained projects.

Attention: The order in with the subproject are compiled is not defined by the SUBDIRS content. And this could lead to a wrong processing order. So, the app project could not be build if it is processed first; it would miss the libraries lib and lib2. One could assume that the build order is given by the order of the entities in the SUBDIRS variable (so example given above would be wrong). But many modern compile tools offer parallel compilation. If the order of the entity would be forcing the build order, then parallel compilation on the projects level would not be possible. Per default the sub-projects are compiled in any order the used build-systems logic prefers.

Defining Project Dependencies: .depends attributes

To support parallel processing of the projects on modern computers, an additional information must be given: Which project must be finished before another project can be processed.

In this example we had noted that app can only be build, when lib and lib2 are build. There are two ways to tell qmake this dependency.

By telling the dependency explicitly in the meta project file, qmake can build processing instructions that allow a high degree of parallelism.

 template = subdirs

 SUBDIRS = \
           lib2 \   # sub-project names
           lib  \
           app

 # where to find the sub projects - give the folders
 lib2.subdir = src/lib2
 lib.subdir  = src/lib
 app.subdir  = src/app

 # what subproject depends on others
 app.depends = lib lib2
 

Defining a processing sequence: CONFIG += ordered

With this option the processing of the sub-project can be set to follow the order in which they are listed in the SUBDIRS variable.

 template = subdirs

 SUBDIRS = \
           src/lib2\   
           src/lib \
           src/app

 # build the project sequentially as listed in SUBDIRS !
 CONFIG += ordered
 

By having the listing order: src/lib2, src/lib, src/app the build tool will follow that order and build the project one at a time. This eliminates the possibility to process independent projects in parallel. However each projects might process in parallel in its local context - e.g. the .cpp files within lib2 might get compiled in parallel.

Background on Windows Build Systems

There are different ways to compile a project (or meta project) under windows.

Makefiles

There are command line tools to compile project with the help of a compile rule file: a Makefile.

For a meta-project qmake must be called with the recursive flag in the FooApp folder, the one with the subdirs.pro file:

qmake -r

This will generate a Makefile in the same directory. In addition qmake will process all the subdirs given in the SUBDIRS variable and create Makefiles in each of the sub-project folders, containing rules to create the sub-project.

 /FooApp
 |--- Makefile
 |--- subdirs.pro
 `--- src
      |
      |--- app
      |    |--- app.pro
      |    |--- Makefile
      |    |--- Makefile.Debug
      |    |--- Makefile.Release
      |    `--- ... (source files of app)
      |--- lib
      |    |--- lib.pro
      |    |--- Makefile
      |    |--- Makefile.Debug
      |    |--- Makefile.Release
      |    `--- ... (source files of lib)
      `--- lib2
           |--- lib2.pro
           |--- Makefile
           |--- Makefile.Debug
           |--- Makefile.Release
           `--- ... (source files of lib2)
 

nmake

The windows nmake tool reads a Windows Makefile that was generated by qmake. nmake will understand the rules in the Makefile and will call the compiler and linker to build the libraries and application.

Calling nmake in the meta project folder FooApp via

nmake

will find the Makefile and will process it. As this Makefile is one for a meta-project, it contains rules to process the Makefiles of the sub-project.

nmake has no feature to call compile or link in parallel. An option -j n as known with Unix make or gnumake does not exist. So nmake start compiles file after file and probably the listing order of the sub-projects within the SUBDIRS variable will be used. But this is nothing to rely on.

jom

The missing parallel execution feature of nmake is inefficient on modern computer architectures. With CPUs having many kernels for parallelism and fast storage units, a modern computer is bored by using nmake and project compilation takes longer as it could take. This lead to the creation of a new make tool: Jom. Jom is basically a nmake clone extended with the '/J <n>' option to compile in parallel. To compile a meta projects Makefile the call has to be

jom

in the FooApp folder. Jom will recurse automatically in the sub-project folders and use the Makefile of the sub-folders. But all in a parallel fashion. To see the dependency graph that jom processes, the command

jom /DUMPGRAPH

can be called. For the insufficient example

 template = subdirs

 SUBDIRS = \
           src/app \   # relative paths
           src/lib \
           src/lib2
 

the output of jom is

first
 make_first
  sub-src-app-make_first
   FORCE
  sub-src-lib-make_first
   FORCE
  sub-src-lib2-make_first
   FORCE
  FORCE
 

All three sub-src-xxx-make_first are on the same level and they have no dependency under it (aside from FORCE). So all is build in parallel, if the number of Kernels is high enough.

For an example with dependency given explicitly:

 template = subdirs

 SUBDIRS = \
           lib2 \   # sub-project names
           lib  \
           app

 # where to find the sub projects - give the folders
 lib2.subdir = src/lib2
 lib.subdir  = src/lib
 app.subdir  = src/app

 # what subproject depends on others
 app.depends = lib lib2
 

jom output for the graph:

first
 make_first
  sub-src-lib-make_first
   FORCE
  sub-src-lib2-make_first
   FORCE
  sub-src-app-make_first
   sub-src-lib-make_first
    FORCE
   sub-src-lib2-make_first
    FORCE
   FORCE
  FORCE
 

Here sub-src-app-make_first depends on sub-src-lib-make_first and sub-src-lib2-make_first. So lib and lib2 must first be build before app can be processed. But lib and lib2 can be processed in parallel, they have no dependencies.

Finally for the sequential case

 template = subdirs

 SUBDIRS = \
           src/lib2\   
           src/lib \
           src/app

 # build the project sequentially as listed in SUBDIRS !
 CONFIG += ordered
 

jum output for the graph is:

first
 make_first
  sub-src-lib2-make_first-ordered
   FORCE
  sub-src-lib-make_first-ordered
   sub-src-lib2-make_first-ordered
    FORCE
   FORCE
  sub-src-app-make_first-ordered
   sub-src-lib-make_first-ordered
    sub-src-lib2-make_first-ordered
     FORCE
    FORCE
   FORCE
  FORCE
 

So app can be build, if lib was build and lib can be build if lib2 was build: first lib2, then lib, then app. No parallelism on the project level is possible.

With

jom /J 10

ten parallel compiles will be started if enough independent compiles are found.

As there are no dependencies recognized in the simple SUBDIRS example we gave, the parallel processing might result in a build error due to race conditions: The linker of the app project could be started even though the necessary libraries are not build already!

.sln files / .vcxproj files

The standard Microsoft Visual Studio toolchain has another method to define the operation that has to be done to compile a project or meta project. The .sln solution files and the .vcxproj project files are used. These files can be processed by the MSVS2010 IDE or with the msbuild tool as a command line tool.

to be continued....

--Moellney (talk) 22:30, 18 April 2015 (UTC)