View source for Cross-compiling Qt 6.5 for both armhf and aarch64 architectures for Raspberry Pi OS

*We build all modules and enable examples.
*We'll be using Clang-13 as the compiler and lld-13 as the linker. This is mostly to avoid building a toolchain, which would be required with GCC and binutils.
*We are copying the sysroot from a 64-bit installation of Raspberry Pi OS. However it will contain both 32-bit and 64-bit libraries, [https://wiki.debian.org/Multiarch/HOWTO installed according to the Debian Multiarch HOWTO].

===Host build===

Cross-compiling Qt-6 requires that we perform a '''host build first''' in order to have tools like <code>moc</code> and <code>lupdate</code> available for the cross-build.

*Install the packages for the compiler and the linker and other needed build tools, with package names depending on the system. For OpenSUSE-Tumbleweed I needed to install
  llvm13-libclang13 clang13-devel lld13  cmake ninja
'''NOTE''': Avoid having multiple versions of Clang installed (it caused linkage problems obscure enough to be detected only when the <code>lupdate</code> binaries were segfaulting!)

*Install required libraries for building Qt6.
*Configure the host build. I use '''developer-build''' to not need installation. Then build.
<pre>
cd $HOME/qt6-host-developer-build
$SRC_DIR/configure -developer-build -linker lld -nomake tests -nomake examples \
    --  -DCMAKE_C_COMPILER=clang-13 -DCMAKE_CXX_COMPILER=clang++-13
ninja
</pre>

===Preparing the sysroot on Raspberry Pi OS with both 32-bit and 64-bit libraries===

To cross-compile for a system, you need to have all the system libraries and header files of the target copied into the host. This is called a '''sysroot'''.

To prepare that, the best way is to try compiling on the target itself. You don't need to actually complete the compilation, but at least the configuration step should be successful.

After lots of trial and error, these are the packages that I installed on Raspberry Pi OS:
<pre>
LIBDEPS="libfontconfig1-dev libfreetype6-dev libx11-dev libx11-xcb-dev libxext-dev libxfixes-dev libxi-dev libxrender-dev libxcb1-dev libxcb-glx0-dev libxcb-keysyms1-dev\
    libxcb-image0-dev libxcb-shm0-dev libxcb-icccm4-dev libxcb-sync-dev libxcb-xfixes0-dev libxcb-shape0-dev libxcb-randr0-dev libxcb-render-util0-dev libxcb-util-dev\
    libxcb-xinerama0-dev libxcb-xkb-dev libxkbcommon-dev libxkbcommon-x11-dev\
    libdrm-dev libdrm-etnaviv1 libdrm-freedreno1 libdrm-tegra0 libgl-dev libglx-dev mesa-common-dev\
    libegl-dev libegl1-mesa-dev libgbm-dev libgles-dev libgles2-mesa-dev libglvnd-dev libopengl-dev libopengl0 libvulkan-dev mesa-vulkan-drivers\
    libxcb-cursor-dev libxcursor-dev\
    libssl-dev libjpeg-dev"

sudo apt update
sudo dpkg --add-architecture armhf

# Install 64-bit libraries
sudo apt install  $LIBDEPS

# Install 32-bit libraries
sudo apt install crossbuild-essential-armhf
echo $LIBDEPS | sed -E -e 's/ +|$/:armhf /g' | xargs sudo apt install
</pre>
'''WARNING''': do not install <code>clang-*-dev</code> on the target, it causes linking problems on tools like <code>lupdate</code> and is not required for a successful build.

===Copy the sysroot to the host===

On the x86_64 host:
<pre>
SYSROOT=$HOME/sysroot-RPiOS
rsync --info=progress2 -ar --copy-unsafe-links --delete -z --zc lz4 pi@rasbperry-pi:/usr $SYSROOT/
</pre>
This will copy all of <code>/usr</code> into the $SYSROOT directory, using fast compression for quicker transfer over the network. 
It will also replace symlinks that escape the directory, with the actual file contents.
====A workaround is needed for clang and ninja to avoid rebuilding everything on every rebuild attempt:====
<pre>
ln -s usr/lib     $SYSROOT/lib
ln -s usr/include $SYSROOT/include
ln -s usr/arm-linux-gnueabihf $SYSROOT/arm-linux-gnueabihf
</pre>

===CROSS-COMPILING===
On the host, create and enter build directories. Then:

For targeting 64-bit Raspberry Pi OS, set:
<pre>
ARCH=aarch64
TRIPLET=aarch64-linux-gnu
</pre>
For targeting 32-bit Raspberry Pi OS, ''retaining compatibility all the way back to the original Raspberry Pi'', set:
<pre>
ARCH=armhf
TRIPLET=arm-linux-gnueabihf
</pre>
And then configure the actual cross-compilation:
<pre>
SYSROOT=$HOME/sysroot-RPiOS
QT_HOST_PATH="$HOME/qt6-host-developer-build/qtbase"
PREFIX=$HOME/qt6-$ARCH

CC=clang-13
CXX=clang++-13
CFLAGS="-O1 -target $TRIPLET --sysroot $SYSROOT"
LDFLAGS="-Wl,-rpath-link,$SYSROOT/usr/lib/$TRIPLET"
CXXFLAGS="$CFLAGS"
ASMFLAGS="$CFLAGS"

PKG_CONFIG_PATH=  \
PKG_CONFIG_SYSROOT_DIR="$SYSROOT" \
PKG_CONFIG_LIBDIR="$SYSROOT/usr/lib/$TRIPLET/pkgconfig:$SYSROOT/usr/share/pkgconfig" \
    $SRC_DIR/configure \
    -make examples -xcb -opengl es2 -debug -linker lld \
    -sysroot "$SYSROOT" -prefix "$PREFIX" \
    -qt-host-path "$QT_HOST_PATH" \
    --  -DCMAKE_C_COMPILER="$CC" -DCMAKE_CXX_COMPILER="$CXX" \
        -DCMAKE_C_FLAGS="$CFLAGS" -DCMAKE_CXX_FLAGS="$CXXFLAGS" -DCMAKE_ASM_FLAGS="$ASMFLAGS" \
        -DCMAKE_SHARED_LINKER_FLAGS="$LDFLAGS" -DCMAKE_EXE_LINKER_FLAGS="$LDFLAGS" -DCMAKE_MODULE_LINKER_FLAGS="$LDFLAGS" \
        -DCMAKE_SYSROOT="$SYSROOT" -DCMAKE_SYSTEM_NAME=Linux \
        -DCMAKE_FIND_ROOT_PATH_MODE_PROGRAM=NEVER \
        -DCMAKE_FIND_ROOT_PATH_MODE_LIBRARY=ONLY -DCMAKE_FIND_ROOT_PATH_MODE_INCLUDE=ONLY -DCMAKE_FIND_ROOT_PATH_MODE_PACKAGE=ONLY
</pre>
'''NOTE''': do not omit the flag <code>-DCMAKE_SYSTEM_NAME=Linux</code>, this is what tells CMake that we are actually doing a cross-build!

'''Compile''' using ninja and '''install'''. Then transfer the whole PREFIX directory to any Raspberry Pi and enjoy!

=== Spread the build using icecream ===
Did you know [https://github.com/icecc/icecream icecream distributed compiler] will happily send the cross-compilation jobs to other x86_64 hosts, just like it does with the native ones?

Given that you are part of a compile cluster (i.e. you are running <code>iceccd</code> on your host and it has successfully connected to the scheduler on your local network), all you have to do is setup symlinks for your compiler:
<pre>
mkdir -p $HOME/.local/bin
export PATH=$HOME/.local/bin:$PATH
ln -s /usr/bin/icecc $HOME/.local/bin/clang-13
ln -s /usr/bin/icecc $HOME/.local/bin/clang++-13
</pre>

Then enjoy the glory of <code>ninja -j 200</code> !