Qt Contributors Summit 2022 - Program/Qt Infrastructure for CI
Session Summary
A presentation where the hardware side of the CI is presented along with monitoring of the system
Presentation / Discussion (30 min)
Session Owners
- Tony Sarajärvi
Notes
(Taken by Samuel Gaist)
Delivery and Automation team
Coin CI 8 members
Oslo 3 members
Stages in short
- Code review
- Coin gets notified
- Build
- Release artifacts
- Packaging
- Release
What is COIN ? (COntinuous INtegration)
- From scratch in python
Does
- Archive and reuse built modules
- Read instructions from files within the git repositories
- Customize the size of the VMs needed to run specific jobs (qtbase requires more than other smaller modules, tests do not require the same)
- A GUI that lets select what is to be done based on changing possibilities
COIN allows to reuse modules rather than re-build every modules from scratch each time a patch is added to a module.
COIN itself
~50 platform configurations for each integration
4 million VM created last year
1700+ VMS for a full qt5 integration
1200+ VMS to test the build results
The infrastructure changes every month
- Hardware gets old and support ends
- New hardware replaces old
- New hardware expands the current setup
- Old hardware is upgraded
- Can't just replace enterprise grade items with "standard" one (e.g. DELL SDD with commercial grade SDD)
- Old hardware is used until it literally dies then for spare parts
This also includes planning the electrical capacity of the infrastructure (Dell power sources are not the same as other). The networking must also be adapted and evolved to be efficient but there might be hardware either not yet created or not available.
Cloud service providers have come down in price however running and maintaining own hardware is still cost efficient.
It also allows to connect custom hardware like Apple systems, embedded systems, etc.
- Health
- Bandwidth
- Loads (CPU, Memory, swaps, disk spaces, latencies)
- Warranties
- Certificates
- PowerStore/Compellent, hosts, VMs, switches, firewall, idracs
All through Grafana
For example, there's a service downloading a big package regularly to ensure that the bandwith is correctly used.
Setup
- Ansible is slowly replacing many manual tasks
- Security:
- SSH keys
- Segmentation
- Automation
The usual question
- Why not AWS and Azure ?
- COIN would be way very complicated to setup there (not impossible though)
- The current system is pretty monolithic
- Offloading some task would be possible however the main issues would be data transfer
- Distribute 7TB of data daily
Q&A
- Distribute image to approvers and maintainers for debugging ?
- Doable for Linux image
- Not with Windows
- Likely not for Apple as well
- What about an ssh / VNC session ?
- Doable but not had much request for that
- Currently would require to be a subcontractor because of NDA and other internal requirements.
- There's going some work to be done in order to be able to grant access to external people in a secured manner to specific machines.
- Apple arm64 is not yet virtualized so each Apple machine must be cleaned after each build.