How to reproduce autotest fails
The current CI system suffers from high CPU load and since the virtual machines are using shared resources, on high load sometimes something glitches and the autotests may crash. Here are a few pointers on how to try to reproduce such issues on different platforms.
As a rule of thumb you need to put load on your system, reduce niceness and memory, and fix processor affinity.
If you don't want to jeopardize your own machine, you can ask someone from the CI team to clone a VM for you. Anyone from the release team can help you to create a VM. Just ask in #qt-qa or #qt-labs in the IRC. Also note, that if you create a new bug report for the failing autotest, remember to add labels: 'autotest' and 'flaky' to the label field, so that it gets tracked properly.
When diagnosing a crashing test, passing the command line option "-nocrashhandler" to the test suppresses dumping of the stack trace by QTestlib and makes it possible to attach a debugger or have it launched automatically by the OS (post-mortem).
Linux
1. 'stress' imposes certain types of compute stress on your system
- Install 'stress' (on Ubuntu)
sudo apt-get install stress
- Example usage:
stress --cpu 8 --io 4 --vm 2 --vm-bytes 128M
- you can check with 'top' that it runs
See more information from: https://linux.die.net/man/1/stress https://www.cyberciti.biz/faq/stress-test-linux-unix-server-with-stress-ng/
2. Reduce niceness
nice -n -20 ./test
See: http://bencane.com/2013/09/09/setting-process-cpu-priority-with-nice-and-renice/
3. Use 'taskset' to set process affinity
taskset -c 1
Which means: "use second core". From the 'taskset' man page: -c, --cpu-list "specify a numerical list of processors instead of a bitmask. The list may contain multiple items, separated by comma, and ranges. For example, 0,5,7,9-11." See more information: https://linux.die.net/man/1/taskset
4. Run tests in a loop over and over
Launch 'stress' (see above, 1) Run tests in a loop:
for i in {0..100}; do taskset -c 1 ./tst_example >> log.txt 2>&1; done
You can also try to run the same tests from two different terminals and set the process affinity.
5. What if it's a segmentation fault/core dump you cannot get a crash log from:
In a different terminal window, set:
export LD_PRELOAD=/lib/x86_64-linux-gnu/libSegFault.so
ulimit -c unlimited
after the crash in the same terminal you have set the LD_PRELOAD:
gdb ./testCrash ./core
and, in gdb,
bt
6. Use rr
See https://rr-project.org/ and especially chaos mode for rr https://robert.ocallahan.org/2016/02/introducing-rr-chaos-mode.html to get threads to run with different priorities, hopefully reproducing races.
Mac OS
1. stress testing CPU
Repeat the word “yes” at such speed that it consumes all available processor resources. In a terminal do:
yes > /dev/null & yes > /dev/null & yes > /dev/null & yes > /dev/null
Check with 'top' that you see 4 'yes' processes running. Run "killall yes" to kill all instances.
See: http://osxdaily.com/2012/10/02/stress-test-mac-cpu/
Windows
1. stress testing CPU:
install CPUSTRES.EXE from https://blogs.msdn.microsoft.com/vijaysk/2012/10/26/tools-to-simulate-cpu-memory-disk-load/ Activate all threads (select with tick marks) set 'Thread Priority' of the threads to be 'time critical' or 'highest' and 'Activity' to 'Busy'
2. launch test using only one thread:
start /B /WAIT /affinity 1 test.exe
1 == use CPU 0, 2 == use CPU 1 etc. See the table from: https://blogs.msdn.microsoft.com/santhoshonline/2011/11/24/how-to-launch-a-process-with-cpu-affinity-set/
3. run the tests in a loop:
for /L %i in (1, 1, 10); do start /B /WAIT /affinity 1 tst_example.exe >> log.txt 2>&1
(Note that the >> ensures that the output is appended to log.txt, rather than overwriting its contents after each run - see https://technet.microsoft.com/en-us/library/bb490982.aspx)
3.1. run several tests in a loop simultaneously You can also try to run the same tests from two different terminals and set the process affinity Note: When writing a .bat script, use: for /L %%i in (1, 1, 10)....