Qt Creator ManualTests DebuggerLldb: Difference between revisions

From Qt Wiki
Jump to navigation Jump to search
(Made paths to test projects italic as requested by tester)
(Clarify what the comments in simple.pro are for)
 
(3 intermediate revisions by 2 users not shown)
Line 2: Line 2:


{| class="wikitable"
{| class="wikitable"
|+ Tests using ''tests/manual/debugger/simple/simple.pro'' from Qt Creator's source repository
|+Tests using ''tests/manual/debugger/simple/simple.pro'' from Qt Creator's source repository
! Test  
!Test
! Result  
!Result
! Annotation  
!Annotation
|-
|-
| Create new project. Can you build, run and debug it?  
|Create new project. Can you build, run and debug it?
| automated
|automated
|  
|
|-
|-
| Set breakpoint, press F5 / Cmd + Y to build and run debugger, verify that program stops at a breakpoint that you set:
|Set breakpoint, press F5 / Cmd + Y to build and run debugger, verify that program stops at a breakpoint that you set:
* in main function before program is started
 
* while the program is running
*in main function before program is started
* in dynamically loaded plugins, especially in constructors
*while the program is running
* on a bit of code that was commented out. Make sure it is moved on debugger startup to the first line producing real code below that position and that it is hit there.  
*in dynamically loaded plugins, especially in constructors
|  
*on a bit of code that was commented out. Make sure it is moved on debugger startup to the first line producing real code below that position and that it is hit there.
|  
|
|
|-
|-
| "Step into" a couple of times. Can you step into Qt source code ('''*.cpp''' file under QTDIR)?
|"Step into" a couple of times. Can you step into Qt source code ('''*.cpp''' file under QTDIR)?
(Mac: switch on 'Use Debug Versions of Frameworks' in run configuration. You need Qt sources.)  
(Mac: switch on 'Use Debug Versions of Frameworks' in run configuration. You need Qt sources.)  
|  
|
|  
|
|-
|-
| Test debugging helpers/python lldb: Do classes like QImage or std::string show beautiful information instead of the raw structure?  
|Test debugging helpers/python lldb: Do classes like QImage or std::string show beautiful information instead of the raw structure?
| automated  
|automated
|  
|
|-
|-
| Step through some (not all) test* functions and check whether the displayed data looks sane
|Step through some (not all) test* functions and check whether the displayed data looks correct
|  
|
|  
|The code contains comments with the expected displayed data. These are meant for semi-automatic runs and might differ from what you see. Use your own judgement what's correct and what's not.
|-
|-
| Comment out the return statement inside the following functions one by one. Check whether you'll end up with a proper stack trace & locals display.
|Comment out the return statement inside the following functions one by one. Check whether you'll end up with a proper stack trace & locals display.
* testNullPointerDeref()
 
* testNullReference()
*testNullPointerDeref()
* testEndlessLoop(); (break manually)
*testNullReference()
* testEndlessRecursion();
*testEndlessLoop(); (break manually)
* testUncaughtException();
*testEndlessRecursion();
|  
*testUncaughtException();
|  
|
|
|-
|-
| Test a breakpoint in a QThread:
|Test a breakpoint in a QThread:
# Uncomment the call to qthread::testQThread()
 
# Place a breakpoint in qthread::Thread::run()
#Uncomment the call to qthread::testQThread()
# Run this in the debugger
#Place a breakpoint in qthread::Thread::run()
# Make sure that:
#Run this in the debugger
#* the breakpoint receives the right number of hits
#Make sure that:
#* "Locals and Expressions" and "Stack" views show reasonable data, including the object names of the threads
#*the breakpoint receives the right number of hits
#*"Locals and Expressions" and "Stack" views show reasonable data, including the object names of the threads
|
|
|
|
|-
|-
| Switch on temporarily 'Operate by Instruction' (small icon above the stack trace) and check whether you see disassembler output and can step by instruction
|Switch on temporarily 'Operate by Instruction' (small icon above the stack trace) and check whether you see disassembler output and can step by instruction
|
|
|}
|}


{| class="wikitable"
{| class="wikitable"
|+ Tests using ''tests/manual/debugger/cli-io/cli-io.pro'' from Qt Creator's source repository
|+Tests using ''tests/manual/debugger/cli-io/cli-io.pro'' from Qt Creator's source repository
! Test
!Test
! Result
!Result
! Annotation
!Annotation
|+
|+
| Check I/O (qDebug, std::cout, std::cerr)  
|Check I/O (qDebug, std::cout, std::cerr)
|  
|
|  
|
|-
|-
| Check "Run in Terminal". Use Terminal for input.
|Check "Run in Terminal". Use Terminal for input.
|  
|
|  
|
|-
|-
| Check nothing bad happens on a simple int main() {} program with no breakpoints set  
|Check nothing bad happens on a simple int main() {} program with no breakpoints set
| automated
|automated
|
|
|}
|}


{| class="wikitable"
{| class="wikitable"
|+ Tests using ''tests/manual/debugger/gui/gui.pro'' from Qt Creator's source repository
|+Tests using ''tests/manual/debugger/gui/gui.pro'' from Qt Creator's source repository
! Test
!Test
! Result
!Result
! Annotation
!Annotation
|+
|+
|
|
# Run the project.
{| class="wikitable" style="background-color: #eeeeee;"
# Click the button "Crash".<br/>It should write a core file to the working directory. Depending on the operating system, you might have to set "ulimit &ndash;c unlimited" and/or disable OS-specific crash-reporting mechanisms.
|-
# Attach Creator to the core file.<br/>Verify that you see a stack trace, variable values and code markers just as if you had run the application in the debugger from the beginning.
|
|  
* '''Notes for macOS:'''
|  
** Be sure you have the "/core" (under the root) folder (with sufficient permissions if you don't want to run your app as a root).
** Check if the core dumps are enabled with "'''sysctl kern.coredump'''". It should return 1. Otherwise you can set it by "'''sysctl kern.coredump 1'''".
** The application must be signed, which the project build should do automatically. If you face issues, check that the build resulted in a "codesign" call. The codesign call also includes the right "entitlements" via the entitlements.plist that is included in the sources, which sets "get-task-allow". This is required to get "attachable" permission to your application (as of macOS 12.5, see  [https://developer.apple.com/library/archive/technotes/tn2415/_index.html#:~:text=and%20UIPasteboard%20sharing.-,get%2Dtask%2Dallow,-The%20boolean%20value Apple official documentation]). Then you can access the memory of your app and get the core dump.
** Be sure to enable core dumps with "ulimit -c unlimited". Note that this is limited to the shell that you called that in, so run it in Terminal and start the application (or Qt Creator and from there the application) from that Terminal (e.g. with "open <path>/Qt\ Creator.app").
|}
#Run the project.
#Click the button "Crash".<br />It should write a core file to the working directory. Depending on the operating system, you might have to set "'''ulimit &ndash;c unlimited'''" and/or disable OS-specific crash-reporting mechanisms.
#Attach Creator to the core file.<br />Verify that you see a stack trace, variable values and code markers just as if you had run the application in the debugger from the beginning.
|
|
|-
|-
|
|
# Start the project from outside Creator, e.g. from the command line.
#Start the project from outside Creator, e.g. from the command line.
# Attach Creator to the running process.<br/>Verify that you see a stack trace, variable values and code markers just as if you had run the application in the debugger from the beginning. You should also be able to pause/continue the execution.
#Attach Creator to the running process.<br />Verify that you see a stack trace, variable values and code markers just as if you had run the application in the debugger from the beginning. You should also be able to pause/continue the execution.
|  
|
|  
|
|-
|-
| Test unusual situations: Kill X 'externally' while debugging (both in a 'running' and 'stopped' state), where X is
|Test unusual situations: Kill X 'externally' while debugging (both in a 'running' and 'stopped' state), where X is
* the debugged program
 
* lldb  
*the debugged program
|  
*lldb
|  
|
|
|}
|}


{| class="wikitable" style="width: 80%"
{| class="wikitable" style="width: 80%"
|+ Test run as root from inside QC - any simple application (Linux, macOS)
|+Test run as root from inside QC - any simple application (Linux, macOS)
! Test
!Test
! Result
!Result
! Annotation
!Annotation
|+
|+
|
|
Run the project without checking "Run as root user" on the run settings page.
Run the project without checking "Run as root user" on the run settings page.
# Verify application is starting and can be used as expected.
 
|  
#Verify application is starting and can be used as expected.
|  
|
|
|-
|-
|
|
Debug the project without checking "Run as root user" on the run settings page.
Debug the project without checking "Run as root user" on the run settings page.
# Verify application and debugger are starting correctly.
 
# Verify you can interrupt the application and hit a breakpoint. Check some locals whether they contain sane data.
#Verify application and debugger are starting correctly.
|  
#Verify you can interrupt the application and hit a breakpoint. Check some locals whether they contain sane data.
|  
|
|
|-
|-
|
|
Run the project with checking "Run as root user" on the run settings page.
Run the project with checking "Run as root user" on the run settings page.
# Verify application and debugger are starting correctly.
 
# Verify you can interrupt the application and hit a breakpoint. Check some locals whether they contain sane data.
#Verify application and debugger are starting correctly.
# Run "sudo -k" to ensure credentials are resetted.
#Verify you can interrupt the application and hit a breakpoint. Check some locals whether they contain sane data.
# Re-run and check whether credentials are requested again.
#Run "sudo -k" to ensure credentials are resetted.
# Re-do verifications above.
#Re-run and check whether credentials are requested again.
|  
#Re-do verifications above.
|  
|
|
|-
|-
|
|
Debug the project with checking "Run as root user" on the run settings page.
Debug the project with checking "Run as root user" on the run settings page.
# Verify application and debugger are starting correctly.
 
# Verify you can interrupt the application and hit a breakpoint. Check some locals whether they contain sane data.
#Verify application and debugger are starting correctly.
# Run "sudo -k" to ensure credentials are resetted.
#Verify you can interrupt the application and hit a breakpoint. Check some locals whether they contain sane data.
# Re-run and check whether credentials are requested again.
#Run "sudo -k" to ensure credentials are resetted.
|  
#Re-run and check whether credentials are requested again.
|
|
|
|-
|-
Line 141: Line 158:
Is entering invalid credentials handled correctly?
Is entering invalid credentials handled correctly?
|
|
|  
|
|}
|}

Latest revision as of 10:46, 30 June 2023


Tests using tests/manual/debugger/simple/simple.pro from Qt Creator's source repository
Test Result Annotation
Create new project. Can you build, run and debug it? automated
Set breakpoint, press F5 / Cmd + Y to build and run debugger, verify that program stops at a breakpoint that you set:
  • in main function before program is started
  • while the program is running
  • in dynamically loaded plugins, especially in constructors
  • on a bit of code that was commented out. Make sure it is moved on debugger startup to the first line producing real code below that position and that it is hit there.
"Step into" a couple of times. Can you step into Qt source code (*.cpp file under QTDIR)?

(Mac: switch on 'Use Debug Versions of Frameworks' in run configuration. You need Qt sources.)

Test debugging helpers/python lldb: Do classes like QImage or std::string show beautiful information instead of the raw structure? automated
Step through some (not all) test* functions and check whether the displayed data looks correct The code contains comments with the expected displayed data. These are meant for semi-automatic runs and might differ from what you see. Use your own judgement what's correct and what's not.
Comment out the return statement inside the following functions one by one. Check whether you'll end up with a proper stack trace & locals display.
  • testNullPointerDeref()
  • testNullReference()
  • testEndlessLoop(); (break manually)
  • testEndlessRecursion();
  • testUncaughtException();
Test a breakpoint in a QThread:
  1. Uncomment the call to qthread::testQThread()
  2. Place a breakpoint in qthread::Thread::run()
  3. Run this in the debugger
  4. Make sure that:
    • the breakpoint receives the right number of hits
    • "Locals and Expressions" and "Stack" views show reasonable data, including the object names of the threads
Switch on temporarily 'Operate by Instruction' (small icon above the stack trace) and check whether you see disassembler output and can step by instruction
Tests using tests/manual/debugger/cli-io/cli-io.pro from Qt Creator's source repository
Test Result Annotation
Check I/O (qDebug, std::cout, std::cerr)
Check "Run in Terminal". Use Terminal for input.
Check nothing bad happens on a simple int main() {} program with no breakpoints set automated
Tests using tests/manual/debugger/gui/gui.pro from Qt Creator's source repository
Test Result Annotation
  • Notes for macOS:
    • Be sure you have the "/core" (under the root) folder (with sufficient permissions if you don't want to run your app as a root).
    • Check if the core dumps are enabled with "sysctl kern.coredump". It should return 1. Otherwise you can set it by "sysctl kern.coredump 1".
    • The application must be signed, which the project build should do automatically. If you face issues, check that the build resulted in a "codesign" call. The codesign call also includes the right "entitlements" via the entitlements.plist that is included in the sources, which sets "get-task-allow". This is required to get "attachable" permission to your application (as of macOS 12.5, see Apple official documentation). Then you can access the memory of your app and get the core dump.
    • Be sure to enable core dumps with "ulimit -c unlimited". Note that this is limited to the shell that you called that in, so run it in Terminal and start the application (or Qt Creator and from there the application) from that Terminal (e.g. with "open <path>/Qt\ Creator.app").
  1. Run the project.
  2. Click the button "Crash".
    It should write a core file to the working directory. Depending on the operating system, you might have to set "ulimit –c unlimited" and/or disable OS-specific crash-reporting mechanisms.
  3. Attach Creator to the core file.
    Verify that you see a stack trace, variable values and code markers just as if you had run the application in the debugger from the beginning.
  1. Start the project from outside Creator, e.g. from the command line.
  2. Attach Creator to the running process.
    Verify that you see a stack trace, variable values and code markers just as if you had run the application in the debugger from the beginning. You should also be able to pause/continue the execution.
Test unusual situations: Kill X 'externally' while debugging (both in a 'running' and 'stopped' state), where X is
  • the debugged program
  • lldb
Test run as root from inside QC - any simple application (Linux, macOS)
Test Result Annotation

Run the project without checking "Run as root user" on the run settings page.

  1. Verify application is starting and can be used as expected.

Debug the project without checking "Run as root user" on the run settings page.

  1. Verify application and debugger are starting correctly.
  2. Verify you can interrupt the application and hit a breakpoint. Check some locals whether they contain sane data.

Run the project with checking "Run as root user" on the run settings page.

  1. Verify application and debugger are starting correctly.
  2. Verify you can interrupt the application and hit a breakpoint. Check some locals whether they contain sane data.
  3. Run "sudo -k" to ensure credentials are resetted.
  4. Re-run and check whether credentials are requested again.
  5. Re-do verifications above.

Debug the project with checking "Run as root user" on the run settings page.

  1. Verify application and debugger are starting correctly.
  2. Verify you can interrupt the application and hit a breakpoint. Check some locals whether they contain sane data.
  3. Run "sudo -k" to ensure credentials are resetted.
  4. Re-run and check whether credentials are requested again.

Is entering invalid credentials handled correctly?