Qt for Python Development Notes

2018

12. April 2018

- Getting Started guide under works by doc engineer
- focus on fixing coin on Mac & RedHat 7.4 (looks promising so far)
- shiboken documentation and diagrams being updated
  - old and unused docs dir shall be cleaned out
- Tab Browser example improved
- License page generated based on attribution files
- lots of discussion when to release Qt for Python TP
  - after Qt 5.11 release (scheduled for 31.5.) 
  - likely to be mid June (stay tuned)
- WebEngine issues
  - needs updated qt.conf files to be deployed for apps
- Heap types (work in progress)
- still more work in pipeline to get code PEP 384 ready (Heap types are prerequisite)
- Qt for Python blog posts being worked on

5. April 2018

- first doc snapshot is up and running: https://doc-snapshots.qt.io/qtforpython/
- more work to be done to automate generation of snapshots
- briefly talked about technical blog post opportunities
- pdb files from Windows wheels removed (reducing the wheel size)
- improving the tab browser example
- coin currently not accepting changes
  - currently lots of pending patches start to create backlog
- Heap type taking longer than expected
- qApp macro reference leaks finally resolved
- quite a few formatting changes to enforce a common coding style
- lots of smaller fixes

29. March 2018

- skipped due to Easter holidays

22. March 2018

- Release process (do we follow Qt?) 
  - Release manager provides packages
  - RTA testing to be setup -> for each package we run RTA
  - Request to be put on mailing list
  - Current Test setup proposal
    - Win10 (Python 3.6)
    - Linux (Python 3.6/2.7)
    - Mac (Python 3.6/2.7)
  - To be defined a test set (//TODO)
  - Testing information to be collected in wiki
- Documentation
  - QML advanced tutorial documentation updated
  - Landing page in the works
  - snapshot still delayed by other qdoc bugs
  - style sheet update considered
  - so far only dealt with class docs, started to include unrelated doc pages
- snapshot testing (angle & webengine not properly packaged on Windows)
- enum support improved (dealing with scoped enums and enum values)
- general bugfixing to increase stability and address mem leaks
- Talks to PyData submitted
- Qt Charts examples ported
- Proper support for QFlags being worked on
- PYSIDE-595 fixed
- PYSIDE-595 first heap type protoype under review proving that concept works
- PYSIDE-560 depends on PYSIDE-595
- CI side
  - coin did Python updates which temporarily broke Linux wheels
  - briefly discussed manylinux request on mailing list (Qt forms bottom line, no point to extend beyond scope of Qt)
  - final list of wheels for release (win, linux, mac) x 2 (2.7 & 3.x Python)

15. March 2018

- Fix to create 3.6 Linux wheels under review.
- Fix scriptableapplication build on Windows, and other small bugs.
- Two QtCharts examples were ported and abstracts were written to submit PyData talks
- Updating the QML advanced tutorial and adding index.rst and gettingstarted.rst
- Tried the wheel pkgs for python 27 on Linux. Works OK except the webengine example, which gives random crashes.
- Docs: PYSIDE-620 Shadow build issue fixed, fixed usage of deprecated Sphinx API, 
- Continued on PYSIDE-560, currently  Windows problem with Linkage
- Debugging on PYSIDE-595, final stages
- Fixed PYSIDE-258

08. March 2018

- msvcrt.dll to be shipped on Windows
- wheel snapshots are automatically generated
  - start with public availability under http://download.qt.io/snapshots/ci/
  - snapshots to public pypi server under discussions
- discussion on how to increase value of official pyside mailing list http://lists.qt-project.org/mailman/listinfo/pyside
- looking at build issues on Windows (improving error handling)
- Participation in Python User Group meeting in Berlin (https://www.meetup.com/Python-Users-Berlin-PUB/events/xhqwhpyxfblb/)
- PYSIDE-595 - Heap Type conversions ongoing
  - test bed created for easier development
- Documentation
  -> Managed to make libxml, libxsl an optional dependency falling back to QtXmlPatterns (despite QTBUG-66925)
  -> Small fixes to doc modifications
  -> Fix class inheritance diagrams for nested classes

01. March 2018

- automatic snapshot process almost in place (pending reviews) (RTA to follow)
- general bug fixing (including build and version problems)
- most community Pyside1 pages point to official Pyside 2 (Qt for Python) wiki pages
- PYSIDE-595 - Heap Type conversions ongoing
- documentation improving
  -> class diagrams being generated
  -> missing links and references
  -> dependencies to qdoc bugs
  -> snapshot process in works
- added support for scoped enums in pyside
- Important things discussed:
  - PySide2 versioning (current proposal WIP patch at https://codereview.qt-project.org/#/c/221767/)
    - Do official releases follow Qt branch names? - 
      -> yes (minor release of Pyside is bound to same minor release of Qt
      -> patch level version may differ
    - Will we have snapshots, and if so, will they be uploaded to pip? How will we handle snapshot versions?
      -> tentative yes, if public pip server supports snapshots Pyside should be good citizen
    - alpha / beta / RC versioning?
      -> Pyside should follow the Qt schema
    - Probably need to take into account PEPs for versions pushed to Pypi.
    - Shiboken / PySide2 versions in sync?
      -> yes, they should be in sync
    - Will we specify a version for 5.6 branch?
      -> yes, but no packages being made
    - Are we sticking to the package name PySide2, but follow Qt 5.x versions as PySide2 versions?
      -> Pyside2_5.11.0 as in https://codereview.qt-project.org/#/c/221767/
  - Do we want to deploy msvcrt dlls into Windows packages?
    -> needs more research - is it really needed ?
  - Do we want to ship debug symbols for Qt inside the PySide2 packages? 
    -> not for now
  - Do we want to ship debug Qt shared libraries for macOS packages? (not possible on Windows, not applicable on Linux afaik).
    -> not for now
  - How will the pip upload process look like? Manual uploads? Who takes care of it?
    -> Qt release team will take care of it
  - Current branching policies:
    -> 5.6 branch stops being merged into 5.9 (after pending patches have merged) -> cherry-pick mode
    -> 5.9 is base line for patches and being forward merged to 5.11
    -> no packages from 5.9 & 5.6 (interested parties need to get it from git)
    -> 5.11 becomes branch for first TP
    -> dev is feature development
    -> all first TP release related patches go to 5.9 (and 5.11 subsequently)

22. February 2018

- dev branch is still/again broken in CI, seems like pyside issues
- changes to create automatic snapshots out of CI is still under review
- Numerous small fixes to the doc generation, significantly bringing down the number of documentation warnings
- Massaged build rules to improve built packages
  - Removed unnecessary _utils.py file PYSIDE-600
  - Changed wheel package names to contain both PySide2 version and Qt version against which it was built PYSIDE-613
  - Finally got libclang to be deployed into packages
  - Improved build process not to copy unnecessary symlinks and duplicating libraries when creating wheel (so that you don't get 3x size increase because of 3x number of WebEngine libraries) PYSIDE-495
  - Fixed wheel package names to contain correct minimum deployment target with which macOS was built with (on further analysis, this will need further improvement) PYSIDE-612
  - Fixed rpath handling for QtWebEngineProcess when creating standalone packages on macOS PYSIDE-605
- Added CMake build rules for scriptable application example PYSIDE-597
- Investigated and fixed a heap corruption bug in a test that caused constant crashes on macOS CI runs
- Reduced number of warnings when running shiboken to create user bindings (like scriptable application example) PYSIDE-587
- PYSIDE-595: Making slow progress, still debugging. It's a difficult process, but assumption is that it will work.
- Havent's merge this new patch because we should be 200% sure it's properly working.
- Working on a few bugs PYSIDE-106 PYSIDE-570
- One of the bugs PYSIDE-611 was valid only for 5.6, but I noticed a strange behaviour on 5.9, so I will keep working on it, even after we move 5.6 to cherry pick mode.

15. February 2018

- CI broken, 5.6 & dev branch broken, 5.9 just got working again
- first TP will be 5.11 based (needs 5.11 git branches, 5.6 branch goes into cherry-pick mode, 5.9 branch remains main target)
- more modules added to doc, qt3d & qtquick has some special handling
- snippet work ongoing
- snapshot generation on the agenda
- lots of bug fixing e.g.: PYSIDE-104
- PYSIDE-560

8. February 2018

- Looking through previous EuroPython talks to write a proposal for this year.          
- Working on a few bugs (fixed a couple related to objects references),                 
  - that sadly will open new issues with memory management. A further analysis on this matter could improve the current memory leaks on PySide.                           
- Trying to "organise" a little the typesystem XML files.                               
- Working on PYSIDE-104 that ended up being a general issue on PySide about a proper verification between slots and signals arguments, to find the best match.
- Merged a few build related changes in 5.6
  - Pending change for PYSIDE-593
  - Pending change for PYSIDE-605
  - Merged changes for PYSIDE-603 and PYSIDE-604
- Investigated and attached minimal reproducible example for PYSIDE-585
- Investigated some macOS build related aspects for package release
- 5.6 CI is unblocked, but 5.9 CI is blocked due to qtbase issues (version bumping)
- PYSIDE-560 waiting for view
- PYSIDE-595 work ongoing, not ready yet
- PYSIDE-363  Documentation: Fixed images, fix handling of qdocconf files  that contain several modules. Looking at snippet resolution.
- Problems with Qt3D due to refactoring of Qt3Dextras in 5.10. Pushed  patch suggestion.

1. February 2018

- Continued work on documentation PYSIDE-363, fixed static pages, add more modules, fixed parameters to qdoc, images
- The GettingStarted wiki is getting better, besides a cleaning it a bit
  - moved the topics around to have guidelines by supported platform (a la Qt)
  - have received feedback and trying to add more information
- Addressed bugs:
  - PYSIDE-34
  - PYSIDE-264
  - cleaning a bit Shiboken's XML files.
  - PYSIDE-560
    - ported to 5.9, extended to 18 header files
    - solved unicode problem
    - reduced the patches to be minimal (avoid many #ifdef's, use a verbose macro name if possible)
    - made everything compile but one announced file: "object.h" depends on the heaptypes solution (PYSIDE-595). Removed WIP status,review pending.
  - PYSIDE-595
    - Started experimenting
    - turned out that the signature extension barfed because it could no longer find the type "__name__". The function PyType_Type.tp_new needs to be called to provide the "ht_name" attribute that is used in heaptypes.
- solving the attribution file issues.
- Pending patches to fix libICU issues
- Pending patches to fix OpenSSL issues PYSIDE-599
- Pending patches to deploy libclang library for shiboken
- Improved documentation about OpenSSL dependency
- Pending patches for cleaning up build scripts a bit (logging and refactoring)
- Pending patches to reduce barrage of warnings PYSIDE-587
- Pending patch for adding CMake build rules for the scriptable_application example (useful as a reference on how to use shiboken for custom bindings) PYSIDE-597
- Old 5.6 CI integration issue fix, but a new issue appeared that was fixed for 5.9 and dev, but still blocks 5.6
- Updated the qml basics tutorial

25. January 2018

- Pyside not renamed
- Releasing preparation
  - pip packages created by coin
  - Automated release testing (RTA) (installing pip, launching an example)
  - final selection of release package targets to be define closer to release (depending on current stable ABI work)
- Investigation doc issues when using prefix & shadow builds
- Need to identify prime demos/examples
- Started to rework Getting Started Guide wiki 
- Addressed cmake build issues
- Handling libICU issues (causing linker issues when importing QtCore)
- Continued work on PYSIDE-560

18. January 2018

 - macOS framework fix (should ignore qt4 system headers)
 - another patch on top to fix OpenGL integer types on mac / Linux (GLuint64, GLint64)
 - Fixed so that PySide 5.6/5.9 is not linked against the Python shared / static library (this was wrong because the symbols need to be picked up by the dynamic loader from the python interpreter itself). http://bugreports.qt.io/browse/PYSIDE-522 
 - Completed move of examples from pyside-examples.git to pyside-setup
 - added additional OpenGL2 core example
 - fixed testrunner (these silently failed since forever) https://codereview.qt-project.org/#/c/216732/
 - fixed issues whereby wrong QObject ctors were selected
 - removed webkit from Pyside2 build  
 - talk about Windows specific instructions for Getting Started Guide 
 - Qt3D issues in Qt 5.10 fixes
 - clazy warning fixes
 - Stable ABI work (started to convert implementations) (see PYSIDE-560)
   - requires adoption of HeapTypes
 - QML tutorial updates
 - Doc snapshots being prepared

11. January 2018

- multi pyside packages (not easy to do)
  - unlikely to achieve until TP
  - a package per python release (3.4, 3.5, 3.6, 2.7) 
  - package per platform (win 32bit|64bit, mac, linux)
  - long term task to convert to reduced Python API (to support multi 3x packages)
  => have to cut some targets: one 3x and 2.7 release
  -> create doc howto to encourage custom builds for not covered platform combo 
  -> do we need commercial for release
- pysinstaller is an installer 
  -> deployqt tool not necessary
  -> documentation required stating the option
- rename import options
  - rename via alias possible 
  -> still an issue when looking up particular types but probably manageable   
  - renaming option: Qt4Pyton, QtPy, QtPython, Python4Qt, Qt
- example shift
  - https://codereview.qt-project.org/#/c/215900/
- Basics for doc generation toolchain
- Issues where cmake picks up platform Qt references where it should use a Qt custom install
- Qt3D bindings still under progress
- PYSIDE-34 & PYSIDE-560 
- Documentation
  - minor doc fixes
  - documenting examples

4. January 2018

- multi pyside packages (not easy to do)
- renaming Pyside to "Qt for Python"
  - quite challenging due to implied import and lib changes
  - alias for existing imports would be beneficial
- discussion around existing function test
  -> working assumption is that we ignore added function and check for existing base line only
  -> effort too large to maintain every single new addition
- discussed issues around concurrent existing Qt builds while building pyside (potential cross leakage)
- various fixes to doc generation steps (see PYSIDE-363)
- discussed need for good examples as part of documentation
  -> considered options to cleanup and remove need for submodule

2017

Expand

21. December 2017

- Issues building for Windows for Python 2 and 3 (on coin)
- discussing potential pyside demo/example for RTA purposes
- Continued work on macOS / Qt 4 header issues, progress is there
  - breaks GL signature test on Linux, needs further investigation
- Investigated PyInstaller (potential option to deploy apps written with Pyside)
  - seems to work almost out-of-the-box for deploying PySide apps
- Investigated building one package compatible with multiple Python 3.x minor versions
  - not possible currently, will require a lot of work, possibly not feasible
  => we need to build 3.4,3.5,3.6 * win, linux, mac packages = 9 packages
  => 3 more for each platform and Python 2.x
  => commercial packages?
- lots of discussions and fixing of doc tool chain (qdoc in dev currently broken)
  - adding documentation has begun in earnest
- continued work on PYSIDE-578

7. December 2017

- pip wheel installation 
  - adding support for simultaneously python 2 and 3 builds
  - open issues: test installation towards multi python 3 versions
  - delivery channel for Pyside binaries (pip server? local install from qt account download?)
- investigate options to have sth like pyinstallqt (pysinstaller is a possibility)
- final keyword support by shiboken
- Qt3D bindings patch close to completion - > requires some Qt changes in 5.9.4 or later Qt
- fixing pyside build issues when encountering Qt 4 headers on macOS
- PYSIDE-578 (testing procedure/testrunner) -> adds heuristic to adapt to failure rates (flakiness handling)
  - concerns that running all tests 5 times adds too much load on the CI (test runs needed)

30. November 2017

- packaging related coin changes
- last weeks merges between 5.6 and 5.9+ blocking coin have been fixed (in particular on RHEL)
  - 5.9 + 5.6 are passing, dev has some private Qt changes which will affect pyside/dev
- lots of coin infrastructure issues 
- RTA test wheels added (install and launch single example)
- crash bugs in 5.6 branch reported but not details known yet (waiting for more details)
- some discussion what changes should be put to 5.6 vs 5.9 branch
- Qt3D bindings started
  - some changes in qt3d/5.9 required)
  - shiboken bugs discovered in the process (virtual function guessing & some comparison operators for namespace generated)
- small regression related to rpath handling on Linux
- open pip wheel issues:
  - handling of multi Python version wheels
- testrunner bugs addressed and reducing flakiness
- issues in Pyside when building with gcc 7.x
- qdoc from qt5/dev is required for pyside due to webxml support
  - it is a problem for the pyside/5.9 branch

23. November 2017

- pip wheel support issues on Linux solved (not yet verified)
  -> libICU usage on Linux should be enabled via pyside setup script
  -> downloaded from qt account download area
  - discussion how to do wheel builds for different Qt base lines -> implies to different wheel packages
  - need to evaluate how different python versions on various Linux distros impact wheel binaries
  - target: 6 wheels (2 per platform for 2.7 & 3.5+ Python)
  - OpenGL example
  - merge 5.6-> 5.9 attempted
    - signature update to match Qt 5.9 ABI/signatures
    - plenty of testing flakiness and also test crashes 
  - improvements to signature tests

16. November 2017

- Build system cleanup. It is now possible to build only a subset of Qt modules per cmd line option
  -  this is not complete and may need further patches
- Added QAxContainer, more classes from 5.10
- Preparing to add Qt3D
- Removed flaky test
- further research on how pip wheels might be used for Pyside
  - seems to work well on win & macOS
  - problems on Linux related to libICU (or in general due to different system libraries)
- implemented and tested --standalone feature for the main Qt platforms PYSIDE-558
- open question remains how 3rdparty Pyside deployment path might look like
- PYSIDE-510

9. November 2017

- Work on simplifying the build system, made it possible to specify a module subset to build
- PYSIDE-558 Fixing rpaths and making --standalone work on the 3 platforms. Result: Self-contained binary wheels. Needs Review.
- Trying to setup the project for building the docs. I managed to build the rsts from the patch in gerrit but running sphinx on those rsts results in errors about python 2 style code.
- PYSIDE-510 work continued

2. November 2017

- Releasing:
  - continued work on macOS installer 
  - binaries must be patched during the install from repo (rpath setting)
  - adopting the scripts from Qt for Pyside
  - made pip install work on Mac -> would be alternative to online installer approach
    - two deployment choices available
- discussions on how void* pointer should be handled, two options on the table
  - add additional Pyside module handling such pointers
  - add feature to shiboken to transparently add the support to any arbitrary C++/pyside project using shiboken
- documentation snapshot generated
- simplified setup-pyside and related build script options

26. October 2017

- Releasing:
  - test repo for online installer created (mac seems to deploy)
  - first evaluation of suggested installer approach on various platforms
- Documentation:
  - Qt doc team joined the effort
  - first experiences are collected
  - should new documentation be done in rst rather than qdoc format?
  - Getting started, General platform requirements and general fixes turning up during generation
- PYSIDE-571 received further fixes and reviews
- REDHAT not working on Coin (caused by gcc header issues)
- other minor improvements

19. October 2017

- Discuss open Pyside 2 release & documentation
- Release packaging suggestion
  - Qt for Python installed via Qt installer and python scripts pointed to module via PYTHON_PATH
  - create test online repository for Qt installer
  - follow up how this Pyside installation approach might impact deployment of Python apps using Qt for Python
  - Linux distros shipping Pyside? (not sure yet either)
- coin
  - breakage in CI provisioning due to competing Python binaries coming from MSVC 2017
  - coin template needs updating
- PYSIDE-571 work progressing (ready for review)
- lots of fixes in Pyside dev branch and 
- feedback from Qt World Summit:
  - is Pyside on Android and iOS?
  - iOS probably out since platform cannot have interpreted code
  - Scriptable application use case was a sought after feature
  - qdoc got WebXML patch -> open question which version to use for doc generation across the various pyside versions
    - add script to support single command to generate documentation

12. October 2017

- no update due to Qt World Summit 2017

05. October 2017

- Qt DataVisualization port has finished including an example
- Coin no progress due to pending Qt 5.9.2 release
- webxml support being implemented inside qdoc (beginning of the pyside documentation toolchain)

28. September 2017

- 5.6 and 5.9 branches are *finally* enforced by CI - yeeeaaahh
- Dev branch is still missing some forward merges before it can be enabled.
- port of QtDataVisualization progressing
- fixed bugs: PYSIDE-510, PYSIDE-517, PYSIDE-562, PYSIDE-564
- PYSIDE-156 in progress, first example of qApp protection, benchmark of impact ongoing

21. September 2017

- Prototyping and exploring of Pyside packaging options
- PYSIDE-510 done and merged
- PYSIDE-156 being addressed thanks to PYSIDE-510
- PYSIDE-487 Qt Data Visualization port progressing

14. September 2017

- Coin
  - 5.6 testing is back online (5.9 target unblocking pending a merge)
  - Packaging of Pyside started to become a discussion topic
    - Standard PIP deployment
    - Separate delivery via the usual Qt account channel
- void* support has a pending patch (https://codereview.qt-project.org/#/c/205124/ and related example https://codereview.qt-project.org/#/c/205272/)
- Documentation work ongoing
  - WebXML support was added to qdoc - see PYSIDE-363
  - inputting the resulting WebXML to shiboken gave promising results
  - Qt Creator gain ability to fold away Python license headers 
- Signature patch (https://codereview.qt-project.org/#/c/198654/)

7. September 2017

- Pyside CI broken (timeout for tests is zero)
  - Pyside provisioning scripts are very fragile
  - Qt's own  provisioning is interfering when e.g. Pyside relevant tools get changed by Qt
  - new ideas and redesign is likely to be needed 
- New Bindings: QRegularExpression, QVersionNumber, QUuid, 
- Discussions around Documentation for pyside
  - couple of options for generation pipeline
    1.) qdoc generates WebXML -> Shiboken converts WebXML to ReST (used to work in Qt 4 days)
    2.) qdoc is lib which creates AST - > shiboken uses the AST to generate ReST
    3.) \pyside to document python classes (lots of documentation effort and not ReST integratable)
    4.) minimalistic approach is to merely link to the Qt object class reference (python specific docs are somewhat separate
- PYSIDE-516 further investigation how to deal with void* APIs
  -> prototyping ongoing
- PYSIDE-510 redesigned once again (no more internal Python dependencies, Python version independent implementation)
  

31. August 2017

- 5.9 passes on all platforms but one (10.10 still failing)
  -> fix for provisioning scripts pending
- 5.6 no problems at hand (all passing)
- PYSIDE-516 hit a dead end
  -> fundamental way of passing vertex data (void*) pointer between Python and C++ not addressed yet
- PYSIDE-510 complete

24. August 2017

- Coin 5.9 provisioning is in place, but there are some issues with paths to tools which are being addressed
- Build issue on Windows 10 5.9 branch, addressed by patch in review https://codereview.qt-project.org/#/c/203406/
- Fix to macOS build when old Qt 4.8 is present, addressed by patch https://codereview.qt-project.org/#/c/203313/
- PYSIDE-516 progress on bringing in more of the QOpenGL classes
- PYSIDE-560 stable ABI was investigated, decision is required
- PYSIDE-510 finalizing license modifications
- PYSIDE-156 ongoing implementation for fixing qApp crashes
- PYSIDE-552 in process of being merged
- PYSIDE-562 pending patch
- PYSIDE-563 in process of being merged
- PySide 2 presentation being prepared for Qt World Summit 2017

17. August 2017

- Almost everyone is still on holiday - not much progress
- PySide 5.9 branch CI changes are still pending integration / ongoing due to issues with the CI infrastructure switch
- Investigation ongoing regarding QSSL classes on 5.9 branch
- PYSIDE-510 - ongoing work to make separation of different licensed code
- PYSIDE-156 - initial work to stop crashes when using methods that require an allocated QApplication instance
- PYSIDE-558 - initial investigation on how to proceed
- Updated the PySide2 wiki documentation to clear up build steps and requirements.

10. August 2017

- holiday time is hampering project at the moment -  not much progress
- CI changesin Qt affected Pyside project such that focus was on different CI matters
- PYSIDE-557 being addressed
- some very minor issues were fixed

03. August 2017

- PYSIDE-510 under code review https://codereview.qt-project.org/198654
  -> license issues have to be addressed
  -> some testing aspects still open (delayed till later on)
  -> investigation of impact on deployment across multiple python run-time versions required
- missing bindings script/wiki page updated: Missing PySide2 bindings
- first working array support has been delivered (some adjustments still required), now proceeding to extend for OpenGL PYSIDE-516
- Missing classes in works (e.g. QAbstractnativeEventFilter, QFileLock,....)
  -> revealed some issues of handling void* pointers

27. July 2017

- Fix for PYSIDE-331 causes lots crashes in auto tests
  -> more investigation ongoing
- coin
  - since 5.6 is now enforced by Qt CI the focus shifted to the 5.9 branch of Pyside
  - still encountering build issues on 5.6 though
  - cmake upgrade for macOS 10.10
  - missing virtual env for macOS 10.12 being deployed
  - msvc 2013 being removed from CI targets
  - clang on OpenSuSE deployment ongoing
  - lots of effort went into debugging the above Coin issues
- numpy support for 2d array feature development progressing (PYSIDE-354)

20. July 2017

- coin
  - license test passing
  - setup issues wit Python 3 and msvc 2008 vs 2015 issues
  => need to install newer Python 3 binaries
  => otherwise looks like enforced testing can be set alive on 5.6 branch
 - 5.9 still having clang header issues
- 2D array support work in progress
- PYSIDE-331 -> waiting for PYSIDE-510 dependency
- PYSIDE-510-> issues with Python 2.x left
- need to get a handling what functions are not supported by pyside due to exclusions or missing support in shiboken

13. July 2017

- coin
  - Production coin was updated today and all fixes are finally executed
  - 5.6 
 	- macOS 10.11 machines have Python 2.6, which causes test script to fail due its usage of a 
         module added in 2.7. Investigation in progress on the best way to fix this.
 	- build failure with Windows 8.1 (msvc2013-x86) 
  - 5.9 
 	- macOS 10.12 missing virtual env (https://codereview.qt-project.org/200006)
 	- Rest Linux/macOS configs are failing various build issues
  - dev is still pending for qt5 merge from 5.9   
- PYSIDE-510 development completed, test added, needs review
- PYSIDE-331 ongoing work
- PYSIDE-550 investigated, needs a working solution to allow proper building on distros that separate Qt private headers into separate packages
- PYSIDE-354 ongoing work on C / C++ array support in PySide, some questions remains on how to deal with xml syntax and multiple dimensions.

29. June 2017

- coin
  - Qt 5.6 all coin changes for Pyside are done but waiting for coin production update (after Qt 5.9.1 release)
  - Pyside/Qt 5.6 on new OpenNebula infrastructure confirmed to run
  - Qt 5.9 based setup still misses clang libs (merge pending)
  - Pyside/Qt 5.9 test failure fixes on macOS
- worked on array support (passing arrays between Pyside and Qt API) - work in progress (PYSIDE-354)
- continued work on debug builds across the platforms
- embedded Python example under review (further testing across the platforms)
- PYSIDE-510 finished (not yet merged) -> revealed a few more  problems in PYSIDE-331 & PYSIDE-308
- added a way to dynamically detect the available Qt modules in Pyside

22. June 2017

- building pyside on Win with debug and non-debug
- QSSL* classes do not build on macOS & Windows (patch in progress)
- dll name handling work finished (build system issues were addressed)
- PYSIDE-531 discussions 
- coin
  - followup from 18 May (windows issues in coin)
  - test runner changes being merged should address this 
  - new regression on macOS targets though
  - 5.9 issues due to clang continue to exist (will be addressed after 5.6 is enforced)

15. June 2017

- dll name handling fixed across the platforms (e.g. pkgconfig handling)
- PYSIDE-500 fixed
- PYSIDE-331 is still not done (in parts a module by module fix is required)
- PYSIDE-510 ongoing
- no further progress on coin
- Getting started guide has been updated dealing with Qt 5.6 and Qt 5.9 based Pyside builds
- scriptable Pyside example been developed to demonstrate SDK-like usage of Pyside
- lots of small infrastructure changes/fixes

8. June 2017

- Proposed solution for PYSIDE-500
- Further work on example for PYSIDE-526, work on shiboken2 command line handling to ease usage with qmake
- Updated Getting started Wiki
- Introspection work continued (PYSIDE-510)

1. June 2017

- PYSIDE-500
- https://wiki.qt.io/PySide2_GettingStarted needs a serious cleanup
  - 5.6 vs 5.9 differences not explained
  - specify exact version matching conditions
- pyside repo merge completed
 - Coin changes required to catchup - > recent CI issues have caused delay
- Introspection work continued (PYSIDE-510) -> not yet complete
- qtcharts and related examples have been ported (5.9 only)
- license header checks of Pyside for Coin able to distinguish branches
- Array support for Pyside missing (numpy_array support?)

18. May 2017

- Coin
 - Win nodes didn't run the cleanup functions -> test failures due to artifacts -> no solution yet
 - Clang issues persist and this must be addressed in the broader scope of Qt/qdoc depending on libclang too
   -> until this point in time the setup is a manual one for pyside/5.9+
- merge of psyide repositories -> no objections against the plan in the community
  -> merge will happen in the next few days (pyside.git, pyside-setup.git and shiboken.git become one repo)
  -> examples, wiki and tools will remain as they are
- QOpenGL* porting continuing
- qttext2speech ported to Pyside2
- failing QML tests in Qt 5.9 based Pyside 
- PYSIDE-510 - introspection solution in the works
- Qt World Summit Berlin 2017 (talk for Pyside being handed in)

12. May 2017

- looked into the Windows tests failures, and the reason might be related to how we copy around build artifacts, leading to an incorrect folder structure. Coin team to look at it.
- Fixed a small regression in 5.9 with building on macOS after some cmake changes that happened in 5.6
- Implemented, tested, reviewed and committed the multimedia widgets examples
- documentation from Pyside1 days can not be generated anymore
  -> new approach required using exiting Qt5 doc tools
- QOpenGL* port to Pyside has started -> missing support for arrays in shiboken
  - more thorough design discussion needed
- Coin
  - provisioning for clang in Qt CI ongoing: https://codereview.qt-project.org/#/c/192517/10

04. May 2017

- Coin
 - libclang 4.0 provisioning available (code reviews ongoing -> enabler for Pyside and qdoc testing)
 - 5.6 coin issues resolved
 - macOS related CI/cmake failures were resolved
 - some CI sync issues among pyside submodules (always submodules HEAD is tested)
 => may have to change once we come closer to the final release ->
 => fine as long as we are in heavy development mode
 - discussion around Ubuntu and OpenSuse configurations
 - minimal cmake requirements not met on all distros
   - 5.6 only be build using RedHat 6.6
   - 5.9 builds have updated Linux distros -> may be able to obtain a new cmake version without hacking the standard toolchain
- QtMultimedia and QRaster* API bindings further completed
- added tests and examples for the above bindings/APIs
- PYSIDE-510 (required for simplified and more generic testing of bindings)
- PYSIDE-504 work is ongoing
- PYSIDE-507 -> avoid hardcoded includes when certain Qt modules are not available
- brief discussion whether to merge the shiboken and pyside repo
  -> advantages from a CI perspective and reduces pyside setup complexity

26. April 2017

- Coin 
  - provision for 5.9+ branches -> we would like to use libclang 4.0
  - test still not yet executed due to branch mismatch which pulls in dev branches
- missing bindings in QtCore/QtGui/QtMultimedia - work in progress
- clang support done (except for a Mac issue)
  - Pyside clang version required Qt 5.7 or later
- PYSIDE-500 done
- work on PYSIDE-500, 502, 504, 497 will continue
- qdoc work started trying to recover whatever documentation is left from Pyside 1.x

20. April 2017

- Fix for PYSIDE-331 broke the Windows build; under investigation
- Refactoring, fixed build warnings
- Looked at debug builds on Windows

13. April 2017

- discussion around merge policies between 5.9 and 5.6 branch
- Linux debug library issues fixed
- PYSIDE-488 fixed via workaround
- fixed a unit tests
- worked on qdoc/html generation in Pyside (failed so far - waiting for qdoc maintainer feedback)
- PYSIDE-331 patches merged -> more bugs found in the meantime

6. April 2017

- 5.6 branch created
- 5.9 branch to be created as well (easier setup for Qt 5.9 testing in Coin)
- Coin
  - not passing build platforms have been disabled
  - otherwise COIN passing on 5.6 (except for 2 failing test) -> soon to be enforced
  - license checker passing in dev
- clang changes merged to dev branch (after reviews)
  - clang provisioning on Coin still missing
- for now shiboken is not a generic C++ bindings generator (targeting Qt use cases only)
- TODO: enabling the generation of documentation from repos
- fixed mixed usage of debug and release build (no debug builds for windows)
- [PYSIDE-331] work continuing - fixed, tests are missing

30. March 2017

- Coin
  - some platforms passing and will be enabled by default
  - MinGW, some OSX and cross compile targets remain out
- will branch pyside dev branches to 5.6, new dev becomes clang branch
  - regular merges from 5.6 -> dev will start happening
  - Pyside 5.6 branch will continue to work against Qt 5.6
- need reviews for PYSIDE-323 to merge clang changes
  - Qt 5.10, 5.9 (on main desktop plawtforms working)
  - for now the Pyside dev development will be based on Qt 5.9 (to be bumped up later on)
- [PYSIDE-331] - in progress
- [PYSIDE-156] - 

23. March 2017

- Coin
  - progressing
- More tests fixed
- wip/clang branch created, will receive Clang parser port with instructions
- Refactoring of shiboken, replaced QtXml classes by QXmlStreamReader allowing for stricter error checking

16. March 2017

- Coin
  - the fixes are still integrating or are under review (no further progress until this is done)
- clang parser replacement PYSIDE-323
  - most test failures related to clang are fixed
  - tests ran pretty much through on Qt 5.9 (Windows and Linux - Mac not yet verified)
  - code cleanups in shiboken
  - will create feature branch on all pyside repos to get the clang patch series under CI control
  - merges from regular pyside branches into clang branch will commence
  - readme required that explains how clang is to be built (if not provided by platform)
- [PYSIDE-331] - in progress

09. March 2017

- Coin
  - lots of changes in Gerrit for Pyside and coin
  - the outcome has to be check once everything merged
  - upcoming clang dependencies in shiboken introduces new requirements for Coin
  - clang changes to be dealt with after pyside branching
- clang
  - first complete bindings generation with Qt 5.6 and Qt 5.9
  - some failing unit tests which have to be looked at individually
  - actual merge depends on pyside branching which in turn depends on recent Coin changes
- smart pointer support
  - patches generally done, gerrit review ongoing
  - rather large patch
- created list of missing bindings in Pyside (Missing Bindings)
  - mostly class level view, global functions at al not covered
- Jira cleanup
  - PYSIDE-464, PYSIDE-217, Pyside-224 fixed
  - most bugs reviewed now and valid ~100 bugs were identified

02. March 2017

- Coin
  - Coin runs tests on Windows and OSX (some failures in the test - to be investigated)
  - Windows provisioning somewhat blocked due to CI issues
  - no progress on Linux
  - OSX 10.8 ran tests with expected results
  - OSX 10.9 has issues due to Pyside not supporting namespace builds of Qt
- clang
  - preprocessor has remaining issue
    -> with clang compiler but gcc seems OK
    -> Qt Core wrapper close to compiling (QHash issues remaining)
- Smart pointer work continues
  - some uses cases work, other use cases might remain
- PYSIDE-364 was fixed (patch pending)
- General error review on bugreports.qt.io
  - 50% done (prioritized bugs are handled/checked bugs - "done" bugs)
- ran address and leak sanitizer over Pyside which resulted in some worrying cases (need to be addressed going forward)

22. February 2017

- PYSIDE-462 Improved solution approved
- PYSIDE-205 Hard-to-detect memory leak plugged
  - slow memory leak found in shiboken with thousands of false positives
  - used differential analysis of valgrind output
- Clang: Fixed Preprocessor handling (set defines, include paths), got minimal binding tests to pass, now adapting MetaBuilder and PySide to what Clang finds when parsing Qt. Good news: Very little need to set Q_DOC or other magic defines
- Coin: Slow progress, currently struggling with provisioning
- extensive bug triaging ongoing in Pyside Jira project

16. February 2017

- Coin (no update - other priorities in release/CI team)
- Clang (progressing on pre-processor work)
- PYSIDE-315 closed
- PYSIDE-462 more review needed
- Shared pointer API very slowly progressing (lots of different fixes needed)
- QMimeDataBase support added
- QUrlQuery support blocked on QDOC defines in Qt sources (shiboken sets the wrong define)
- chasing memory leaks (PSYIDE-205)
- extensive bug triaging ongoing in Pyside Jira project
  - confirming and testing the reported bugs (70 of 200 processed)
  - some smaller bugs were fixed in the process
- quite a few bugs were discovered this week while working on the above items, e.g.:
  - deployment of pyside applications partly broken due to hard-coded install paths in Qt libs
  - Qt events are swallowed when Pyside error/exception occurs

9. February 2017

- Clang progressed
  - pure C++ tests for API extractions are passing (C++ parsing based on clang)
  - even works with cmake
  - old C++ preprocessor still to be replaced
  - working through old preprocessor and checking what magic it did and how clang might be able to replace it
- PYSIDE-315 & PYSIDE-462 fixes pending for review
- COIN
  - continue progress an getting more platforms to pass (unblocking one step at a time)
  - Windows (32bit) and mac are building, Win 64bit still not building
  - OSX and Windows are stopping in the test runner phase
  - missing cmake update still on Linux (provisioning update required)

26 January 2017

- PYSIDE-315
  - principle fix merged but some minor improvements still pending
- Pyside-462
  - essentially a feature request as C++ default parameters are not yet supported by Pyside
  - discussion ongoing how to address the problem
- COIN 
  - 10.11, 10.9 & 10.8 build passes
  - 10.10 still failing during build
  - Progress on Windows and running into new problems further down the path
  - Redhat failed to general brokenness of the platform in CI
  - Ubuntu no further progress
- clang parser work progressing
  - completeness of work is currently measured by passing unit tests
  - there is a long way to go 
  - see PYSIDE-323 and associated patches for progress monitoring
- Fixed regression which prevented Pyside to compile with Qt 5.5.1
- shared pointer support
- no update on 2nd February 2017 (next week)

19 January 2017

- PYSIDE-315
  - a possible fix is pending, some minor performance improvements are still possible
  - caused by different signal/slot ordering in Qt4 and Qt 5
- COIN 
  - issues on 10.8 and 10.9 platforms should pass now
  - 10.11 & 10.10 still have issues
  - Windows timeout problem fix in COIN (but no COIN update until 5.8.0 released)
  - Linux builds are failing
    - Ubuntu 14.04 fails due to cmake issue
    - Redhat 6.6 & OpenSuse 13.01 deferred
  - current COIN freeze for 5.8.0 release affecting patching of COIN for Pyside
  - next steps in priority -> run tests on 10.8/10.9 and get Linux running
- PYSIDE-79 regression fixed as well (some interaction between PYSIDE-315 & PYSIDE-79)
- PYSIDE-462 to be looked at next
- clang parser work progressing

12 January 2017

- PYSIDE-315
  - sorting of slots/signal connection changed on Qt side and Pyside side has not caught up
  - had similar connection issues on the QML side, need to investigate whether there is a connection
- PYSIDE-79
  - caused a regressions (not yet investigated)
- fixed OpenGL types not being recognized on MacOS (partly fixed)
- API's with shared/smart pointers in Qt don't work
  - has potential long term effects and investigation has started into the reasons
- Refactoring shiboken in preparation for clang
- COIN currently runs tests with namespaced Qt
  - short term fix to exclude namespaced Qt builds with Pyside (broken on MacOS 10.9)
  - other failures: missing libraries on MacOS (10.11), missing provisioning on Windows, cmake issues on 10.10
  - unknown state on Linux (current Redhat too old)
- Prioritization after status round:
  - PYSIDE-315 to be investigated based on recent signal/slot patches for QML
  - Smart pointer issues reduced in priority to provide space for PYSIDE-315 & PYSIDE-79 regression
- verify that COIN runs testrunner (not just building Pyside)
  -> not yet verified since we are still failing builds in COIN

5 January 2017

- COIN update
  - COIN changes merged (no further patches pending)
  - need to run an integration test
  - issues related to different build platforms still to be expected
- Continue with clang
  - backtracking a bit (reusing some node API's during parsing but otherwise use clang to populate the tree)
- test blacklisting reviewed (some removed - mostly signal related ones, one new regression)
- regressions in QtQuick were worked around (caused by recent Qt Quick changes)
- issues with macos framework style includes in Qt
- PYSIDE-315 debugging ongoing, very hard to track down
  - was it a regression from a previous Pyside release (e.g. Qt 5.4?!?
- hard-copied Qt 5.4 based headers still in existence
  - updated needed but would shut Qt 5.4 users out

2016

Expand

22 December 2016

- lots of discussions around the COIN patches
  - source archive setup being under review 
  - Qt 5.6 provisioning patches merged
  - CI uses Python 2.7 at this stage
- clang C++ parsing continues
- fix of some Pyside unit tests (now have a clean slate again)
- finished QtQuick port
- some overflow problems have been fixed in shiboken

15 December 2016

- COIN changes pending for testing infrastructure
  -> https://codereview.qt-project.org/177136 (Pyside change)
  -> https://codereview.qt-project.org/176968 (Coin change)
  -> https://codereview.qt-project.org/179662 (Provisioning changes)
  -> https://codereview.qt-project.org/179663 (Provisioning changes)
- waiting for COIN development team to review/accept the pending changes
- no update in PYSIDE-315 (under investigation)
- Flushed out a couple of shiboken, QML, qml example bugs
- Some bugs related to parser delayed until clang parser task done (https://bugreports.qt.io/browse/PYSIDE-322)
- most basic shiboken API extractor test to pass (global enumeration test)
- some trouble with int size data types
  -> Python 2 & 3 are different and the relevant C++ data types are yet again different from platform to platform

8 December 2016

- COIN patches pending approval
  - Windows provisioning reviewed
  - Linux, Pyside, COIN itself
- Clang changes progressing, comparing AST tree from old parser and clang
- PYSIDE-79 done
- PYSIDE-315 under investigation
- QtQuick patches taking shape (some template magic and function pointer 
  features in Qt cannot be parsed by shiboken)

1 December 2016

- COIN 
- PYSIDE-79 there seems to be a final work around (https://codereview.qt-project.org/#/c/176374/)
  - fixing a few tests in the process
- clang update
  - dumping AST, identifying the required info
  - work in progress...
- problems with global static QObjects on the Qt side calling back into Python during app exit
  - may require some changes on the Qt side

24 November 2016

- automatic COIN triggering for submodules work in progress
  - several discussions on this topic this week
- Clang investigation (what library to use)
- fixing bugs on Qt side for Pyside
- Qt QML support almost done, work will continue with Qt Quick
- PYSIDE-79 work story continues
  - reference counting not quite accurate but point of deallocation of ref count not identifiable

17 November 2016

- Pyside team suffering from sickness -> not much progress this week
- Pyside-79 fix had a lot of negative side effects (breaking existing tests)
  - internal object reference counting is the predominant issue at hand
  - continue to work on the bug (no resolution yet as delayed due to sickness)
- COIN some progress but still open discussions on branching policy required

11 November 2016

- PYSIDE-79 being fixed
- PYSIDE-315 to be addressed
- Shiboken and Clang
  - familiarizing with clang and its parser's inner working
  - added experimental qmake project definition for shiboken (makes work in Qt Creator easier)
- Qt QML on Pyside work progressing
- work on COIN did not progress due to conflicting priorities inside COIN development

3 November 2016

- working through the QML stack esnuring all required API's are exported (https://codereview.qt-project.org/#/c/175682/)
- small build system patches
- PYSIDE-79 work progressing
- OpenGL support fixed (PYSIDE-314)
- COIN issues
  - repo interdepencies not working yet, suggestion under discussion and to be implemented
  - most license checks have been fixed
  - Qt 5.6 based Pyside to skip check
  - after branching for C++11 work the dev branch should work from license check perspective
  - eventually the entire CI needs to run through (more hidden problems could be hidden)
- Started working on C++11 compliant parser for shiboken
  - libclang will be used
  - first target is to replace the AST tree implementation
  - requires clang setup in the CI

27 October 2016

- Work on WebKit/WebEngine support
- Further work on COIN support
- Planning meeting
- Workshop for C++11 support in Pyside (PYSIDE-323)

20 October 2016

- Qt Quick support submitted
- QtOpenGL support submitted
- Further work on COIN support, license headers

13 October 2016

- Qt CI update
  - Coin changes have merged but integration not yet working
  - Qt CI enables on Pyside side merged 
  => COIN integration fails with license issues
  => requires review of license conditions for all files, some files are not even relicensable
  => skipping license check for now, most likely to be done later again but requires changes to license check script
  - more issues of interworking between QtCI and Pyside expected (won't be visible until license problems resolved)
  Open issue: does a change in Pyside repo trigger a rebuild of everything?
- Qt QML support progressing and first patches merged
  - serious bugs have been fixed, more complex QML examples are now working

29 September 2016

- Coin integration close but not merged
- Bugfixing in particular on the shiboken parser side
- QML/Python binding and tests fixed -> general check all day-to-day aspects of QML are working
- QML examples porting 
  -> still some failing tests
  -> Check all QML/Quick class are exported

22 September 2016

- Refactored Shiboken, udnerstanding build sequences
- black list for unit tests defined and tested
- fixing of tests
- QML example fixing continued (QML bindings not working)
- Pyside side for COIN done (pending integration checks)
- COIN integration still wip due to long test and retest cycles
- pyside and shiboken repo to be relicensed similar to other Qt products
=> this should address any issues regarding the status of generated code too

16 September 2016

- Pyside side for CI testing ready for testing
- status of COIN side to be determined. Code exists need help from CI team to confirm status.
- build system infrastructure improvements in pyside setup
- Qt logging now working
- and more
- Update on bug handling
  - PYSIDE-88 continuing
  - PYSIDE-349 (Multimedia ported)
  - PYSIDE-344 fix pending on codereview
  - make debug builds of Pyside work (OSX works, Linux has work in progress patch, Windows side awaiting contribution)
- QML support work in progress
  - examples are slowly ported with aim to identify bugs

25 August 2016

- Properly implementing QML experience in Pyside
- Unit test fixing
- automated CI testing for Pyside
  - working locally but still failures occurring on various other test machines

4 August 2016

- Additional Pyside examples under review
- Fixed warnings coming from Shiboken

21 July 2016

- main Pyside 2 example port done
- additional examples to fill gaps are being ported (as per prio list)
- CI patches running, but still gaps (should be done by next week)
- about 86% of auto test working (failing count of test 80+ auto test)
- Project Test Status

14 July 2016

- Pyside 2 examples ported
- OpenGL & SVG not working
- QtQuick 2 is in strange situation (QtQml depends on QtQuick)
- Python 3 realed Unicode handling not working with Qt
- QMessageBox hangs
- no documentation for any example
- lots of warnings when building wrapper
- results: https://paste.kde.org/pgje2toyh/mlbqic
- COIN setup for pyside https://codereview.qt-project.org/#/c/158336/
- Qt 5.7 still blocked due to missing C++11 support in shiboken
  - food for ideas: https://steveire.wordpress.com/

7 July 2016

- automated CI testing
- patches for Pyside and Qt CI side required
- Testing somewhat more complicated due to closed nature of Qt CI 
- script to port examples to Qt 5 -> https://bugreports.qt.io/issues/?jql=labels%20%3D%20exampleport5
- all examples to remain BSD licensed

Backlog

The list below represents some short term targets and long term ideas for Pyside2 development. There is no guarantee that this will happen as written down. It is meant as a short reminder and idea collection for future development. In the long term this list is likely to move to https://bugreports.qt.io/browse/PYSIDE.

1. First time with Pyside (Document how to get started on this wiki)
2. Testing Infrastructure
   1. What has been done so far?
   2. Qt CI support (PYSIDE-431)
      1. make test
      2. cmake based build system (all repos)
      3. pip installable Pyside
         1. CI has to setup virtual Python environment
         2. Use pip to install/build Pyside (release bundling)

1. Future changes to PySide
   1. Gaps (modules & API)
      1. Identify the extent of the gaps by porting existing examples from Qt4 to Qt5 (done)
      2. Plugin system (in what form or shape do we need it is uncertain - some failing tests exist already)
      3. Prepare a summary of gaps for other modules
   2. Fundamental Architecture changes
      1. Shiboken
         1. Clang support (PYSIDE-323)
      2. QtQML support (PYSIDE-355)
   3. MyPy (type annotation - fully typed Python API)
   4. General Python features and their adoption in PySide (e.g. Async I/O)
   5. Creator and tooling
      1. Fix QtC debugger to understand Python/C++ mixed internals
   6. Documentation generation
   7. Deployment of apps
   8. Debugging support

1. Must have tasks for first release
   1. Packaging (e.g. pip deployment? )
   2. Clarify Installer story
   3. Documentation
   4. Product placement decisions
      1. Licensing
      2. Naming
      3. etc
   5. Complete CI coverage