Qt Network Workshop 2016

A meeting held on the eve of the Qt contributors' summit 2016.

Attendees:

Those with knowledge

Peter Hartman, Richard Moore, Daniel (danimo), Markus (guruz), Thiago (late arrival).

TQtC Core/Network team

Fredrik de Vibe, Jesus Fernandez, Timur Pocheptsov, Jędrzej Nowacki, Edward Welbourne (taking notes).

Initial random discussion

• Product management "stories" should be more public.
• Richard: widgets won't die any time soon.

Recursive event-loops are a bug.

• Authentication tends to need it.
• Peter made socket layer free of it; needs similar changes on the HTTP layers.
• Getting there incrementally is tricky.

Authorization is rather poorly supported. Client certificate can only be preset, not in response to each request.

Redirects and persistence

Redirect support is problematic.

• Can (dis|en)able; should have signal-based handling; should support HSTS.
• Permanent HTTP redirects.
• Lack of permanent storage of details, except cookie-jar.
• Default no cache, due to no disk; have in-memory cacheing.
• Breaks incomplete cert-chains.
• Should cache intermediate certs (like FF).
• On MS, we do similar, if win-update has it.
• (Timur: can get MS to forget a cert.)

Need to extend disk cache (or add a separate TLS permanent store).

• TLS session tickets.
• QNAM helps with SSL setup (cipher suites, certs).
• Session tickets extend that, but only in memory.
• SSL context.

Any storage is good.

• E.g. we can have QSettings or SQLite-based back-ends (and we need something similar to QAbstractNetworkCache as a base class).
• Should store certs, HSTS policies, TLS session tickets.
• Avoid depend on QtSql module (so we probably can use QSqlite's API directly).
• (Symbol clash ?)
• QSettings likely easiest (could be the default 'back-end').
• Latency is a big issue (another big issue is probably multi-threading - it's probably not possible/not easy to make this store 'asynchronous' in QNAM's context).
• Need to set sane permissions (rw-------) on disk files.
• Secure storage may be sensible generally (core).

OwnCloud, BlackBerry had real issues with the above. Persistent store enables many things.

Redirect off-by-default; needs enabled per request.

• Breaks ubiquitous use of redirects. (Many sites use them extensively.)
• Need security policy to automate at least the main cases.
• Current policy is over-strict.

Some apps, that handle redirects themselves, might break; but most should be competent to cope with transition:

• first add QNAM option to turn on redirects as default per connection;
• later make this option on by default in QNAM.

Can do arbitrary fancy if turned off (current default); need to make default sane for convenient uses. Site-redirects (within protocol) either happen or error: need policy-based to e.g. trust from-sites, to-sites, as easy to set up - presently has to be handled by slotting signal on redirect.

(Digression: tranfer-encoding vs content-encoding; decompression is cheap.)

Net-test server

Jed: need to reinvent. Currently can't get image of old box ! Expect we shall. But why ?

Should move to Vagrant.

• Can CI-control; shall need some CI fixes.
• Just need to know what's on present server.

Tried before w/ puppet, but problems.

Fairly sure vagrant sounder.

Need to make it work for end-devs; two download servers vagrant consults, internal to CI and a public one. Need lists of

• services we need for present tests
• some services we should be testing against
• services we shall want

Peter's VMware image is used (Richard supplies).

For flaky tests: dev-boxes too powerful.

• Need to stress box while running tests.
• VSphere messes w/ clocks, too.

Vagrant issue on first run : fetch full Ubunu image.

• (Ubuntu LTS, not debian; SSL breaks.)
• LUbuntu (lightweight) maybe.

Can cope.

Probably need to fix many tests, too.

Should also have a win-net-test server to test against.

• Future; license ok for manual testing; ok on vagrant.
• Test against real MS-version of service.
• e.g. NTLM service.

OpenSSL

• We currently (unofficially) support versions >= 0.9.8 ok.
• We only admit to supporting >= 1.0.2 (LTS);; and we would rather make this reality.

Features are tricky between OpenSSL versions:

• We have compile-time checks for feature support;
• run-time checks on library loaded, for feature available.
• But defines are -ve; need version info to un-negate.
• Richard has way to centralise and transform to positive tests of defines.
• To add: runtime checks to match.

Aim to disable 0.9.8: problem – search finds wrong version on Mac; dlopen woes; can fix, but there are complications. Richard has changes for this.

To complicate the feature situation further:

• Debian disabled TLS 1.0 without setting the right define. Can't work round this.
• RH set defines right, but disabled SSL 3; this broke tests. (We can work round this.)
• Discovering supported feature-set is a problem, even when we know version.
• Need someone on deb-test to catch this before it hits stable and breaks us.

HTTP/2 needs OpenSSL, Apple's Secure Transport is not adequate.

We want OpenSSL 1.0.2 and >= 1.1

• For 1.1, Richard has a clean slate, due to big API changes.
• The new back-end works.
• Needs to do an update now that 1.1 has been released (this week); fix config tests; should now be stable.
• Wait for LTS of it before we recomend it; keep 1.0.2 as recommended version until then.

Forks of OpenSSL complicate life.

• Some re-use OpenSSL names, complicating version-checks.
• We should insist on a new back-end for each, just as for 1.1

QNAM Architecture

Peter's blog post covers some of this.

• Markus drew diagrams and talked us through them, we discussed them.
• Photos of diagrams: Media:QtCon2016NetArch.jpg, Media:QtCon2016Socket.jpg

SockePrivate

cross-platform

SocketEngine

implements socketry; platform-native or virtualized (e.g. via proxy or SSL)

We have to create the SSL socket initially as a normal socket:

• Some protocols (e.g. SMTP) have to talk raw over the Ssl socket before they can begin encryption.

QSslSocket could lose its internal TCP, but MS's SSL sockets aren't the same as plain sockets (? at least WinCE - but WinCE is going away) can't simply use a QTcpSocket.

WebSocket is different, buggy; needs review of open bugs; but works.

Network is in core so that DBus can use it;

• DBus is used by accessibility, xcb plugin, printing, ... (ask Thiago).
• Can't move until Qt6 – moving symbols between libs is bad.
• Qt Lite can disable http, QNAM.

Performance

Connection management:

• Bearer thread could go away.
• Bearer code could all go ?
• Check with Lorn Potter.

Being on "a network" isn't the same as on "the internet".

• Should distinguish, using probe URL google provides.
• Ask user to log in via captive network if merely so.

Bearer thread scans for wifi; is expensive. Lorn/Ubuntu is expert.

NetworkConfig vs session

Use-cases for network in Qt

• See above, DBus.
• Nokia wanted it.
• New things ?
• CAN, IoT, OTA.

REST - can we do better - QJsonDocument -> QNetworkReply ? how ? Look at XmlHttpRequest. Make REST facade to hide most of mess from client. Handle repetitive header work. Result object can be JSON, XML, various custom forms (incliding borked JSON, XML); need to return as QByteArray, convenience methods for Json, ...

Could be external app, example code.

HTTP server - several on github - review them. Embed in an app as daemon. Add-ons for particular use-cases. REST client & server. Iillustration code, simple implementations of basics.

μSGI (micro-server gateway interface). pythonic web-server plugins.

REST details

Copied from Richard Moore's Google doc

• Provide convenience API for REST services
• Server support
• Client support
• SSL and plain text
• Separate module (or perhaps 2 one for client and one for server)

Client

• Support for ‘pre-setting’ headers to be added to requests (Eg. for auth)
• Preset base location of the API
• Have a default QNAM

• Make QML etc. use the default one normally
• Talked to tronical and even though the QML one is in the wrong thread right now, that’s just an implementation detail and can easily be changed to use the main thread
• Allows caching etc. to be setup once
• Makes use of a single QNAM much easier than it is currently
• Still allows for multiple QNAMs when the app needs different security contexts etc.
• Need to ensure the creation is thread-safe
• What do we do with the old one? Just delete it and document that you should set it early in the app lifecycle?
• Beware of people who are already holding a pointer to it

• possibly some static methods to make performing an HTTP request trivial

• QNetworkReply *QNetworkAccessManager::get(“https://example.com”);
• Not convinced these are a good idea, but they’re possible

• Easy conversions to QJsonDocument, QXmlStreamReader, others?
• Try to minimise the amount of boilerplate code needed by users

Server

• Provide a convenience QSslServer class (in qtbase)
• Provide a base-class for handling HTTP (in qtbase)
• Provide more specialised handlers such as a RESTy thing in the addon module
• Make it trivial to support both plain text and TLS with TLS being the (strongly) recommended approach

Research

• Look at some popular REST APIs
• Identify common elements that could be made easier by some convenience APIs

IoT

Auto tends to use Qt only for UI and server-side.

• OEMs don't want to be tied to us, so core is in-house.

PAN

Bluetooth LE (low energy) - we support it.

Competes with ZigBee - lower level than our networking.

MDNS

maybe worth exploring.

Many devices are minimal Linux; many even more minimal - TCP stack + event loop.

• AMQP.

Worth reviewing (and ensuring) API coherence between our bluetooth and network code, so can tunnel latter via former tidily.

Thiago reprise

Most of this write-up is chronological, but we skipped back to IoT when Thiago arrived

DTLS

datagram SSL - TLS over UDP.

CoApp, CoRE group at IETF.

More UDP support would be good.

Can ignore proxying of UDP - no-one does it.

IoT consists in reality of isolated objects talking to phone apps via cloud.

MPTCT

multi-path TCP; e.g. wifi and phone data as paths.

TCP can open a second path to switch to it.

Widgets for QNetwork

e.g. danimo has widget to display URLs, indicating security &c, anti-phish cues.

• Cert display dialogs, immune to XSS attacks &c.
• SSL config.
• candidate: HTTP request progress-bar.

We presently leave these to client code;

• all easy to get wrong;
• all error-prone;
• errors are all security holes.

Important add-ons, important to do in QML as well.

• Worth doing once, well, supported.
• So clients are more secure and our fixes benefit all.

Good candidate for add-on module.

Design revolution ?

Could we replace large parts of QNAM or even of our network stack ?

• Need for stable APIs - lacking in some
• libcurl – has more SSL backends than us, stable, good candidate.
• apache portable runtime ?
• node.js ?

We dlopen() OpenSSL for legal loop-holes, crypto export via USA;

• would need to hack libcurl (or other replacement) to do similar - not a big problem.
• Good stable API in libcurl.
• Lacks server-side; we would need those.
• Worth looking at; ditch lots of code.
• Kill QNAM.
• Support sftp, file, ...

Original Plan

For reference, here are the things we planned to discuss, linked to sections above in so far as we actually did so.

Proposed Discussion Topics

1. Redirection support
   • Should it be on by default
   • more configurable...
2. The test server
   • How can we get the image - should we replace / update it? The Vagrant solution?
   • How can we add services etc.
3. How can we better manage the various openssl builds?
   • The variations in build options, versions etc. between the platforms and distros has got to the point of insanity
4. OpenSSL 1.1 support - status update and roadmap
5. What are the main use cases for QtNetwork: HTTP(S)?
   • Are there new use cases for e.g. IoT (UDP / UPnP etc.)?
   • Implement a generic REST service to make it simple to interface with a Qt application?
6. IoT
   • Identify areas of QtNetwork we should focus on (HTTP(S) / low level sockets)
   • Are there areas in QtNetwork to support the IoT story?
   • Is there input from sales or so?
7. Performance
   • try to get rid of using the bearer classes (and thread) by default, they are seldom used (this would imply deprecating QNetworkAccessManager::networkAccessible())
8. Providing widgets for QtNetwork
   • What to do about dependencies?
   • What should we include?
   • URLBar widget (danimo)
   • SSL configuration widget (rich)
   • What about a certificate dialog?
9. Should QtNetwork's design / architecture be modified?
   • Is it outdated in any area?
   • Could it be made redundant by using a 3rd party something?
10. Providing more server support for QtNetwork
    • QSslServer
    • An HTTP server?
    • Identify use-cases

Presentations

It would be nice if those in the know could teach the rest of us:

1. High-level design and structure of QtNetwork
   • Rationale for the design.
2. New in 5.8
   • HTTP/2
   • OAuth 1+2