Qt Network Workshop 2016

A meeting held on the eve of the Qt contributors summit 2016.

Proposed Discussion Topics

Eddy has notes that shall transform the agenda into a contents list. Just need to bring tablet to work ...

1. Redirection support
   • Should it be on by default
   • more configurable...
2. The test server
   • How can we get the image - should we replace / update it? The Vagrant solution?
   • How can we add services etc.
3. How can we better manage the various openssl builds?
   • The variations in build options, versions etc. between the platforms and distros has got to the point of insanity
4. OpenSSL 1.1 support - status update and roadmap
5. What are the main use cases for QtNetwork: HTTP(S)?
   • Are there new use cases for e.g. IoT (UDP / UPnP etc.)?
   • Implement a generic REST(?) service to make it simple to interface with a Qt application?
6. IoT
   • Identify areas of QtNetwork we should focus on (HTTP(S) / low level sockets)
   • Are there areas in QtNetwork to support the IoT story?
   • Is there input from sales or so?
7. Performance
   • try to get rid of using the bearer classes (and thread) by default, they are seldomly used (this would imply deprecating QNetworkAccessManager::networkAccessible())
8. Providing widgets for QtNetwork
   • What to do about dependencies?
   • What should we include?
   • URLBar widget (danimo)
   • SSL configuration widget (rich)
   • What about a certificate dialog?
9. Should QtNetwork's design / architecture be modified?
   • Is it outdated in any area?
   • Could it be made redundant by using a 3rd party something?
10. Providing more server support for QtNetwork
    • QSslServer
    • An HTTP server?
    • Identify use-cases

Presentations

1. High-level design and structure of QtNetwork
   • Rationale for the design.
   • Peter's Qt HTTP class diagram
2. New in 5.8
   • HTTP/2
   • OAuth 1+2

REST Stuff

Copied from Richard Moore's Google doc

• Provide convenience API for REST services
• Server support
• Client support
• SSL and plain text
• Separate module (or perhaps 2 one for client and one for server)

Client

• Support for ‘pre-setting’ headers to be added to requests (Eg. for auth)
• Preset base location of the API
• Have a default QNAM

• Make QML etc. use the default one normally
• Talked to tronical and even though the QML one is in the wrong thread right now, that’s just an implementation detail and can easily be changed to use the main thread
• Allows caching etc. to be setup once
• Makes use of a single QNAM much easier than it is currently
• Still allows for multiple QNAMs when the app needs different security contexts etc.
• Need to ensure the creation is thread-safe
• What do we do with the old one? Just delete it and document that you should set it early in the app lifecycle?
• Beware of people who are already holding a pointer to it

• possibly some static methods to make performing an HTTP request trivial

• QNetworkReply *QNetworkAccessManager::get(“https://example.com”);
• Not convinced these are a good idea, but they’re possible

• Easy conversions to QJsonDocument, QXmlStreamReader, others?
• Try to minimise the amount of boilerplate code needed by users

Server

• Provide a convenience QSslServer class (in qtbase)
• Provide a base-class for handling HTTP (in qtbase)
• Provide more specialised handlers such as a RESTy thing in the addon module
• Make it trivial to support both plain text and TLS with TLS being the (strongly) recommended approach

Research

• Look at some popular REST APIs
• Identify common elements that could be made easier by some convenience APIs

Write-up

In need of tidy-up and reorganisation, but let's have it visible ASAP ...

Attendees:

Those with knowledge

Peter Hartman, Richard Moore, Daniel (danimo), Markus (guruz).

TQtC Core/Network team

Fredrik de Vibe, Jesus Fernandez, Timur Pocheptsov, Edward Welbourne (taking notes).

Prod mgmt "stories" should be more public.

Richard: widgets won't die any time soon.

Recursive event-loops are a bug. Authentication tends to need it. Peter made SSL free of it. Getting there incrementally is tricky.

Auth rather poorly supported. Client cert only preset, not in response to request.

Redirect support problematic. Can enable/dis; should have signal-based handling; should support HSTS. Permanent HTTP redirects. Lack of permanent storage of details, except cookie-jar. Default no cache, due to no disk; have in-memory cacheing. Breaks incomplete cert-chains. Should cache intermediate certs (like FF). On MS, we do similar, if win-update has it. (Timur: can get MS to forget a cert.)

Need to extend disk cache. TLS session tickets.

QNAM helps with SSL setup (cipher suites, certs). Session tickets extend that, but only in memory. SSL context.

Any storage good. e.g. QSettings or SQLite. Certs, HSTS policies, client tickets. QNetworkDiskCache solved, Avoid depend on QtSql module. (Symbol clash ?) QSettings likely easiest. Latency is big issue. Need to set sane permissions rw------- on disk files.

Secure storage may be sensible generally (core).

OwnCloud, BlackBerry had real issues with the above. Persistent store enables many things.

Redirect off-by-default; needs enabled per request. Breaks ubiquitous use of redirects. Need security policy to automate at least the main cases. Current policy over-strict.

Some apps, that handle redirects self, might break; but most should be competent to cope with transition:

• first add QNAM option to default on per connection;
• later make this option on by default in QNAM.

Can do arbitrary fancy if torn off (current default); need to make default sane for convenient uses.

Site-redirects (within protocol) either happen or error: need policy-based to e.g. trust from-sites, to-sites, as easy to set up - presently has to be handled by slotting signal on redirect.

(digress: tranfer-encoding vs content-encoding; decompression is cheap.)

Net-test server

Jed: need to reinvent. Currently can't get image of old box ! Expect we shall. But why ?

Should move to Vagrant. Can CI-control; shall need some CI fixes. Just need to know what's on present server.

Tried before w/ puppet, but problems. Fairly sure vagrant sounder. Need to make it work for end-devs; two download servers vagrant consults, internal to CI and a public one. Need lists of

• services we need for present tests
• some services we should be testing against
• services we shall want

Peter's VMware image is used (Richard supplies).

For flaky tests: dev-boxes too powerful. Need to stress box while running tests. VSphere messes w/ clocks, too.

Vagrant issue on first run : fetch full Ubunu image. (Ubuntu LTS, not debian; SSL breaks.) LUbuntu (lightweight) maybe. Can cope.

Probably need to fix many tests, too.

Should also have a win-net-test server to test against. Future; license ok for manual testing; ok on vagrant. Test against real MS-version of service. e.g. NTLM service.

OpenSSL

versions >= 0.9.8 ok. should be >= 1.0.2 (LTS)

We have compile-time checks, feature support; and run-time checks on library loaded, feature available. But defines are -ve; need version info to un-negate. Richard has way to centralise and transform to positive tests of defines. To add: runtime checks to match.

disable 0.9.8: problem - search finds wrong version on Mac; dlopen woes; can fix, but there are complications. Richard has changes for this.

Debian disables TLS 1.0 without right define. Can't work round this. RH set defines right, but disabled SSL 3; broke tests. (Can work round.) Discovering supported feature-set is a problem, even when we know version. Need someone on deb-test to catch this before it hits stable and breaks us.

HTTP/2 needs OpenSSL, Apple's secure transport not adequate. Want 1.0.2 and >= 1.1

For 1.1, Richard has a clean slate, due to big API changes. It works. Need update now 1.1 is released; fix config tests; should now be stable. Wait for LTS of it before we recomend it; keep 1.0.2 as recommend until then.

Non-OpenSSL variants complicate life. Some re-use OpenSSL names, complicate version-checks.

QNAM architecture

Peter's blog post - link in wiki - find in earth-team IRC (9:53) or in e-mail. Markus draws diagram, talks us through it. (I have photo.) Discussion. Second picture.

SockePrivate - cross-platform SocketEngine - implements socketry; plat-native or virtualized (e.g. via proxy or SSL)

Some protocols (eg SMTP) have to talk raw over the Ssl socket before they can begin encryption. QSslSocket could lose its internal TCP, but MS's SSL sockets aren't the same as plain sockets (? at least WinCE) can't simply use a QTcpSocket.

WebSocket different, buggy; needs review of open bugs; but works.

Network is in core so DBus can use; DBus is used by accessibility, xcb plugin, printing, ... (ask Thiago). Can't move until Qt6 - moving symbols between libs is bad. Qt Lite can disable http, QNAM.

Bearer thread could go away. Bearer code could all go ? Connection management. Check with Lorn Potter.

Being on "a network" isn't the same as on "the internet". Should distinguish, using probe URL google provides. Ask user to log in via captive network if merely so.

Bearer thread scans for wifi; is expensive. Lorn/Ubuntu is expert.

NetworkConfig vs session

Use-cases for network in Qt

See above, DBus. Nokia wanted it. New things ?

CAN, IoT, OTA.

REST - can we do better - QJsonDocument -> QNetworkReply ? how ? Look at XmlHttpRequest. Make REST facade to hide most of mess from client. Handle repetitive header work. Result object can be JSON, XML, various custom forms (incliding borked JSON, XML); need to return as QByteArray, convenience methods for Json, ...

Could be external app, example code.

HTTP server - several on github - review them. Embed in an app as daemon. Add-ons for particular use-cases. REST client & server. Iillustration code, simple implementations of basics.

μSGI (micro-server gateway interface). pythonic web-server plugins.

IoT

Auto tends to use Qt only for UI and server-side. Don't want to be tied to us, so core is in-house.

PAN: Bluetooth LE (low energy) - we support it. Competes with ZigBee - lower level than our networking. MDNS - maybe worth exploring.

Many devices minimal Linux; many even more minimal - TCP stack + event loop. AMQP.

Worth making API coherence between our bluetooth and network code, so can tunnel latter via former tidily.

= Thiago reprise

DTLS - datagram SSL - UDP TLS. CoApp, CoRE group at IETF. More UDP support would be good. Can ignore proxying of UDP - no-one does it.

IoT all isolated objects talking to phone apps via cloud.

MPTCT - multi-path TCP; e.g. wifi and phone data as paths. TCP can open a second path to switch to it.

Performance

see bearer, abovve

Widgets for QNetwork

e.g. danimo has widget to display URLs, indicating security &c, anti-phish cues. Cert display dialogs, immune to XSS attacks &c. SSL config.

We leave these to client code; all easy to get wrong; all error-prone; errors are all security holes. Important add-ons, important to do in QML as well. Worth doing once, well, supported. So clients are more secure and our fixes benefit all.

HTTP request progress-bar.

Good candidate for add-on module.

Design revolution ?

Need for stable APIs - lacking in some. but

• libcurl - more SSL backends than us, good candidate.
• apache portable runtime ?
• node.js ?

We dlopen() OpenSSL for legal loop-holes, crypto export via USA; would need to hack libcurl (or ...) to do similar - not big problem. Good stable API in libcurl. Lacks server-side; we would need those. Worth looking at; ditch lots of code. Kill QNAM. Support sftp, file, ...

More servers

See above, use-cases, IoT.