Qt Network Workshop 2016: Difference between revisions

From Qt Wiki
Jump to navigation Jump to search
(Missing section title)
(undo)
 
(18 intermediate revisions by 4 users not shown)
Line 1: Line 1:
[[Category:QtCS2016]]
[[Category:QtCS2016]]
A meeting held on the eve of the [[Qt contributors summit 2016]].
[[Category:Developing Qt::Network]]
A meeting held on the eve of the [[:Category:QtCS2016|Qt contributors' summit 2016]].


Attendees:
Attendees:
Line 8: Line 9:
== Initial random discussion ==
== Initial random discussion ==


* Product mgmt "stories" should be more public.
* Product management "stories" should be more public.
* Richard: widgets won't die any time soon.
* Richard: widgets won't die any time soon.


Recursive event-loops are a bug.
Recursive event-loops are a bug.
* Authentication tends to need it.
* Authentication tends to need it.
* Peter made SSL free of it.
* Peter made socket layer free of it; needs similar changes on the HTTP layers.
* Getting there incrementally is tricky.
* Getting there incrementally is tricky.


Authorization is rather poorly supported.
Authorization is rather poorly supported.
Client certificate can only be preset, not in response to each request.
Client certificate can only be preset, not in response to each request.
=== Redirects and persistence ===


Redirect support is problematic.
Redirect support is problematic.
Line 29: Line 32:
* (Timur: can get MS to forget a cert.)
* (Timur: can get MS to forget a cert.)


Need to extend disk cache.
Need to extend disk cache (or add a separate TLS permanent store).
* TLS session tickets.
* TLS session tickets.
* QNAM helps with SSL setup (cipher suites, certs).
* QNAM helps with SSL setup (cipher suites, certs).
Line 36: Line 39:


Any storage is good.
Any storage is good.
* e.g. QSettings or SQLite.
* E.g. we can have QSettings or SQLite-based back-ends (and we need something similar to QAbstractNetworkCache as a base class).
* Certs, HSTS policies, client tickets.
* Should store certs, HSTS policies, TLS session tickets.
* QNetworkDiskCache solved,
* Avoid depend on QtSql module (so we probably can use QSqlite's API directly).
* Avoid depend on QtSql module.
* (Symbol clash ?)
* (Symbol clash ?)
* QSettings likely easiest.
* QSettings likely easiest (could be the default 'back-end').
* Latency is big issue.
* Latency is a big issue (another big issue is probably multi-threading - it's probably not possible/not easy to make this store 'asynchronous' in QNAM's context).
* Need to set sane permissions (<kbd>rw-------</kbd>) on disk files.
* Need to set sane permissions (<kbd>rw-------</kbd>) on disk files.
* Secure storage may be sensible generally (core).
* Secure storage may be sensible generally (core).
Line 134: Line 136:
* We should insist on a new back-end for each, just as for 1.1
* We should insist on a new back-end for each, just as for 1.1


== QNAM architecture ==
== QNAM Architecture ==


Peter's [http://www.peter.hartmann.tk/single-post/2012/02/26/Inside-the-Qt-HTTP-stack blog post] covers some of this.
Peter's [http://www.peter.hartmann.tk/single-post/2012/02/26/Inside-the-Qt-HTTP-stack blog post] covers some of this.
* Markus drew diagrams and talked us through them, we discussed them.
* Markus drew diagrams and talked us through them, we discussed them.
* TODO: add photos
* Photos of diagrams: [[Media:QtCon2016NetArch.jpg]], [[Media:QtCon2016Socket.jpg]]


;SockePrivate: cross-platform
;SockePrivate: cross-platform
Line 295: Line 297:
* Support sftp, file, ...
* Support sftp, file, ...


= Original Plan =
== Original Plan ==
For reference, here are the things we planned to discuss:
For reference, here are the things we planned to discuss, linked to sections above in so far as we actually did so.


== Proposed Discussion Topics ==
=== Proposed Discussion Topics ===


# Redirection support
# [[#Redirects_and_persistence|Redirection support]]
#* Should it be on by default
#* Should it be on by default
#* more configurable...
#* more configurable...
# The test server
# [[#Net-test_server|The test server]]
#* How can we get the image - should we replace / update it? The Vagrant solution?
#* How can we get the image - should we replace / update it? The Vagrant solution?
#* How can we add services etc.
#* How can we add services etc.
# How can we better manage the various openssl builds?
# [[#OpenSSL|How can we better manage the various openssl builds?]]
#* The variations in build options, versions etc. between the platforms and distros has got to the point of insanity
#* The variations in build options, versions etc. between the platforms and distros has got to the point of insanity
# OpenSSL 1.1 support - status update and roadmap
# [[#OpenSSL|OpenSSL 1.1 support - status update and roadmap]]
# What are the main use cases for QtNetwork: HTTP(S)?
# [[#Use-cases_for_network_in_Qt|What are the main use cases for QtNetwork: HTTP(S)?]]
#* Are there new use cases for e.g. IoT (UDP / UPnP etc.)?
#* Are there new use cases for e.g. IoT (UDP / UPnP etc.)?
#* Implement a generic REST(?) service to make it simple to interface with a Qt application?
#* Implement a [[#REST_details|generic REST]] service to make it simple to interface with a Qt application?
# IoT
# [[#IoT|IoT]]
#* Identify areas of QtNetwork we should focus on (HTTP(S) / low level sockets)
#* Identify areas of QtNetwork we should focus on (HTTP(S) / low level sockets)
#* Are there areas in QtNetwork to support the IoT story?
#* Are there areas in QtNetwork to support the IoT story?
#* Is there input from sales or so?
#* Is there input from sales or so?
# Performance
# [[#Performance|Performance]]
#* try to get rid of using the bearer classes (and thread) by default, they are seldomly used (this would imply deprecating QNetworkAccessManager::networkAccessible())
#* try to get rid of using the bearer classes (and thread) by default, they are seldom used (this would imply deprecating QNetworkAccessManager::networkAccessible())
# Providing widgets for QtNetwork
# [[#Widgets_for_QNetwork|Providing widgets for QtNetwork]]
#* What to do about dependencies?
#* What to do about dependencies?
#* What should we include?
#* What should we include?
Line 324: Line 326:
#* SSL configuration widget (rich)
#* SSL configuration widget (rich)
#* What about a certificate dialog?
#* What about a certificate dialog?
# Should QtNetwork's design / architecture be modified?
# [[#Design_revolution_.3F|Should QtNetwork's design / architecture be modified?]]
#* Is it outdated in any area?
#* Is it outdated in any area?
#* Could it be made redundant by using a 3rd party something?  
#* Could it be made redundant by using a 3rd party something?  
Line 332: Line 334:
#* Identify use-cases
#* Identify use-cases


== Presentations ==
=== Presentations ===
It would be nice if those in the know could teach the rest of us:
It would be nice if those in the know could teach the rest of us:
# High-level design and structure of QtNetwork
# [[#QNAM_Architecture|High-level design and structure of QtNetwork]]
#* Rationale for the design.
#* Rationale for the design.
# New in 5.8
# New in 5.8
#* HTTP/2
#* HTTP/2
#* OAuth 1+2
#* OAuth 1+2

Latest revision as of 15:45, 16 December 2017

A meeting held on the eve of the Qt contributors' summit 2016.

Attendees:

Those with knowledge
Peter Hartman, Richard Moore, Daniel (danimo), Markus (guruz), Thiago (late arrival).
TQtC Core/Network team
Fredrik de Vibe, Jesus Fernandez, Timur Pocheptsov, Jędrzej Nowacki, Edward Welbourne (taking notes).

Initial random discussion

  • Product management "stories" should be more public.
  • Richard: widgets won't die any time soon.

Recursive event-loops are a bug.

  • Authentication tends to need it.
  • Peter made socket layer free of it; needs similar changes on the HTTP layers.
  • Getting there incrementally is tricky.

Authorization is rather poorly supported. Client certificate can only be preset, not in response to each request.

Redirects and persistence

Redirect support is problematic.

  • Can (dis|en)able; should have signal-based handling; should support HSTS.
  • Permanent HTTP redirects.
  • Lack of permanent storage of details, except cookie-jar.
  • Default no cache, due to no disk; have in-memory cacheing.
  • Breaks incomplete cert-chains.
  • Should cache intermediate certs (like FF).
  • On MS, we do similar, if win-update has it.
  • (Timur: can get MS to forget a cert.)

Need to extend disk cache (or add a separate TLS permanent store).

  • TLS session tickets.
  • QNAM helps with SSL setup (cipher suites, certs).
  • Session tickets extend that, but only in memory.
  • SSL context.

Any storage is good.

  • E.g. we can have QSettings or SQLite-based back-ends (and we need something similar to QAbstractNetworkCache as a base class).
  • Should store certs, HSTS policies, TLS session tickets.
  • Avoid depend on QtSql module (so we probably can use QSqlite's API directly).
  • (Symbol clash ?)
  • QSettings likely easiest (could be the default 'back-end').
  • Latency is a big issue (another big issue is probably multi-threading - it's probably not possible/not easy to make this store 'asynchronous' in QNAM's context).
  • Need to set sane permissions (rw-------) on disk files.
  • Secure storage may be sensible generally (core).

OwnCloud, BlackBerry had real issues with the above. Persistent store enables many things.

Redirect off-by-default; needs enabled per request.

  • Breaks ubiquitous use of redirects. (Many sites use them extensively.)
  • Need security policy to automate at least the main cases.
  • Current policy is over-strict.

Some apps, that handle redirects themselves, might break; but most should be competent to cope with transition:

  • first add QNAM option to turn on redirects as default per connection;
  • later make this option on by default in QNAM.

Can do arbitrary fancy if turned off (current default); need to make default sane for convenient uses. Site-redirects (within protocol) either happen or error: need policy-based to e.g. trust from-sites, to-sites, as easy to set up - presently has to be handled by slotting signal on redirect.

(Digression: tranfer-encoding vs content-encoding; decompression is cheap.)

Net-test server

Jed: need to reinvent. Currently can't get image of old box ! Expect we shall. But why ?

Should move to Vagrant.

  • Can CI-control; shall need some CI fixes.
  • Just need to know what's on present server.

Tried before w/ puppet, but problems.

Fairly sure vagrant sounder.

Need to make it work for end-devs; two download servers vagrant consults, internal to CI and a public one. Need lists of

  • services we need for present tests
  • some services we should be testing against
  • services we shall want

Peter's VMware image is used (Richard supplies).

For flaky tests: dev-boxes too powerful.

  • Need to stress box while running tests.
  • VSphere messes w/ clocks, too.

Vagrant issue on first run : fetch full Ubunu image.

  • (Ubuntu LTS, not debian; SSL breaks.)
  • LUbuntu (lightweight) maybe.

Can cope.

Probably need to fix many tests, too.

Should also have a win-net-test server to test against.

  • Future; license ok for manual testing; ok on vagrant.
  • Test against real MS-version of service.
  • e.g. NTLM service.

OpenSSL

  • We currently (unofficially) support versions >= 0.9.8 ok.
  • We only admit to supporting >= 1.0.2 (LTS);; and we would rather make this reality.

Features are tricky between OpenSSL versions:

  • We have compile-time checks for feature support;
  • run-time checks on library loaded, for feature available.
  • But defines are -ve; need version info to un-negate.
  • Richard has way to centralise and transform to positive tests of defines.
  • To add: runtime checks to match.

Aim to disable 0.9.8: problem – search finds wrong version on Mac; dlopen woes; can fix, but there are complications. Richard has changes for this.

To complicate the feature situation further:

  • Debian disabled TLS 1.0 without setting the right define. Can't work round this.
  • RH set defines right, but disabled SSL 3; this broke tests. (We can work round this.)
  • Discovering supported feature-set is a problem, even when we know version.
  • Need someone on deb-test to catch this before it hits stable and breaks us.

HTTP/2 needs OpenSSL, Apple's Secure Transport is not adequate.

We want OpenSSL 1.0.2 and >= 1.1

  • For 1.1, Richard has a clean slate, due to big API changes.
  • The new back-end works.
  • Needs to do an update now that 1.1 has been released (this week); fix config tests; should now be stable.
  • Wait for LTS of it before we recomend it; keep 1.0.2 as recommended version until then.

Forks of OpenSSL complicate life.

  • Some re-use OpenSSL names, complicating version-checks.
  • We should insist on a new back-end for each, just as for 1.1

QNAM Architecture

Peter's blog post covers some of this.

SockePrivate
cross-platform
SocketEngine
implements socketry; platform-native or virtualized (e.g. via proxy or SSL)

We have to create the SSL socket initially as a normal socket:

  • Some protocols (e.g. SMTP) have to talk raw over the Ssl socket before they can begin encryption.

QSslSocket could lose its internal TCP, but MS's SSL sockets aren't the same as plain sockets (? at least WinCE - but WinCE is going away) can't simply use a QTcpSocket.

WebSocket is different, buggy; needs review of open bugs; but works.

Network is in core so that DBus can use it;

  • DBus is used by accessibility, xcb plugin, printing, ... (ask Thiago).
  • Can't move until Qt6 – moving symbols between libs is bad.
  • Qt Lite can disable http, QNAM.

Performance

Connection management:

  • Bearer thread could go away.
  • Bearer code could all go ?
  • Check with Lorn Potter.

Being on "a network" isn't the same as on "the internet".

  • Should distinguish, using probe URL google provides.
  • Ask user to log in via captive network if merely so.

Bearer thread scans for wifi; is expensive. Lorn/Ubuntu is expert.

NetworkConfig vs session

Use-cases for network in Qt

  • See above, DBus.
  • Nokia wanted it.
  • New things ?
  • CAN, IoT, OTA.

REST - can we do better - QJsonDocument -> QNetworkReply ? how ? Look at XmlHttpRequest. Make REST facade to hide most of mess from client. Handle repetitive header work. Result object can be JSON, XML, various custom forms (incliding borked JSON, XML); need to return as QByteArray, convenience methods for Json, ...

Could be external app, example code.

HTTP server - several on github - review them. Embed in an app as daemon. Add-ons for particular use-cases. REST client & server. Iillustration code, simple implementations of basics.

μSGI (micro-server gateway interface). pythonic web-server plugins.

REST details

Copied from Richard Moore's Google doc

  • Provide convenience API for REST services
  • Server support
  • Client support
  • SSL and plain text
  • Separate module (or perhaps 2 one for client and one for server)

Client

  • Support for ‘pre-setting’ headers to be added to requests (Eg. for auth)
  • Preset base location of the API
  • Have a default QNAM
  • Make QML etc. use the default one normally
  • Talked to tronical and even though the QML one is in the wrong thread right now, that’s just an implementation detail and can easily be changed to use the main thread
  • Allows caching etc. to be setup once
  • Makes use of a single QNAM much easier than it is currently
  • Still allows for multiple QNAMs when the app needs different security contexts etc.
  • Need to ensure the creation is thread-safe
  • What do we do with the old one? Just delete it and document that you should set it early in the app lifecycle?
  • Beware of people who are already holding a pointer to it
  • possibly some static methods to make performing an HTTP request trivial
  • QNetworkReply *QNetworkAccessManager::get(“https://example.com”);
  • Not convinced these are a good idea, but they’re possible
  • Easy conversions to QJsonDocument, QXmlStreamReader, others?
  • Try to minimise the amount of boilerplate code needed by users

Server

  • Provide a convenience QSslServer class (in qtbase)
  • Provide a base-class for handling HTTP (in qtbase)
  • Provide more specialised handlers such as a RESTy thing in the addon module
  • Make it trivial to support both plain text and TLS with TLS being the (strongly) recommended approach

Research

  • Look at some popular REST APIs
  • Identify common elements that could be made easier by some convenience APIs

IoT

Auto tends to use Qt only for UI and server-side.

  • OEMs don't want to be tied to us, so core is in-house.
PAN
Bluetooth LE (low energy) - we support it.
Competes with ZigBee - lower level than our networking.
MDNS
maybe worth exploring.

Many devices are minimal Linux; many even more minimal - TCP stack + event loop.

  • AMQP.

Worth reviewing (and ensuring) API coherence between our bluetooth and network code, so can tunnel latter via former tidily.

Thiago reprise

Most of this write-up is chronological, but we skipped back to IoT when Thiago arrived

DTLS
datagram SSL - TLS over UDP.
CoApp, CoRE group at IETF.
More UDP support would be good.
Can ignore proxying of UDP - no-one does it.

IoT consists in reality of isolated objects talking to phone apps via cloud.

MPTCT
multi-path TCP; e.g. wifi and phone data as paths.
TCP can open a second path to switch to it.

Widgets for QNetwork

e.g. danimo has widget to display URLs, indicating security &c, anti-phish cues.

  • Cert display dialogs, immune to XSS attacks &c.
  • SSL config.
  • candidate: HTTP request progress-bar.

We presently leave these to client code;

  • all easy to get wrong;
  • all error-prone;
  • errors are all security holes.

Important add-ons, important to do in QML as well.

  • Worth doing once, well, supported.
  • So clients are more secure and our fixes benefit all.

Good candidate for add-on module.

Design revolution ?

Could we replace large parts of QNAM or even of our network stack ?

  • Need for stable APIs - lacking in some
  • libcurl – has more SSL backends than us, stable, good candidate.
  • apache portable runtime ?
  • node.js ?

We dlopen() OpenSSL for legal loop-holes, crypto export via USA;

  • would need to hack libcurl (or other replacement) to do similar - not a big problem.
  • Good stable API in libcurl.
  • Lacks server-side; we would need those.
  • Worth looking at; ditch lots of code.
  • Kill QNAM.
  • Support sftp, file, ...

Original Plan

For reference, here are the things we planned to discuss, linked to sections above in so far as we actually did so.

Proposed Discussion Topics

  1. Redirection support
    • Should it be on by default
    • more configurable...
  2. The test server
    • How can we get the image - should we replace / update it? The Vagrant solution?
    • How can we add services etc.
  3. How can we better manage the various openssl builds?
    • The variations in build options, versions etc. between the platforms and distros has got to the point of insanity
  4. OpenSSL 1.1 support - status update and roadmap
  5. What are the main use cases for QtNetwork: HTTP(S)?
    • Are there new use cases for e.g. IoT (UDP / UPnP etc.)?
    • Implement a generic REST service to make it simple to interface with a Qt application?
  6. IoT
    • Identify areas of QtNetwork we should focus on (HTTP(S) / low level sockets)
    • Are there areas in QtNetwork to support the IoT story?
    • Is there input from sales or so?
  7. Performance
    • try to get rid of using the bearer classes (and thread) by default, they are seldom used (this would imply deprecating QNetworkAccessManager::networkAccessible())
  8. Providing widgets for QtNetwork
    • What to do about dependencies?
    • What should we include?
    • URLBar widget (danimo)
    • SSL configuration widget (rich)
    • What about a certificate dialog?
  9. Should QtNetwork's design / architecture be modified?
    • Is it outdated in any area?
    • Could it be made redundant by using a 3rd party something?
  10. Providing more server support for QtNetwork
    • QSslServer
    • An HTTP server?
    • Identify use-cases

Presentations

It would be nice if those in the know could teach the rest of us:

  1. High-level design and structure of QtNetwork
    • Rationale for the design.
  2. New in 5.8
    • HTTP/2
    • OAuth 1+2