Qt Network Workshop 2016: Difference between revisions

From Qt Wiki
Jump to navigation Jump to search
(→‎= Thiago reprise: oops - formatting.)
(undo)
 
(24 intermediate revisions by 4 users not shown)
Line 1: Line 1:
A meeting held on the eve of the [[Qt contributors summit 2016]].
[[Category:QtCS2016]]
 
[[Category:Developing Qt::Network]]
== Proposed Discussion Topics ==
A meeting held on the eve of the [[:Category:QtCS2016|Qt contributors' summit 2016]].
 
''Eddy has notes that shall transform the agenda into a contents list.  Just need to bring tablet to work ...''
 
# Redirection support
#* Should it be on by default
#* more configurable...
# The test server
#* How can we get the image - should we replace / update it? The Vagrant solution?
#* How can we add services etc.
# How can we better manage the various openssl builds?
#* The variations in build options, versions etc. between the platforms and distros has got to the point of insanity
# OpenSSL 1.1 support - status update and roadmap
# What are the main use cases for QtNetwork: HTTP(S)?
#* Are there new use cases for e.g. IoT (UDP / UPnP etc.)?
#* Implement a generic REST(?) service to make it simple to interface with a Qt application?
# IoT
#* Identify areas of QtNetwork we should focus on (HTTP(S) / low level sockets)
#* Are there areas in QtNetwork to support the IoT story?
#* Is there input from sales or so?
# Performance
#* try to get rid of using the bearer classes (and thread) by default, they are seldomly used (this would imply deprecating QNetworkAccessManager::networkAccessible())
# Providing widgets for QtNetwork
#* What to do about dependencies?
#* What should we include?
#* URLBar widget (danimo)
#* SSL configuration widget (rich)
#* What about a certificate dialog?
# Should QtNetwork's design / architecture be modified?
#* Is it outdated in any area?
#* Could it be made redundant by using a 3rd party something?
# Providing more server support for QtNetwork
#* QSslServer
#* An HTTP server?
#* Identify use-cases
 
== Presentations ==
 
# High-level design and structure of QtNetwork
#* Rationale for the design.
#* [http://www.peter.hartmann.tk/single-post/2012/02/26/Inside-the-Qt-HTTP-stack Peter's Qt HTTP class diagram]
# New in 5.8
#* HTTP/2
#* OAuth 1+2
 
== REST Stuff ==
''Copied from [https://docs.google.com/document/d/1jw_BPDwLZBRf4blox2k165aTaG_El773veZHOI8ANWk/edit?usp=sharing Richard Moore's Google doc]''
* Provide convenience API for REST services
* Server support
* Client support
* SSL and plain text
* Separate module (or perhaps 2 one for client and one for server)
 
=== Client ===
 
* Support for ‘pre-setting’ headers to be added to requests (Eg. for auth)
* Preset base location of the API
* Have a default QNAM
:* Make QML etc. use the default one normally
:* Talked to tronical and even though the QML one is in the wrong thread right now, that’s just an implementation detail and can easily be changed to use the main thread
:* Allows caching etc. to be setup once
:* Makes use of a single QNAM much easier than it is currently
:* Still allows for multiple QNAMs when the app needs different security contexts etc.
:* Need to ensure the creation is thread-safe
:* What do we do with the old one? Just delete it and document that you should set it early in the app lifecycle?
:* Beware of people who are already holding a pointer to it
* possibly some static methods to make performing an HTTP request trivial
:* QNetworkReply *QNetworkAccessManager::get(“https://example.com”);
:* Not convinced these are a good idea, but they’re possible
* Easy conversions to QJsonDocument, QXmlStreamReader, others?
* Try to minimise the amount of boilerplate code needed by users
 
=== Server ===
* Provide a convenience QSslServer class (in qtbase)
* Provide a base-class for handling HTTP (in qtbase)
* Provide more specialised handlers such as a RESTy thing in the addon module
* Make it trivial to support both plain text and TLS with TLS being the (strongly) recommended approach
 
=== Research ===
* Look at some popular REST APIs
* Identify common elements that could be made easier by some convenience APIs
 
= Write-up =
 
In need of tidy-up and reorganisation, but let's have it visible ASAP ...


Attendees:
Attendees:
;Those with knowledge: Peter Hartman, Richard Moore, Daniel (danimo), Markus (guruz).
;Those with knowledge: Peter Hartman, Richard Moore, Daniel (danimo), Markus (guruz), Thiago (late arrival).
;TQtC Core/Network team: Fredrik de Vibe, Jesus Fernandez, Timur Pocheptsov, Edward Welbourne (taking notes).
;TQtC Core/Network team: Fredrik de Vibe, Jesus Fernandez, Timur Pocheptsov, Jędrzej Nowacki, Edward Welbourne (taking notes).


Prod mgmt "stories" should be more public.
== Initial random discussion ==


Richard: widgets won't die any time soon.
* Product management "stories" should be more public.
* Richard: widgets won't die any time soon.


Recursive event-loops are a bug.
Recursive event-loops are a bug.
Authentication tends to need it.
* Authentication tends to need it.
Peter made SSL free of it.
* Peter made socket layer free of it; needs similar changes on the HTTP layers.
Getting there incrementally is tricky.
* Getting there incrementally is tricky.
 
Auth rather poorly supported.
Client cert only preset, not in response to request.


Redirect support problematic.
Authorization is rather poorly supported.
Can enable/dis; should have signal-based handling; should support HSTS.
Client certificate can only be preset, not in response to each request.
Permanent HTTP redirects.
Lack of permanent storage of details, except cookie-jar.
Default no cache, due to no disk; have in-memory cacheing.
Breaks incomplete cert-chains.
Should cache intermediate certs (like FF).
On MS, we do similar, if win-update has it.
(Timur: can get MS to forget a cert.)


Need to extend disk cache.
=== Redirects and persistence ===
TLS session tickets.


QNAM helps with SSL setup (cipher suites, certs).
Redirect support is problematic.
Session tickets extend that, but only in memory.
* Can (dis|en)able; should have signal-based handling; should support HSTS.
SSL context.
* Permanent HTTP redirects.
* Lack of permanent storage of details, except cookie-jar.
* Default no cache, due to no disk; have in-memory cacheing.
* Breaks incomplete cert-chains.
* Should cache intermediate certs (like FF).
* On MS, we do similar, if win-update has it.
* (Timur: can get MS to forget a cert.)


Any storage good.
Need to extend disk cache (or add a separate TLS permanent store).
e.g. QSettings or SQLite.
* TLS session tickets.
Certs, HSTS policies, client tickets.
* QNAM helps with SSL setup (cipher suites, certs).
QNetworkDiskCache solved,
* Session tickets extend that, but only in memory.
Avoid depend on QtSql module.
* SSL context.
(Symbol clash ?)
QSettings likely easiest.
Latency is big issue.
Need to set sane permissions rw------- on disk files.


Secure storage may be sensible generally (core).
Any storage is good.
* E.g. we can have QSettings or SQLite-based back-ends (and we need something similar to QAbstractNetworkCache as a base class).
* Should store certs, HSTS policies, TLS session tickets.
* Avoid depend on QtSql module (so we probably can use QSqlite's API directly).
* (Symbol clash ?)
* QSettings likely easiest (could be the default 'back-end').
* Latency is a big issue (another big issue is probably multi-threading - it's probably not possible/not easy to make this store 'asynchronous' in QNAM's context).
* Need to set sane permissions (<kbd>rw-------</kbd>) on disk files.
* Secure storage may be sensible generally (core).


OwnCloud, BlackBerry had real issues with the above.
OwnCloud, BlackBerry had real issues with the above.
Line 136: Line 52:


Redirect off-by-default; needs enabled per request.
Redirect off-by-default; needs enabled per request.
Breaks ubiquitous use of redirects.
* Breaks ubiquitous use of redirects. (Many sites use them extensively.)
Need security  policy to automate at least the main cases.
* Need security  policy to automate at least the main cases.
Current policy over-strict.
* Current policy is over-strict.


Some apps, that handle redirects self, might break;
Some apps, that handle redirects themselves, might break;
but most should be competent to cope with transition:
but most should be competent to cope with transition:
* first add QNAM option to default on per connection;
* first add QNAM option to turn on redirects as default per connection;
* later make this option on by default in QNAM.
* later make this option on by default in QNAM.


Can do arbitrary fancy if torn off (current default); need to
Can do arbitrary fancy if turned off (current default); need to make default sane for convenient uses.
make default sane for convenient uses.
 
Site-redirects (within protocol) either happen or error:
Site-redirects (within protocol) either happen or error:
need policy-based to e.g. trust from-sites, to-sites,
need policy-based to e.g. trust from-sites, to-sites,
Line 153: Line 67:
signal on redirect.
signal on redirect.


(digress: tranfer-encoding vs content-encoding;
(Digression: tranfer-encoding vs content-encoding; decompression is cheap.)
decompression is cheap.)


== Net-test server ==
== Net-test server ==


Jed: need to reinvent.
Jed: need to reinvent. Currently can't get image of old box ! Expect we shall. But why ?
Currently can't get image of old box !
Expect we shall.
But why ?


Should move to Vagrant.
Should move to Vagrant.
Can CI-control; shall need some CI fixes.
* Can CI-control; shall need some CI fixes.
Just need to know what's on present server.
* Just need to know what's on present server.


Tried before w/ puppet, but problems.
Tried before w/ puppet, but problems.
Fairly sure vagrant sounder.
: Fairly sure vagrant sounder.
Need to make it work for end-devs;
Need to make it work for end-devs; two download servers vagrant consults, internal to CI and a public one.
two download servers vagrant consults, internal to CI and a public one.
Need lists of
Need lists of
* services we need for present tests
* services we need for present tests
* some services we should be testing against  
* some services we should be testing against  
* services we shall want
* services we shall want


Peter's VMware image is used (Richard supplies).
Peter's VMware image is used (Richard supplies).


For flaky tests: dev-boxes too powerful.
For flaky tests: dev-boxes too powerful.
Need to stress box while running tests.
* Need to stress box while running tests.
VSphere messes w/ clocks, too.
* VSphere messes w/ clocks, too.


Vagrant issue on first run : fetch full Ubunu image.
Vagrant issue on first run : fetch full Ubunu image.
(Ubuntu LTS, not debian; SSL breaks.)
* (Ubuntu LTS, not debian; SSL breaks.)
LUbuntu (lightweight) maybe.
* LUbuntu (lightweight) maybe.
Can cope.
Can cope.


Line 190: Line 99:


Should also have a win-net-test server to test against.
Should also have a win-net-test server to test against.
Future; license ok for manual testing; ok on vagrant.
* Future; license ok for manual testing; ok on vagrant.
Test against real MS-version of service.
* Test against real MS-version of service.
e.g. NTLM service.
* e.g. NTLM service.


== OpenSSL ==
== OpenSSL ==
versions >= 0.9.8 ok.
* We currently (unofficially) support versions >= 0.9.8 ok.
should be >= 1.0.2 (LTS)
* We only admit to supporting >= 1.0.2 (LTS);; and we would rather make this reality.


We have compile-time checks, feature support;
Features are tricky between OpenSSL versions:
and run-time checks on library loaded, feature available.
* We have compile-time checks for feature support;
But defines are -ve; need version info to un-negate.
* run-time checks on library loaded, for feature available.
Richard has way to centralise and transform to positive tests of defines.
* But defines are -ve; need version info to un-negate.
To add: runtime checks to match.
* Richard has way to centralise and transform to positive tests of defines.
* To add: runtime checks to match.


disable 0.9.8: problem - search finds wrong version on Mac; dlopen woes;
Aim to disable 0.9.8: problem &ndash; search finds wrong version on Mac; dlopen woes;
can fix, but there are complications.
can fix, but there are complications.
Richard has changes for this.
Richard has changes for this.


Debian disables TLS 1.0 without right define.  Can't work round this.
To complicate the feature situation further:
RH set defines right, but disabled SSL 3; broke tests. (Can work round.)
* Debian disabled TLS 1.0 without setting the right define.  Can't work round this.
Discovering supported feature-set is a problem, even when we know version.
* RH set defines right, but disabled SSL 3; this broke tests. (We can work round this.)
Need someone on deb-test to catch this before it hits stable and breaks us.
* Discovering supported feature-set is a problem, even when we know version.
* Need someone on deb-test to catch this before it hits stable and breaks us.


HTTP/2 needs OpenSSL, Apple's secure transport not adequate.
HTTP/2 needs OpenSSL, Apple's Secure Transport is not adequate.
Want 1.0.2 and >= 1.1


For 1.1, Richard has a clean slate, due to big API changes.
We want OpenSSL 1.0.2 and >= 1.1
It works.
* For 1.1, Richard has a clean slate, due to big API changes.
Need update now 1.1 is released; fix config tests; should now be stable.
* The new back-end works.
Wait for LTS of it before we recomend it; keep 1.0.2 as recommend until then.
* Needs to do an update now that 1.1 has been released (this week); fix config tests; should now be stable.
* Wait for LTS of it before we recomend it; keep 1.0.2 as recommended version until then.


Non-OpenSSL variants complicate life.
Forks of OpenSSL complicate life.
Some re-use OpenSSL names, complicate version-checks.
* Some re-use OpenSSL names, complicating version-checks.
* We should insist on a new back-end for each, just as for 1.1


== QNAM architecture ==
== QNAM Architecture ==


Peter's blog post - link in wiki - find in earth-team IRC (9:53) or in e-mail.
Peter's [http://www.peter.hartmann.tk/single-post/2012/02/26/Inside-the-Qt-HTTP-stack blog post] covers some of this.
Markus draws diagram, talks us through it.
* Markus drew diagrams and talked us through them, we discussed them.
(I have photo.)
* Photos of diagrams: [[Media:QtCon2016NetArch.jpg]], [[Media:QtCon2016Socket.jpg]]
Discussion.
Second picture.


SockePrivate - cross-platform
;SockePrivate: cross-platform
SocketEngine - implements socketry; plat-native or virtualized (e.g. via proxy or SSL)
;SocketEngine: implements socketry; platform-native or virtualized (e.g. via proxy or SSL)


Some protocols (eg SMTP) have to talk raw over the Ssl socket before they can begin encryption.
We have to create the SSL socket initially as a normal socket:
* Some protocols (e.g. SMTP) have to talk raw over the Ssl socket before they can begin encryption.
QSslSocket could lose its internal TCP, but
QSslSocket could lose its internal TCP, but
MS's SSL sockets aren't the same as plain sockets (? at least WinCE) can't simply use a QTcpSocket.
MS's SSL sockets aren't the same as plain sockets (? at least WinCE - but WinCE is going away) can't simply use a QTcpSocket.
 
WebSocket is different, buggy; needs review of open bugs; but works.


WebSocket different, buggy; needs review of open bugs; but works.
Network is in core so that DBus can use it;
* DBus is used by accessibility, xcb plugin, printing, ... (ask Thiago).
* Can't move until Qt6 &ndash; moving symbols between libs is bad.
* Qt Lite can disable http, QNAM.


Network is in core so DBus can use; DBus is used by accessibility, xcb plugin, printing, ... (ask Thiago).
=== Performance ===
Can't move until Qt6 - moving symbols between libs is bad.
Qt Lite can disable http, QNAM.


Bearer thread could go away.
Connection management:
Bearer code could all go ? Connection management.
* Bearer thread could go away.
Check with Lorn Potter.
* Bearer code could all go ?
* Check with Lorn Potter.


Being on "a network" isn't the same as on "the internet".
Being on "a network" isn't the same as on "the internet".
Should distinguish, using probe URL google provides.
* Should distinguish, using probe URL google provides.
Ask user to log in via captive network if merely so.
* Ask user to log in via captive network if merely so.


Bearer thread scans for wifi; is expensive.
Bearer thread scans for wifi; is expensive.
Line 259: Line 174:


== Use-cases for network in  Qt ==
== Use-cases for network in  Qt ==
 
* See above, DBus.
See above, DBus.
* Nokia wanted it.
Nokia wanted it.
* New things ?
New things ?
* CAN, IoT, OTA.
 
CAN, IoT, OTA.


REST - can we do better - QJsonDocument -> QNetworkReply ? how ?
REST - can we do better - QJsonDocument -> QNetworkReply ? how ?
Line 283: Line 196:
&mu;SGI (micro-server gateway interface).
&mu;SGI (micro-server gateway interface).
pythonic web-server plugins.
pythonic web-server plugins.
=== REST details ===
''Copied from [https://docs.google.com/document/d/1jw_BPDwLZBRf4blox2k165aTaG_El773veZHOI8ANWk/edit?usp=sharing Richard Moore's Google doc]''
* Provide convenience API for REST services
* Server support
* Client support
* SSL and plain text
* Separate module (or perhaps 2 one for client and one for server)
==== Client ====
* Support for ‘pre-setting’ headers to be added to requests (Eg. for auth)
* Preset base location of the API
* Have a default QNAM
:* Make QML etc. use the default one normally
:* Talked to tronical and even though the QML one is in the wrong thread right now, that’s just an implementation detail and can easily be changed to use the main thread
:* Allows caching etc. to be setup once
:* Makes use of a single QNAM much easier than it is currently
:* Still allows for multiple QNAMs when the app needs different security contexts etc.
:* Need to ensure the creation is thread-safe
:* What do we do with the old one? Just delete it and document that you should set it early in the app lifecycle?
:* Beware of people who are already holding a pointer to it
* possibly some static methods to make performing an HTTP request trivial
:* QNetworkReply *QNetworkAccessManager::get(“https://example.com”);
:* Not convinced these are a good idea, but they’re possible
* Easy conversions to QJsonDocument, QXmlStreamReader, others?
* Try to minimise the amount of boilerplate code needed by users
==== Server ====
* Provide a convenience QSslServer class (in qtbase)
* Provide a base-class for handling HTTP (in qtbase)
* Provide more specialised handlers such as a RESTy thing in the addon module
* Make it trivial to support both plain text and TLS with TLS being the (strongly) recommended approach
==== Research ====
* Look at some popular REST APIs
* Identify common elements that could be made easier by some convenience APIs


=== IoT ===
=== IoT ===


Auto tends to use Qt only for UI and server-side.
Auto tends to use Qt only for UI and server-side.
Don't want to be tied to us, so core is in-house.
* OEMs don't want to be tied to us, so core is in-house.


PAN: Bluetooth LE (low energy) - we support it. Competes with ZigBee - lower level than our networking.
;PAN: Bluetooth LE (low energy) - we support it.
MDNS - maybe worth exploring.
: Competes with ZigBee - lower level than our networking.
;MDNS: maybe worth exploring.


Many devices minimal Linux; many even more minimal - TCP stack + event loop.
Many devices are minimal Linux; many even more minimal - TCP stack + event loop.
AMQP.
* AMQP.


Worth making API coherence between our bluetooth and network code, so can tunnel latter via former tidily.
Worth reviewing (and ensuring) API coherence between our bluetooth and network code, so can tunnel latter via former tidily.


=== Thiago reprise ===
==== Thiago reprise ====
''Most of this write-up is chronological, but we skipped back to IoT when Thiago arrived''


DTLS - datagram SSL - UDP TLS.
;DTLS: datagram SSL - TLS over UDP.
CoApp, CoRE group at IETF.
:CoApp, CoRE group at IETF.
More UDP support would be good.
:More UDP support would be good.
Can ignore proxying of UDP - no-one does it.
:Can ignore proxying of UDP - no-one does it.


IoT all isolated objects talking to phone apps via cloud.
IoT consists in reality of isolated objects talking to phone apps via cloud.


MPTCT - multi-path TCP; e.g. wifi and  phone data as paths.
;MPTCT: multi-path TCP; e.g. wifi and  phone data as paths.
TCP can open a second path to switch to it.
:TCP can open a second path to switch to it.
 
== Performance ==
 
see bearer, abovve


==  Widgets for QNetwork ==
==  Widgets for QNetwork ==


e.g. danimo has widget to display URLs, indicating security &c, anti-phish cues.
e.g. danimo has widget to display URLs, indicating security &c, anti-phish cues.
Cert display dialogs, immune to XSS attacks &c.
* Cert display dialogs, immune to XSS attacks &c.
SSL config.
* SSL config.
* candidate: HTTP request progress-bar.
 
We presently leave these to client code;
* all easy to get wrong;
* all error-prone;
* errors are all security holes.


We leave these to client code; all easy to get wrong; all error-prone; errors are all security holes.
Important add-ons, important to do in QML as well.
Important add-ons, important to do in QML as well.
Worth doing once, well, supported.
* Worth doing once, well, supported.
So clients are more secure and our fixes benefit all.
* So clients are more secure and our fixes benefit all.
 
HTTP request progress-bar.


Good candidate for add-on module.
Good candidate for add-on module.
Line 330: Line 283:
== Design revolution ? ==
== Design revolution ? ==


Need for stable APIs - lacking in some. but
Could we replace large parts of QNAM or even of our network stack ?
* libcurl - more SSL backends than us, good candidate.
* Need for stable APIs - lacking in some
* libcurl &ndash; has more SSL backends than us, stable, good candidate.
* apache portable runtime ?
* apache portable runtime ?
* node.js ?
* node.js ?


We dlopen() OpenSSL for legal loop-holes, crypto export via USA;
We dlopen() OpenSSL for legal loop-holes, crypto export via USA;
would need to hack libcurl (or ...) to do similar - not big problem.
* would need to hack libcurl (or other replacement) to do similar - not a big problem.
Good stable API in libcurl.
* Good stable API in libcurl.
Lacks server-side; we would need those.
* Lacks server-side; we would need those.
Worth looking at; ditch lots of code.
* Worth looking at; ditch lots of code.
Kill QNAM.
* Kill QNAM.
Support sftp, file, ...
* Support sftp, file, ...
 
== Original Plan ==
For reference, here are the things we planned to discuss, linked to sections above in so far as we actually did so.


== More servers ==
=== Proposed Discussion Topics ===


See above, use-cases, IoT.
# [[#Redirects_and_persistence|Redirection support]]
#* Should it be on by default
#* more configurable...
# [[#Net-test_server|The test server]]
#* How can we get the image - should we replace / update it? The Vagrant solution?
#* How can we add services etc.
# [[#OpenSSL|How can we better manage the various openssl builds?]]
#* The variations in build options, versions etc. between the platforms and distros has got to the point of insanity
# [[#OpenSSL|OpenSSL 1.1 support - status update and roadmap]]
# [[#Use-cases_for_network_in_Qt|What are the main use cases for QtNetwork: HTTP(S)?]]
#* Are there new use cases for e.g. IoT (UDP / UPnP etc.)?
#* Implement a [[#REST_details|generic REST]] service to make it simple to interface with a Qt application?
# [[#IoT|IoT]]
#* Identify areas of QtNetwork we should focus on (HTTP(S) / low level sockets)
#* Are there areas in QtNetwork to support the IoT story?
#* Is there input from sales or so?
# [[#Performance|Performance]]
#* try to get rid of using the bearer classes (and thread) by default, they are seldom used (this would imply deprecating QNetworkAccessManager::networkAccessible())
# [[#Widgets_for_QNetwork|Providing widgets for QtNetwork]]
#* What to do about dependencies?
#* What should we include?
#* URLBar widget (danimo)
#* SSL configuration widget (rich)
#* What about a certificate dialog?
# [[#Design_revolution_.3F|Should QtNetwork's design / architecture be modified?]]
#* Is it outdated in any area?
#* Could it be made redundant by using a 3rd party something?
# Providing more server support for QtNetwork
#* QSslServer
#* An HTTP server?
#* Identify use-cases
 
=== Presentations ===
It would be nice if those in the know could teach the rest of us:
# [[#QNAM_Architecture|High-level design and structure of QtNetwork]]
#* Rationale for the design.
# New in 5.8
#* HTTP/2
#* OAuth 1+2

Latest revision as of 15:45, 16 December 2017

A meeting held on the eve of the Qt contributors' summit 2016.

Attendees:

Those with knowledge
Peter Hartman, Richard Moore, Daniel (danimo), Markus (guruz), Thiago (late arrival).
TQtC Core/Network team
Fredrik de Vibe, Jesus Fernandez, Timur Pocheptsov, Jędrzej Nowacki, Edward Welbourne (taking notes).

Initial random discussion

  • Product management "stories" should be more public.
  • Richard: widgets won't die any time soon.

Recursive event-loops are a bug.

  • Authentication tends to need it.
  • Peter made socket layer free of it; needs similar changes on the HTTP layers.
  • Getting there incrementally is tricky.

Authorization is rather poorly supported. Client certificate can only be preset, not in response to each request.

Redirects and persistence

Redirect support is problematic.

  • Can (dis|en)able; should have signal-based handling; should support HSTS.
  • Permanent HTTP redirects.
  • Lack of permanent storage of details, except cookie-jar.
  • Default no cache, due to no disk; have in-memory cacheing.
  • Breaks incomplete cert-chains.
  • Should cache intermediate certs (like FF).
  • On MS, we do similar, if win-update has it.
  • (Timur: can get MS to forget a cert.)

Need to extend disk cache (or add a separate TLS permanent store).

  • TLS session tickets.
  • QNAM helps with SSL setup (cipher suites, certs).
  • Session tickets extend that, but only in memory.
  • SSL context.

Any storage is good.

  • E.g. we can have QSettings or SQLite-based back-ends (and we need something similar to QAbstractNetworkCache as a base class).
  • Should store certs, HSTS policies, TLS session tickets.
  • Avoid depend on QtSql module (so we probably can use QSqlite's API directly).
  • (Symbol clash ?)
  • QSettings likely easiest (could be the default 'back-end').
  • Latency is a big issue (another big issue is probably multi-threading - it's probably not possible/not easy to make this store 'asynchronous' in QNAM's context).
  • Need to set sane permissions (rw-------) on disk files.
  • Secure storage may be sensible generally (core).

OwnCloud, BlackBerry had real issues with the above. Persistent store enables many things.

Redirect off-by-default; needs enabled per request.

  • Breaks ubiquitous use of redirects. (Many sites use them extensively.)
  • Need security policy to automate at least the main cases.
  • Current policy is over-strict.

Some apps, that handle redirects themselves, might break; but most should be competent to cope with transition:

  • first add QNAM option to turn on redirects as default per connection;
  • later make this option on by default in QNAM.

Can do arbitrary fancy if turned off (current default); need to make default sane for convenient uses. Site-redirects (within protocol) either happen or error: need policy-based to e.g. trust from-sites, to-sites, as easy to set up - presently has to be handled by slotting signal on redirect.

(Digression: tranfer-encoding vs content-encoding; decompression is cheap.)

Net-test server

Jed: need to reinvent. Currently can't get image of old box ! Expect we shall. But why ?

Should move to Vagrant.

  • Can CI-control; shall need some CI fixes.
  • Just need to know what's on present server.

Tried before w/ puppet, but problems.

Fairly sure vagrant sounder.

Need to make it work for end-devs; two download servers vagrant consults, internal to CI and a public one. Need lists of

  • services we need for present tests
  • some services we should be testing against
  • services we shall want

Peter's VMware image is used (Richard supplies).

For flaky tests: dev-boxes too powerful.

  • Need to stress box while running tests.
  • VSphere messes w/ clocks, too.

Vagrant issue on first run : fetch full Ubunu image.

  • (Ubuntu LTS, not debian; SSL breaks.)
  • LUbuntu (lightweight) maybe.

Can cope.

Probably need to fix many tests, too.

Should also have a win-net-test server to test against.

  • Future; license ok for manual testing; ok on vagrant.
  • Test against real MS-version of service.
  • e.g. NTLM service.

OpenSSL

  • We currently (unofficially) support versions >= 0.9.8 ok.
  • We only admit to supporting >= 1.0.2 (LTS);; and we would rather make this reality.

Features are tricky between OpenSSL versions:

  • We have compile-time checks for feature support;
  • run-time checks on library loaded, for feature available.
  • But defines are -ve; need version info to un-negate.
  • Richard has way to centralise and transform to positive tests of defines.
  • To add: runtime checks to match.

Aim to disable 0.9.8: problem – search finds wrong version on Mac; dlopen woes; can fix, but there are complications. Richard has changes for this.

To complicate the feature situation further:

  • Debian disabled TLS 1.0 without setting the right define. Can't work round this.
  • RH set defines right, but disabled SSL 3; this broke tests. (We can work round this.)
  • Discovering supported feature-set is a problem, even when we know version.
  • Need someone on deb-test to catch this before it hits stable and breaks us.

HTTP/2 needs OpenSSL, Apple's Secure Transport is not adequate.

We want OpenSSL 1.0.2 and >= 1.1

  • For 1.1, Richard has a clean slate, due to big API changes.
  • The new back-end works.
  • Needs to do an update now that 1.1 has been released (this week); fix config tests; should now be stable.
  • Wait for LTS of it before we recomend it; keep 1.0.2 as recommended version until then.

Forks of OpenSSL complicate life.

  • Some re-use OpenSSL names, complicating version-checks.
  • We should insist on a new back-end for each, just as for 1.1

QNAM Architecture

Peter's blog post covers some of this.

SockePrivate
cross-platform
SocketEngine
implements socketry; platform-native or virtualized (e.g. via proxy or SSL)

We have to create the SSL socket initially as a normal socket:

  • Some protocols (e.g. SMTP) have to talk raw over the Ssl socket before they can begin encryption.

QSslSocket could lose its internal TCP, but MS's SSL sockets aren't the same as plain sockets (? at least WinCE - but WinCE is going away) can't simply use a QTcpSocket.

WebSocket is different, buggy; needs review of open bugs; but works.

Network is in core so that DBus can use it;

  • DBus is used by accessibility, xcb plugin, printing, ... (ask Thiago).
  • Can't move until Qt6 – moving symbols between libs is bad.
  • Qt Lite can disable http, QNAM.

Performance

Connection management:

  • Bearer thread could go away.
  • Bearer code could all go ?
  • Check with Lorn Potter.

Being on "a network" isn't the same as on "the internet".

  • Should distinguish, using probe URL google provides.
  • Ask user to log in via captive network if merely so.

Bearer thread scans for wifi; is expensive. Lorn/Ubuntu is expert.

NetworkConfig vs session

Use-cases for network in Qt

  • See above, DBus.
  • Nokia wanted it.
  • New things ?
  • CAN, IoT, OTA.

REST - can we do better - QJsonDocument -> QNetworkReply ? how ? Look at XmlHttpRequest. Make REST facade to hide most of mess from client. Handle repetitive header work. Result object can be JSON, XML, various custom forms (incliding borked JSON, XML); need to return as QByteArray, convenience methods for Json, ...

Could be external app, example code.

HTTP server - several on github - review them. Embed in an app as daemon. Add-ons for particular use-cases. REST client & server. Iillustration code, simple implementations of basics.

μSGI (micro-server gateway interface). pythonic web-server plugins.

REST details

Copied from Richard Moore's Google doc

  • Provide convenience API for REST services
  • Server support
  • Client support
  • SSL and plain text
  • Separate module (or perhaps 2 one for client and one for server)

Client

  • Support for ‘pre-setting’ headers to be added to requests (Eg. for auth)
  • Preset base location of the API
  • Have a default QNAM
  • Make QML etc. use the default one normally
  • Talked to tronical and even though the QML one is in the wrong thread right now, that’s just an implementation detail and can easily be changed to use the main thread
  • Allows caching etc. to be setup once
  • Makes use of a single QNAM much easier than it is currently
  • Still allows for multiple QNAMs when the app needs different security contexts etc.
  • Need to ensure the creation is thread-safe
  • What do we do with the old one? Just delete it and document that you should set it early in the app lifecycle?
  • Beware of people who are already holding a pointer to it
  • possibly some static methods to make performing an HTTP request trivial
  • QNetworkReply *QNetworkAccessManager::get(“https://example.com”);
  • Not convinced these are a good idea, but they’re possible
  • Easy conversions to QJsonDocument, QXmlStreamReader, others?
  • Try to minimise the amount of boilerplate code needed by users

Server

  • Provide a convenience QSslServer class (in qtbase)
  • Provide a base-class for handling HTTP (in qtbase)
  • Provide more specialised handlers such as a RESTy thing in the addon module
  • Make it trivial to support both plain text and TLS with TLS being the (strongly) recommended approach

Research

  • Look at some popular REST APIs
  • Identify common elements that could be made easier by some convenience APIs

IoT

Auto tends to use Qt only for UI and server-side.

  • OEMs don't want to be tied to us, so core is in-house.
PAN
Bluetooth LE (low energy) - we support it.
Competes with ZigBee - lower level than our networking.
MDNS
maybe worth exploring.

Many devices are minimal Linux; many even more minimal - TCP stack + event loop.

  • AMQP.

Worth reviewing (and ensuring) API coherence between our bluetooth and network code, so can tunnel latter via former tidily.

Thiago reprise

Most of this write-up is chronological, but we skipped back to IoT when Thiago arrived

DTLS
datagram SSL - TLS over UDP.
CoApp, CoRE group at IETF.
More UDP support would be good.
Can ignore proxying of UDP - no-one does it.

IoT consists in reality of isolated objects talking to phone apps via cloud.

MPTCT
multi-path TCP; e.g. wifi and phone data as paths.
TCP can open a second path to switch to it.

Widgets for QNetwork

e.g. danimo has widget to display URLs, indicating security &c, anti-phish cues.

  • Cert display dialogs, immune to XSS attacks &c.
  • SSL config.
  • candidate: HTTP request progress-bar.

We presently leave these to client code;

  • all easy to get wrong;
  • all error-prone;
  • errors are all security holes.

Important add-ons, important to do in QML as well.

  • Worth doing once, well, supported.
  • So clients are more secure and our fixes benefit all.

Good candidate for add-on module.

Design revolution ?

Could we replace large parts of QNAM or even of our network stack ?

  • Need for stable APIs - lacking in some
  • libcurl – has more SSL backends than us, stable, good candidate.
  • apache portable runtime ?
  • node.js ?

We dlopen() OpenSSL for legal loop-holes, crypto export via USA;

  • would need to hack libcurl (or other replacement) to do similar - not a big problem.
  • Good stable API in libcurl.
  • Lacks server-side; we would need those.
  • Worth looking at; ditch lots of code.
  • Kill QNAM.
  • Support sftp, file, ...

Original Plan

For reference, here are the things we planned to discuss, linked to sections above in so far as we actually did so.

Proposed Discussion Topics

  1. Redirection support
    • Should it be on by default
    • more configurable...
  2. The test server
    • How can we get the image - should we replace / update it? The Vagrant solution?
    • How can we add services etc.
  3. How can we better manage the various openssl builds?
    • The variations in build options, versions etc. between the platforms and distros has got to the point of insanity
  4. OpenSSL 1.1 support - status update and roadmap
  5. What are the main use cases for QtNetwork: HTTP(S)?
    • Are there new use cases for e.g. IoT (UDP / UPnP etc.)?
    • Implement a generic REST service to make it simple to interface with a Qt application?
  6. IoT
    • Identify areas of QtNetwork we should focus on (HTTP(S) / low level sockets)
    • Are there areas in QtNetwork to support the IoT story?
    • Is there input from sales or so?
  7. Performance
    • try to get rid of using the bearer classes (and thread) by default, they are seldom used (this would imply deprecating QNetworkAccessManager::networkAccessible())
  8. Providing widgets for QtNetwork
    • What to do about dependencies?
    • What should we include?
    • URLBar widget (danimo)
    • SSL configuration widget (rich)
    • What about a certificate dialog?
  9. Should QtNetwork's design / architecture be modified?
    • Is it outdated in any area?
    • Could it be made redundant by using a 3rd party something?
  10. Providing more server support for QtNetwork
    • QSslServer
    • An HTTP server?
    • Identify use-cases

Presentations

It would be nice if those in the know could teach the rest of us:

  1. High-level design and structure of QtNetwork
    • Rationale for the design.
  2. New in 5.8
    • HTTP/2
    • OAuth 1+2