QtCS2017 Discuss Qt Logging enhancements

From Qt Wiki
Revision as of 16:30, 17 October 2017 by Arnaud Clere (talk | contribs) (added link to ML discussion)
Jump to navigation Jump to search

Discussion

  • Structured logging:
    • We want to extract information from logs and detect common messages. For example:
From To
   Got message "Hello"     
   Got message "%s"       Hello 
   Got message "World"     
   Got message "%s"       World
    • We want to allow logging to store ancillary data, possibly in machine-readable format, likely not part of the message itself
      • Like journald API can do
      • How do we allow different formatting depending on backend? Needs to support formatting user types too.
  • Structured output
    • Store to databases, for example
    • Do we still obey QT_MESSAGE_PATTERN?
    • Select backend with environment variable

Need to review with LTTNG and ETW tracing

ML discussion: http://lists.qt-project.org/pipermail/development/2017-October/031047.html

Summary

Qt Logging lacks regarding 2 different use cases:

1/ High-performance code (say HW handling, big data processing) needs a way to inject a few structured tracepoints to generate large binary traces (say, for profiling)

2/ Existing tracepoints (think about nb of qDebug in Qt user code) should generate more structured traces to:

  • facilitate exploration and diagnostic, and
  • allow immediate use of analysis tools like Python

Service and Device developers need #1. Medical Devices would benefit from #2 for post-market surveillance. Device designers operating in an uncontrolled environment would probably benefit from #2 as well

Prototypes

  • LTTNG and ETW tracing uses external tracepoint data description to generate code for LTTNG and ETW backends (or no code at all)
  • Qt-based modmedLog library (presentation, forge) uses Bind<T> and IData implementations to bind tracepoint data to JSON/XML/Console outputs and possibly read them back for replay

Conclusion

We should sit down and see how these 2 use cases intersect and the best things that are provided by each prototype

Requirements

For #1:

  • as little generated code as possible at the tracepoint:
    • no binary or numeric formatting
    • completely removable from release code (not just bypassing like QNoDebug::operator<<)

For #2:

  • source compatibility with qDebug tracepoints
    • do not require non trivial changes such as changing from qDebug()<< to qDebug(...)
    • allow changing qDebug() to qTrace() or whatever to provide choice of old vs structured tracing
    • extensible to any user data type
  • offer structured trace formats (TSV+JSON, XML, sqlite?) to avoid low-level parsing

Open Questions

  • Can we generate efficient tracepoint code without an external tracepoint description and tool?
  • Should we provide a format+args handler delegating by default to the existing handler?

(*QtMessageHandler2)(...,const QString & format, const QStringList & args);

  • Should we provide a formatting-independant handler based on Bind<T>:

(*QtMessageHandler3)(...,const QString & format, const QList<BindableRef> & args);

  • What about modmedLog performance?
  • When variadic macros and templates are available, can they be used to get the best of both prototypes?