QtWebEngine/VideoAcceleration
Qt WebEngine does not currently support hardware accelerated decoding of videos.
Introduction
One of the most complex operations performed in the internet browsers are the video decoding and encoding. These compute operations can be executed as part of the software implementation which runs on the general purpose CPU, or they can be hardware accelerated and executed by the dedicated hardware. In case of regular PCs, the video hardware is integrated into a GPU (manufactured by Intel, AMD or Nvidia) and for the embedded platforms it is in most cases a part of independent IP core included in SoC.
The use of dedicated video hardware offloads general propose CPU from intensive decodec/codec algorithms. This results in a better interaction performance and it is crucial for embedded devices, where the CPU processing power is limited. Moreover, using a optimized dedicated hardware usually results in the better video performance, quality and the lower power consumption compared to the software one. On the other hand, hardware decoders rarely support all of possible video formats, profiles and codes algorithms. Therefore the video acceleration support is limited in that perspective.
Video playback in Chromium
- https://www.chromium.org/audio-video
- https://www.chromium.org/developers/design-documents/video
- http://www.chromium.org/developers/design-documents/video-playback-and-compositor
- https://www.chromium.org/developers/design-documents/multi-process-architecture
In general the video playback in Chromium can be made using:
- Pepper Plugins APIs, PNaCl (see pp::VideoDecoder), pepper flash plugin, WideVine plugin
- WebRTC video encoders and decoders
- HTML <viedo> tag (using <video> pipeline)
To keep things stable, secure and running with good performance chromium browser has a multi-process-architecture. The video playback is also affected by this design and it is split between separate processes. In the most cases the code that performs encoding or decoding of video executes within the renderer process. Since this process is sandboxed, the video hardware is not accessible from it. IPC is used to communicate between renderer and GPU process, that has the access to the hardware decoders.
In case of HTML <video> tag , blink::WebMediaPlayer interface is used to create video decoding pipeline in render process. The pipeline consists of various elements like: data source, demuxing, audio decoding, video decoding, audio rendering, and video rendering.
Basic video decode class hierarchy looks like:
Two classes are currently used for software based video playback VpxVideoDecoder and FFmpegVideoDecoder. They wrapper two corresponding libraries:
- libvpx to play video formats like vp8,vp9
- ffmpeg to play other video formats for ex ogg, mp4(H.264)
Two classes media::VideoDecodeAccelerator (VDA) and media::VideoEncodeAccelerator (VEA) are main the interfaces for of all video HW acceleration in Chromium.
Chromium video acceleration platform support
Official Chromium enables video acceleration for Windows, macOS, and ChromeOS builds. Regular Linux builds do not have video acceleration enabled, unless patched downstream by Linux distributions.
- Mac using Video toolbox (class VTVideoDecodeAccelerator)
- ChromeOS on x86_64 using VA-API (class VaapiVideoDecodeAccelerator)
- ChromeOS on arm/tegra using V4l2 (class V4L2VideoDecodeAccelerator)
- Android using platform MediaCodec
- Windows using DirectX Video Acceleration (class DXVAVideoDecodeAccelerator)
On Linux, Chromium Project is mostly focused on 'desktop builds' which have enough processing power for video encoding and decoding using software based mulithreaded ffmpeg. Therefore, to avoid dealing with possible large number of hardware related bugs, acceleration support on chrome is by default disabled.
- https://bugs.chromium.org/p/chromium/issues/detail?id=463440
- https://bugs.chromium.org/p/chromium/issues/detail?id=137247
However, there is on going community effort to enable video acceleration using VA-API and some Linux distorts like 'Arch' patch chromium already downstream:
- https://codereview.chromium.org/1743543004
- https://codereview.chromium.org/1872393002
- https://chromium-review.googlesource.com/c/chromium/src/+/532294
Qt WebEngine
Current status of video decode acceleration for Qt WebEngine 5.9 and 5.10 is as follows:
- Windows video acceleration does not work, see reasoning in https://codereview.qt-project.org/#/c/202314/ . Bottom line, this has not worked in any Qt WebEngine release due to ANGLE not working with multiple threads.
- macOS video acceleration does not work, see reasoning in https://bugreports.qt.io/browse/QTBUG-60002 and attached patch. This used to work in versions <= Qt 5.8 (needs confirmation).
- Linux (including embedded) video acceleration does not work. Work-in-progress VA-API support patch can be found at https://codereview.qt-project.org/#/c/209204/