QtWebEngine/VideoAcceleration: Difference between revisions
m (add image) |
(catching all spelling mistakes) |
||
| Line 3: | Line 3: | ||
== Introduction == | == Introduction == | ||
One of the most complex operations performed in the internet browsers | One of the most complex operations performed in the internet browsers is the video decoding and encoding. These compute operations can be executed as part of the software implementation which runs on the general purpose CPU, or they can be hardware accelerated and executed by the dedicated hardware. In case of regular PCs, the video hardware is integrated into a GPU (manufactured by Intel, AMD or Nvidia) and for the embedded platforms it is in most cases a part of independent IP core included in SoC. | ||
The use of dedicated video hardware offloads general propose CPU from intensive decodec/codec algorithms. This results in a better interaction performance and it is crucial for embedded devices, where the CPU processing power is limited. Moreover, using | The use of dedicated video hardware offloads general propose CPU from intensive decodec/codec algorithms. This results in a better interaction performance and it is crucial for embedded devices, where the CPU processing power is limited. Moreover, using an optimized dedicated hardware usually results in the better video performance, quality and the lower power consumption compared than software based decoding. However, hardware decoders rarely support all possible video formats, video profiles and codes' algorithms. Therefore, the video acceleration support is limited in that perspective. | ||
== Video playback in Chromium == | == Video playback in Chromium == | ||
| Line 14: | Line 14: | ||
* https://www.chromium.org/developers/design-documents/multi-process-architecture | * https://www.chromium.org/developers/design-documents/multi-process-architecture | ||
In general the video | In general the video can be played in Chromium using: | ||
* Pepper Plugins APIs, PNaCl (see pp::VideoDecoder), pepper flash plugin, WideVine plugin | * Pepper Plugins APIs, PNaCl (see pp::VideoDecoder), pepper flash plugin, WideVine plugin | ||
* WebRTC video encoders and decoders | * WebRTC video encoders and decoders | ||
* HTML < | * HTML <video> tag (by using <video> pipeline) | ||
To keep things stable, secure and running with good performance chromium browser has a multi-process-architecture. The video playback is also affected by this design and it is split between separate processes. In | To keep things stable, secure and running with good performance chromium browser has a multi-process-architecture. The video playback is also affected by this design and it is split between separate processes. In most cases, the code that performs encoding or decoding of video executes within the renderer process. Since this process is sandboxed, the video hardware is not accessible from it. IPC is used for communication between the renderer and the GPU process that has access to the hardware decoders. | ||
In case of HTML <video> tag , blink::WebMediaPlayer interface is used to create | In case of an HTML <video> tag, blink::WebMediaPlayer interface is used to create the video decoding pipeline in the render process. The pipeline consists of various elements like: data source, demuxer, audio decoding, video decoding, audio rendering, and video rendering. | ||
video decoding pipeline in render process. The pipeline consists of various elements like: data source, | |||
The basic video decode class hierarchy looks like: | |||
[[File:Video Decoder Classes.png]] | [[File:Video Decoder Classes.png]] | ||
Two classes are currently used for software based video playback VpxVideoDecoder and FFmpegVideoDecoder. They | Two classes are currently used for software based video playback, VpxVideoDecoder and FFmpegVideoDecoder. They wrap two corresponding libraries: | ||
* libvpx to play video formats like vp8,vp9 | * libvpx to play video formats like vp8, vp9 | ||
* ffmpeg to play other video formats for | * ffmpeg to play other video formats, for example: ogg, mp4(H.264) | ||
Two classes media::VideoDecodeAccelerator (VDA) and media::VideoEncodeAccelerator (VEA) are main | Two classes, media::VideoDecodeAccelerator (VDA) and media::VideoEncodeAccelerator (VEA), are the main interfaces for of all video hardware acceleration in Chromium. | ||
== Chromium video acceleration platform support == | == Chromium video acceleration platform support == | ||
| Line 46: | Line 45: | ||
* Windows using DirectX Video Acceleration (class DXVAVideoDecodeAccelerator) | * Windows using DirectX Video Acceleration (class DXVAVideoDecodeAccelerator) | ||
On Linux, Chromium Project is mostly focused on 'desktop builds' which have enough processing power for video encoding and decoding using software based | On Linux, Chromium Project is mostly focused on 'desktop builds' which have enough processing power for video encoding and decoding using software based multithreaded ffmpeg. Therefore, to avoid dealing with a possible large number of hardware related bugs, acceleration support on chrome is by default disabled. | ||
* https://bugs.chromium.org/p/chromium/issues/detail?id=463440 | * https://bugs.chromium.org/p/chromium/issues/detail?id=463440 | ||
* https://bugs.chromium.org/p/chromium/issues/detail?id=137247 | * https://bugs.chromium.org/p/chromium/issues/detail?id=137247 | ||
However, there is | However, there is an ongoing community effort to enable video acceleration using VA-API and some Linux distros like 'Arch' patch Chromium already downstream: | ||
* https://codereview.chromium.org/1743543004 | * https://codereview.chromium.org/1743543004 | ||
| Line 71: | Line 70: | ||
* Enabling VA API in Qt 5.11 | * Enabling VA API in Qt 5.11 | ||
* Implementing a video playback using punch hole for basic use cases | * Implementing a video playback using punch hole for basic use cases | ||
* | * Supporting GStreamer see [https://github.com/Samsung/ChromiumGStreamerBackend]() | ||
* | * Supporting i.MX VPU hardware video decoding [https://github.com/Freescale/chromium-imx] | ||
* | * Supporting RaspberryPi hardware decoder [https://github.com/carlonluca/pot] | ||
Revision as of 12:26, 4 December 2017
Qt WebEngine does not currently support hardware accelerated decoding of videos.
Introduction
One of the most complex operations performed in the internet browsers is the video decoding and encoding. These compute operations can be executed as part of the software implementation which runs on the general purpose CPU, or they can be hardware accelerated and executed by the dedicated hardware. In case of regular PCs, the video hardware is integrated into a GPU (manufactured by Intel, AMD or Nvidia) and for the embedded platforms it is in most cases a part of independent IP core included in SoC.
The use of dedicated video hardware offloads general propose CPU from intensive decodec/codec algorithms. This results in a better interaction performance and it is crucial for embedded devices, where the CPU processing power is limited. Moreover, using an optimized dedicated hardware usually results in the better video performance, quality and the lower power consumption compared than software based decoding. However, hardware decoders rarely support all possible video formats, video profiles and codes' algorithms. Therefore, the video acceleration support is limited in that perspective.
Video playback in Chromium
- https://www.chromium.org/audio-video
- https://www.chromium.org/developers/design-documents/video
- http://www.chromium.org/developers/design-documents/video-playback-and-compositor
- https://www.chromium.org/developers/design-documents/multi-process-architecture
In general the video can be played in Chromium using:
- Pepper Plugins APIs, PNaCl (see pp::VideoDecoder), pepper flash plugin, WideVine plugin
- WebRTC video encoders and decoders
- HTML <video> tag (by using <video> pipeline)
To keep things stable, secure and running with good performance chromium browser has a multi-process-architecture. The video playback is also affected by this design and it is split between separate processes. In most cases, the code that performs encoding or decoding of video executes within the renderer process. Since this process is sandboxed, the video hardware is not accessible from it. IPC is used for communication between the renderer and the GPU process that has access to the hardware decoders.
In case of an HTML <video> tag, blink::WebMediaPlayer interface is used to create the video decoding pipeline in the render process. The pipeline consists of various elements like: data source, demuxer, audio decoding, video decoding, audio rendering, and video rendering.
The basic video decode class hierarchy looks like:
Two classes are currently used for software based video playback, VpxVideoDecoder and FFmpegVideoDecoder. They wrap two corresponding libraries:
- libvpx to play video formats like vp8, vp9
- ffmpeg to play other video formats, for example: ogg, mp4(H.264)
Two classes, media::VideoDecodeAccelerator (VDA) and media::VideoEncodeAccelerator (VEA), are the main interfaces for of all video hardware acceleration in Chromium.
Chromium video acceleration platform support
Official Chromium enables video acceleration for Windows, macOS, and ChromeOS builds. Regular Linux builds do not have video acceleration enabled, unless patched downstream by Linux distributions.
- Mac using Video toolbox (class VTVideoDecodeAccelerator)
- ChromeOS on x86_64 using VA-API (class VaapiVideoDecodeAccelerator)
- ChromeOS on arm/tegra using V4l2 (class V4L2VideoDecodeAccelerator)
- Android using platform MediaCodec
- Windows using DirectX Video Acceleration (class DXVAVideoDecodeAccelerator)
On Linux, Chromium Project is mostly focused on 'desktop builds' which have enough processing power for video encoding and decoding using software based multithreaded ffmpeg. Therefore, to avoid dealing with a possible large number of hardware related bugs, acceleration support on chrome is by default disabled.
- https://bugs.chromium.org/p/chromium/issues/detail?id=463440
- https://bugs.chromium.org/p/chromium/issues/detail?id=137247
However, there is an ongoing community effort to enable video acceleration using VA-API and some Linux distros like 'Arch' patch Chromium already downstream:
- https://codereview.chromium.org/1743543004
- https://codereview.chromium.org/1872393002
- https://chromium-review.googlesource.com/c/chromium/src/+/532294
Qt WebEngine
Current status of video decode acceleration for Qt WebEngine 5.9 and 5.10 is as follows:
- Windows video acceleration does not work, see reasoning in https://codereview.qt-project.org/#/c/202314/ . Bottom line, this has not worked in any Qt WebEngine release due to ANGLE not working with multiple threads.
- macOS video acceleration does not work, see reasoning in https://bugreports.qt.io/browse/QTBUG-60002 and attached patch. This used to work in versions <= Qt 5.8 (needs confirmation).
- Linux (including embedded) video acceleration does not work. Work-in-progress VA-API support patch can be found at https://codereview.qt-project.org/#/c/209204/