QtWebEngine/VideoAcceleration: Difference between revisions

From Qt Wiki
Jump to navigation Jump to search
 
(9 intermediate revisions by 2 users not shown)
Line 3: Line 3:
== Introduction ==
== Introduction ==


One of the most complex operations performed in internet browsers are video decoding and encoding. These compute operations can be executed as part of software implementation which runs on CPU, or the can be hardware accelerated and executed by dedicated hardware chips. By using hardware acceleration, the CPU can be offloaded and it is able to run meanwhile other execution tasks. This results in better interaction performance and it is crucial for embedded devices, where CPU processing power is limited. Moreover, using optimized dedicated hardware usually results in better video performance, quality and lower power consumption compared to software one.
One of the most complex operations performed in the internet browsers is the video decoding and encoding. These compute operations can be executed as part of the software implementation which runs on the general purpose CPU, or they can be hardware accelerated and executed by the dedicated hardware. In case of regular PCs, the video hardware is integrated into a GPU (manufactured by Intel, AMD or Nvidia) and for the embedded platforms it is in most cases a part of independent IP core included in SoC.  


=== TODO
The use of dedicated video hardware offloads general propose CPU from intensive decodec/codec algorithms. This results in a better interaction performance and it is crucial for embedded devices, where the CPU processing power is limited. Moreover, using an optimized dedicated hardware usually results in the better video performance, quality and the lower power consumption compared than software based decoding. However, hardware decoders rarely support all possible video formats, video profiles and codes' algorithms. Therefore, the video acceleration support is limited in that perspective.


- Needs to be supported by GPU hardware and drivers, usually only supporting some encoding
== Video playback in Chromium ==
- Link to Chromium docs on video acceleration
 
* https://www.chromium.org/audio-video
* https://www.chromium.org/developers/design-documents/video
* http://www.chromium.org/developers/design-documents/video-playback-and-compositor
* https://www.chromium.org/developers/design-documents/multi-process-architecture
 
In general the video can be played in Chromium using:
 
* Pepper Plugins APIs, PNaCl (see pp::VideoDecoder), pepper flash plugin, WideVine plugin
* WebRTC video encoders and decoders
* HTML <video> tag (by using <video> pipeline)
 
To keep things stable, secure and running with good performance chromium browser has a multi-process-architecture. The video playback is also affected by this design and it is split between separate processes. In most cases, the code that performs encoding or decoding of video executes within the renderer process. Since this process is sandboxed, the video hardware is not accessible from it. IPC is used for communication between the renderer and the GPU process that has access to the hardware decoders.
 
In case of an HTML <video> tag, blink::WebMediaPlayer interface is used to create the video decoding pipeline in the render process. The pipeline consists of various elements like: data source, demuxer, audio decoding, video decoding, audio rendering, and video rendering.
 
The basic video decode class hierarchy looks like:
 
[[File:Video Decoder Classes.png]]
 
Two classes are currently used for software based video playback, VpxVideoDecoder and FFmpegVideoDecoder. They wrap two corresponding libraries:
 
* libvpx to play video formats like vp8, vp9
* ffmpeg to play other video formats, for example: ogg, mp4(H.264)
 
Two classes, media::VideoDecodeAccelerator (VDA) and media::VideoEncodeAccelerator (VEA), are the main interfaces for of all video hardware acceleration in Chromium.


== Chromium video acceleration platform support ==
== Chromium video acceleration platform support ==


Official Chromium enables video acceleration for Windows, macOS, ChromeOS and Android builds.
Official Chromium enables video acceleration for Windows, macOS, and ChromeOS builds.  


* macOs: Video Toolbox
* Mac using Video toolbox (class VTVideoDecodeAccelerator)
* ChromeOS on x86_64: VA-API
* ChromeOS on x86_64 using VA-API (class VaapiVideoDecodeAccelerator)
* ChromeOS on arm/tegra: V4l2
* ChromeOS on arm/tegra using V4l2 (class V4L2VideoDecodeAccelerator)
* Android: platform MediaCodec  
* Android using platform MediaCodec  
* Windows: DirectX Video Acceleration (DXVA)
* Windows using DirectX Video Acceleration (class DXVAVideoDecodeAccelerator)


On Linux, Chromium Project is mostly focused on 'desktop builds' which have enough processing power for video encoding and decoding using software based mulithreaded ffmpeg. Therefore, to avoid dealing with possible large number of hardware related bugs, acceleration support on chrome is by default disabled.
On Linux, Chromium Project is mostly focused on 'desktop builds' which have enough processing power for video encoding and decoding using software based multithreaded ffmpeg. Therefore, to avoid dealing with a possible large number of hardware related bugs, acceleration support on chrome is by default disabled.


* https://bugs.chromium.org/p/chromium/issues/detail?id=463440
* https://bugs.chromium.org/p/chromium/issues/detail?id=463440
* https://bugs.chromium.org/p/chromium/issues/detail?id=137247
* https://bugs.chromium.org/p/chromium/issues/detail?id=137247


However, there is on going community effort to enable video acceleration using VA-API and some Linux distorts like 'Arch' patch chromium already downstream:
However, there is an ongoing community effort to enable video acceleration using VA-API and some Linux distros like 'Arch' patch Chromium already downstream:


* https://codereview.chromium.org/1743543004
* https://codereview.chromium.org/1743543004
Line 31: Line 56:
* https://chromium-review.googlesource.com/c/chromium/src/+/532294
* https://chromium-review.googlesource.com/c/chromium/src/+/532294


== Chromium ==
== Qt WebEngine ==
Official Chromium enables video acceleration for Windows, macOS, and ChromeOS builds.
 
Current status of video decode acceleration for Qt WebEngine 6.5 is as follows:


Regular Linux builds do not have video acceleration enabled, unless patched downstream by Linux distributions.
* Windows video acceleration should work automatically
* macOS video acceleration should work automatically
* Linux video acceleration depends on VA-API, libGBM, and GBM support in the video drivers. But should work automatically if those are all present.


== Qt WebEngine ==
== Some considerations for future ==  
Current status of video decode acceleration for Qt WebEngine 5.9 and 5.10 is as follows:
* Windows video acceleration does '''not''' work, see reasoning in https://codereview.qt-project.org/#/c/202314/ . Bottom line, this has not worked in any Qt WebEngine release due to ANGLE not working with multiple threads.
* macOS video acceleration does "not" currently work, see reasoning in https://bugreports.qt.io/browse/QTBUG-60002 and attached patch. This used to work in versions <= Qt 5.8 (needs confirmation).
* Linux (including embedded) video acceleration does ""not"" currently work. Work-in-progress VA-API support patch can be found at https://codereview.qt-project.org/#/c/209204/


Being considered:
* Supporting i.MX VPU hardware video decoding [https://github.com/Freescale/chromium-imx]
* Enabling VA API in Qt 5.10 (?) for Chromium 61
* Supporting RaspberryPi hardware decoder [https://github.com/carlonluca/pot]
* Implementing a punch hole in scenegraph
* Supporting Zero-Copy for ffmpeg, vpx [https://codereview.chromium.org/1869303004/]
* Supporting GStreamer

Latest revision as of 09:08, 8 May 2023

Qt WebEngine does not currently support hardware accelerated decoding of videos.

Introduction

One of the most complex operations performed in the internet browsers is the video decoding and encoding. These compute operations can be executed as part of the software implementation which runs on the general purpose CPU, or they can be hardware accelerated and executed by the dedicated hardware. In case of regular PCs, the video hardware is integrated into a GPU (manufactured by Intel, AMD or Nvidia) and for the embedded platforms it is in most cases a part of independent IP core included in SoC.

The use of dedicated video hardware offloads general propose CPU from intensive decodec/codec algorithms. This results in a better interaction performance and it is crucial for embedded devices, where the CPU processing power is limited. Moreover, using an optimized dedicated hardware usually results in the better video performance, quality and the lower power consumption compared than software based decoding. However, hardware decoders rarely support all possible video formats, video profiles and codes' algorithms. Therefore, the video acceleration support is limited in that perspective.

Video playback in Chromium

In general the video can be played in Chromium using:

  • Pepper Plugins APIs, PNaCl (see pp::VideoDecoder), pepper flash plugin, WideVine plugin
  • WebRTC video encoders and decoders
  • HTML <video> tag (by using <video> pipeline)

To keep things stable, secure and running with good performance chromium browser has a multi-process-architecture. The video playback is also affected by this design and it is split between separate processes. In most cases, the code that performs encoding or decoding of video executes within the renderer process. Since this process is sandboxed, the video hardware is not accessible from it. IPC is used for communication between the renderer and the GPU process that has access to the hardware decoders.

In case of an HTML <video> tag, blink::WebMediaPlayer interface is used to create the video decoding pipeline in the render process. The pipeline consists of various elements like: data source, demuxer, audio decoding, video decoding, audio rendering, and video rendering.

The basic video decode class hierarchy looks like:

Video Decoder Classes.png

Two classes are currently used for software based video playback, VpxVideoDecoder and FFmpegVideoDecoder. They wrap two corresponding libraries:

  • libvpx to play video formats like vp8, vp9
  • ffmpeg to play other video formats, for example: ogg, mp4(H.264)

Two classes, media::VideoDecodeAccelerator (VDA) and media::VideoEncodeAccelerator (VEA), are the main interfaces for of all video hardware acceleration in Chromium.

Chromium video acceleration platform support

Official Chromium enables video acceleration for Windows, macOS, and ChromeOS builds.

  • Mac using Video toolbox (class VTVideoDecodeAccelerator)
  • ChromeOS on x86_64 using VA-API (class VaapiVideoDecodeAccelerator)
  • ChromeOS on arm/tegra using V4l2 (class V4L2VideoDecodeAccelerator)
  • Android using platform MediaCodec
  • Windows using DirectX Video Acceleration (class DXVAVideoDecodeAccelerator)

On Linux, Chromium Project is mostly focused on 'desktop builds' which have enough processing power for video encoding and decoding using software based multithreaded ffmpeg. Therefore, to avoid dealing with a possible large number of hardware related bugs, acceleration support on chrome is by default disabled.

However, there is an ongoing community effort to enable video acceleration using VA-API and some Linux distros like 'Arch' patch Chromium already downstream:

Qt WebEngine

Current status of video decode acceleration for Qt WebEngine 6.5 is as follows:

  • Windows video acceleration should work automatically
  • macOS video acceleration should work automatically
  • Linux video acceleration depends on VA-API, libGBM, and GBM support in the video drivers. But should work automatically if those are all present.

Some considerations for future

  • Supporting i.MX VPU hardware video decoding [1]
  • Supporting RaspberryPi hardware decoder [2]
  • Supporting Zero-Copy for ffmpeg, vpx [3]