QtWebEngine/VideoAcceleration: Difference between revisions

From Qt Wiki
Jump to navigation Jump to search
m (add "Video playback in Chromium" section)
m (adding more text)
Line 3: Line 3:
== Introduction ==
== Introduction ==


One of the most complex operations performed in internet browsers are video decoding and encoding. These compute operations can be executed as part of software implementation which runs on CPU, or the can be hardware accelerated and executed by dedicated hardware chips. By using hardware acceleration, the CPU can be offloaded and it is able to run meanwhile other execution tasks. This results in better interaction performance and it is crucial for embedded devices, where CPU processing power is limited. Moreover, using optimized dedicated hardware usually results in better video performance, quality and lower power consumption compared to software one.
One of the most complex operations performed in the internet browsers are the video decoding and encoding. These compute operations can be executed as part of the software implementation which runs on the general purpose CPU, or they can be hardware accelerated and executed by the dedicated hardware. In case of regular PCs, the video hardware is integrated into a GPU (manufactured by Intel, AMD or Nvidia) and for the embedded platforms it is in most cases a part of independent IP core included in SoC.
 
The use of dedicated video hardware offloads general propose CPU from intensive decodec/codec algorithms. This results in a better interaction performance and it is crucial for embedded devices, where the CPU processing power is limited. Moreover, using a optimized dedicated hardware usually results in the better video performance, quality and the lower power consumption compared to the software one. On the other hand, hardware decoders rarely support all of possible video formats, profiles and codes algorithms. Therefore the video acceleration support is limited in that perspective.


== Video playback in Chromium ==
== Video playback in Chromium ==


* https://www.chromium.org/audio-video
* https://www.chromium.org/audio-video
* https://www.chromium.org/developers/design-documents/video
* http://www.chromium.org/developers/design-documents/video-playback-and-compositor
* https://www.chromium.org/developers/design-documents/multi-process-architecture
In general the video playback in Chromium can be made using:
* Pepper Plugins APIs, PNaCl (see pp::VideoDecoder), pepper flash plugin, WideVine plugin
* WebRTC video encoders and decoders
* HTML <viedo> tag (using <video> pipeline)
To keep things stable, secure and running with good performance chromium browser has a multi-process-architecture. The video playback is also affected by this design and it is split between separate processes. In the most cases the code that performs encoding or decoding of video executes within the renderer process. Since this process is sandboxed, the video hardware is not accessible from it. IPC is used to communicate between renderer and GPU process, that has the access to the hardware decoders.
In case of HTML <video> tag , blink::WebMediaPlayer interface is used to create
video decoding pipeline in render process. The pipeline consists of various elements like: data source, demuxing, audio decoding, video decoding, audio rendering, and video rendering.
Basic video decode class hierarchy looks like:
Two classes are currently used for software based video playback VpxVideoDecoder and FFmpegVideoDecoder. They wrapper two corresponding libraries:
* libvpx to play video formats like vp8,vp9
* ffmpeg to play other video formats for ex ogg, mp4(H.264)
Two classes media::VideoDecodeAccelerator (VDA) and media::VideoEncodeAccelerator (VEA) are main the interfaces for of all video HW acceleration in Chromium.


== Chromium video acceleration platform support ==
== Chromium video acceleration platform support ==


Official Chromium enables video acceleration for Windows, macOS, ChromeOS and Android builds.
Official Chromium enables video acceleration for Windows, macOS, and ChromeOS builds. Regular Linux builds do not have video acceleration enabled, unless patched downstream by Linux distributions.


* macOs: Video Toolbox
* Mac using Video toolbox (class VTVideoDecodeAccelerator)
* ChromeOS on x86_64: VA-API
* ChromeOS on x86_64 using VA-API (class VaapiVideoDecodeAccelerator)
* ChromeOS on arm/tegra: V4l2
* ChromeOS on arm/tegra using V4l2 (class V4L2VideoDecodeAccelerator)
* Android: platform MediaCodec  
* Android using platform MediaCodec  
* Windows: DirectX Video Acceleration (DXVA)
* Windows using DirectX Video Acceleration (class DXVAVideoDecodeAccelerator)


On Linux, Chromium Project is mostly focused on 'desktop builds' which have enough processing power for video encoding and decoding using software based mulithreaded ffmpeg. Therefore, to avoid dealing with possible large number of hardware related bugs, acceleration support on chrome is by default disabled.
On Linux, Chromium Project is mostly focused on 'desktop builds' which have enough processing power for video encoding and decoding using software based mulithreaded ffmpeg. Therefore, to avoid dealing with possible large number of hardware related bugs, acceleration support on chrome is by default disabled.
Line 31: Line 56:


== Qt WebEngine ==
== Qt WebEngine ==
Current status of video decode acceleration for Qt WebEngine 5.9 and 5.10 is as follows:
Current status of video decode acceleration for Qt WebEngine 5.9 and 5.10 is as follows:
* Windows video acceleration does '''not''' work, see reasoning in https://codereview.qt-project.org/#/c/202314/ . Bottom line, this has not worked in any Qt WebEngine release due to ANGLE not working with multiple threads.
* macOS video acceleration does "not" currently work, see reasoning in https://bugreports.qt.io/browse/QTBUG-60002 and attached patch. This used to work in versions <= Qt 5.8 (needs confirmation).
* Linux (including embedded) video acceleration does ""not"" currently work. Work-in-progress VA-API support patch can be found at https://codereview.qt-project.org/#/c/209204/


Being considered:
* Windows video acceleration does not work, see reasoning in https://codereview.qt-project.org/#/c/202314/ . Bottom line, this has not worked in any Qt WebEngine release due to ANGLE not working with multiple threads.
* Enabling VA API in Qt 5.10 (?) for Chromium 61
 
* Implementing a punch hole in scenegraph
* macOS video acceleration does not work, see reasoning in https://bugreports.qt.io/browse/QTBUG-60002 and attached patch. This used to work in versions <= Qt 5.8 (needs confirmation).
* Supporting GStreamer
 
* Linux (including embedded) video acceleration does not work. Work-in-progress VA-API support patch can be found at https://codereview.qt-project.org/#/c/209204/


== TODO ==
== Some considerations for future ==  


- Needs to be supported by GPU hardware and drivers, usually only supporting some encoding
* Enabling VA API in Qt 5.11
* Implementing a video playback using punch hole for basic use cases
* supporting GStreamer see [https://github.com/Samsung/ChromiumGStreamerBackend]()
* supporting i.MX VPU hardware video decoding [https://github.com/Freescale/chromium-imx]()
* supporting RaspberryPi hardware decoder [https://github.com/carlonluca/pot]()

Revision as of 02:23, 4 December 2017

Qt WebEngine does not currently support hardware accelerated decoding of videos.

Introduction

One of the most complex operations performed in the internet browsers are the video decoding and encoding. These compute operations can be executed as part of the software implementation which runs on the general purpose CPU, or they can be hardware accelerated and executed by the dedicated hardware. In case of regular PCs, the video hardware is integrated into a GPU (manufactured by Intel, AMD or Nvidia) and for the embedded platforms it is in most cases a part of independent IP core included in SoC.

The use of dedicated video hardware offloads general propose CPU from intensive decodec/codec algorithms. This results in a better interaction performance and it is crucial for embedded devices, where the CPU processing power is limited. Moreover, using a optimized dedicated hardware usually results in the better video performance, quality and the lower power consumption compared to the software one. On the other hand, hardware decoders rarely support all of possible video formats, profiles and codes algorithms. Therefore the video acceleration support is limited in that perspective.

Video playback in Chromium

In general the video playback in Chromium can be made using:

  • Pepper Plugins APIs, PNaCl (see pp::VideoDecoder), pepper flash plugin, WideVine plugin
  • WebRTC video encoders and decoders
  • HTML <viedo> tag (using <video> pipeline)

To keep things stable, secure and running with good performance chromium browser has a multi-process-architecture. The video playback is also affected by this design and it is split between separate processes. In the most cases the code that performs encoding or decoding of video executes within the renderer process. Since this process is sandboxed, the video hardware is not accessible from it. IPC is used to communicate between renderer and GPU process, that has the access to the hardware decoders.

In case of HTML <video> tag , blink::WebMediaPlayer interface is used to create video decoding pipeline in render process. The pipeline consists of various elements like: data source, demuxing, audio decoding, video decoding, audio rendering, and video rendering.

Basic video decode class hierarchy looks like:

Two classes are currently used for software based video playback VpxVideoDecoder and FFmpegVideoDecoder. They wrapper two corresponding libraries:

  • libvpx to play video formats like vp8,vp9
  • ffmpeg to play other video formats for ex ogg, mp4(H.264)

Two classes media::VideoDecodeAccelerator (VDA) and media::VideoEncodeAccelerator (VEA) are main the interfaces for of all video HW acceleration in Chromium.

Chromium video acceleration platform support

Official Chromium enables video acceleration for Windows, macOS, and ChromeOS builds. Regular Linux builds do not have video acceleration enabled, unless patched downstream by Linux distributions.

  • Mac using Video toolbox (class VTVideoDecodeAccelerator)
  • ChromeOS on x86_64 using VA-API (class VaapiVideoDecodeAccelerator)
  • ChromeOS on arm/tegra using V4l2 (class V4L2VideoDecodeAccelerator)
  • Android using platform MediaCodec
  • Windows using DirectX Video Acceleration (class DXVAVideoDecodeAccelerator)

On Linux, Chromium Project is mostly focused on 'desktop builds' which have enough processing power for video encoding and decoding using software based mulithreaded ffmpeg. Therefore, to avoid dealing with possible large number of hardware related bugs, acceleration support on chrome is by default disabled.

However, there is on going community effort to enable video acceleration using VA-API and some Linux distorts like 'Arch' patch chromium already downstream:

Qt WebEngine

Current status of video decode acceleration for Qt WebEngine 5.9 and 5.10 is as follows:

Some considerations for future

  • Enabling VA API in Qt 5.11
  • Implementing a video playback using punch hole for basic use cases
  • supporting GStreamer see [1]()
  • supporting i.MX VPU hardware video decoding [2]()
  • supporting RaspberryPi hardware decoder [3]()
Retrieved from "https://wiki.qt.io/index.php?title=QtWebEngine/VideoAcceleration&oldid=32332"