Openmax-decoding-of-images: Difference between revisions

From Qt Wiki
Jump to navigation Jump to search
No edit summary
No edit summary
Line 5: Line 5:
== Description ==
== Description ==


Add OpenMAX IL as a backend for image decoding. Use hardware decoder to directly decode into graphics-memory (zero-copy-decoding).<br />Additional we need to keep [[Openmax backend for mm kit]] in mind, as this is the next logical step in improving performance/features.
Add OpenMAX IL as a backend for image decoding. Use hardware decoder to directly decode into graphics-memory (zero-copy-decoding).
Additional we need to keep [[Openmax backend for mm kit]] in mind, as this is the next logical step in improving performance/features.


== Sub tasks ==
== Sub tasks ==
Line 11: Line 12:
# Investigate OpenMAX IL API and concept
# Investigate OpenMAX IL API and concept
# Implement a prove of concept in application space
# Implement a prove of concept in application space
# First, small benchmark application to compare implementation with &quot;normal&amp;quot; decoding
# First, small benchmark application to compare implementation with "normal" decoding
# Integrate implementation into Qt image-loading code
# Integrate implementation into Qt image-loading code
# Improve benchmark application; Run sophisticated tests
# Improve benchmark application; Run sophisticated tests
Line 19: Line 20:
== Optional sub tasks ==
== Optional sub tasks ==


This sub tasks might or might not be covered, based on the findings and decision happening.<br />* Implement Qt-OpenMAX_IL-wrapper<br />* build/port OpenMAX AL for the rpi<br />* build/port gst-openmax<br />This sub tasks are in relation with [[Openmax backend for mm kit]]
This sub tasks might or might not be covered, based on the findings and decision happening.
* Implement Qt-OpenMAX_IL-wrapper
* build/port OpenMAX AL for the rpi
* build/port gst-openmax
This sub tasks are in relation with [[Openmax backend for mm kit]]


== Logbook ==
== Logbook ==
Line 39: Line 44:
** OpenMAX IL is a async c api, with callback, state-control, ect. (as suspected)
** OpenMAX IL is a async c api, with callback, state-control, ect. (as suspected)


* The conclusion is that there are 3 options to continue:<br /># implement a image-decoder-plugin which directly uses openmax il, don't do much abstraction, just use it.<br /># write a proper Qt/c++ API to access basic OMX IL interfaces (components, tunnels, callbacks, ect.). … and based on that, write a image decoder plugin.<br /># As there is already a &quot;convenience&amp;quot; API &quot;for&amp;quot; OpenMAX IL: Try to get OpenMAX AL running.<br />Option 3 seams to be the most effective and cleanest approach, it's preferable but depends on how easy it is to build/intergrate or of rpi will eventually support it out-of-the-box
* The conclusion is that there are 3 options to continue:
# implement a image-decoder-plugin which directly uses openmax il, don't do much abstraction, just use it.
# write a proper Qt/c++ API to access basic OMX IL interfaces (components, tunnels, callbacks, ect.). … and based on that, write a image decoder plugin.
# As there is already a "convenience" API "for" OpenMAX IL: Try to get OpenMAX AL running.
Option 3 seams to be the most effective and cleanest approach, it's preferable but depends on how easy it is to build/intergrate or of rpi will eventually support it out-of-the-box


=== 2. March - Thomas Senyk ===
=== 2. March - Thomas Senyk ===
Line 48: Line 57:
=== 13. March - Thomas Senyk ===
=== 13. March - Thomas Senyk ===


* Finished implementation. It's &quot;running&amp;quot;, but it's only producing errors so far.
* Finished implementation. It's "running", but it's only producing errors so far.
* The problem now is to understand the behavior, expectation and communication with OpenMAX IL.
* The problem now is to understand the behavior, expectation and communication with OpenMAX IL.
* It could take a while to figurer everything out and get something to the screen
* It could take a while to figurer everything out and get something to the screen
Line 58: Line 67:
* reimplemented the thread-sync code
* reimplemented the thread-sync code
* got to a state where a got a few specific errors which need so be solved
* got to a state where a got a few specific errors which need so be solved
* THE main error I got is that I can't set: OMX_SetParameter(''', OMX_IndexParamPortDefinition, &amp;portSettings) on any of my 2 components (decoder and renderer) … which sucks because I eventually need to set the EGL_Display<br />''' Next thing to do: a lot of cleaning, reading/removing my left-over 'TODO's
* THE main error I got is that I can't set: OMX_SetParameter(''', OMX_IndexParamPortDefinition, &amp;portSettings) on any of my 2 components (decoder and renderer) … which sucks because I eventually need to set the EGL_Display
''' Next thing to do: a lot of cleaning, reading/removing my left-over 'TODO's
* Then: fix problems
* Then: fix problems
* After that: world domination!
* After that: world domination!

Revision as of 10:10, 25 February 2015


OpenMAX decoding of images

Description

Add OpenMAX IL as a backend for image decoding. Use hardware decoder to directly decode into graphics-memory (zero-copy-decoding). Additional we need to keep Openmax backend for mm kit in mind, as this is the next logical step in improving performance/features.

Sub tasks

  1. Investigate OpenMAX IL API and concept
  2. Implement a prove of concept in application space
  3. First, small benchmark application to compare implementation with "normal" decoding
  4. Integrate implementation into Qt image-loading code
  5. Improve benchmark application; Run sophisticated tests
  6. Upstream
  7. white-paper

Optional sub tasks

This sub tasks might or might not be covered, based on the findings and decision happening.

  • Implement Qt-OpenMAX_IL-wrapper
  • build/port OpenMAX AL for the rpi
  • build/port gst-openmax

This sub tasks are in relation with Openmax backend for mm kit

Logbook

28. February - Thomas Senyk

  • Reading OpenMAX IL Specification (and building my first Qt5 for rpi)
  • Searching for example code

29. February - Thomas Senyk

  • Further searching for example code … found gst-openmax, but gst code isn't great as a reference ;)
  • Got some special code underhand :)

1. March - Thomas Senyk

  • First look at the handed OpenMAX IL example indicates the following:
    • OpenMAX IL is a micro-controll API (as suspected)
    • OpenMAX IL is a async c api, with callback, state-control, ect. (as suspected)
  • The conclusion is that there are 3 options to continue:
  1. implement a image-decoder-plugin which directly uses openmax il, don't do much abstraction, just use it.
  2. write a proper Qt/c++ API to access basic OMX IL interfaces (components, tunnels, callbacks, ect.). … and based on that, write a image decoder plugin.
  3. As there is already a "convenience" API "for" OpenMAX IL: Try to get OpenMAX AL running.

Option 3 seams to be the most effective and cleanest approach, it's preferable but depends on how easy it is to build/intergrate or of rpi will eventually support it out-of-the-box

2. March - Thomas Senyk

  • Further investigation of options
  • Started proof-of-concept implementation

13. March - Thomas Senyk

  • Finished implementation. It's "running", but it's only producing errors so far.
  • The problem now is to understand the behavior, expectation and communication with OpenMAX IL.
  • It could take a while to figurer everything out and get something to the screen

14. March - Thomas Senyk

  • Got more understanding of the behavior and expectations of OMX IL
  • Sorted a few setup things out
  • reimplemented the thread-sync code
  • got to a state where a got a few specific errors which need so be solved
  • THE main error I got is that I can't set: OMX_SetParameter(, OMX_IndexParamPortDefinition, &portSettings) on any of my 2 components (decoder and renderer) … which sucks because I eventually need to set the EGL_Display

Next thing to do: a lot of cleaning, reading/removing my left-over 'TODO's

  • Then: fix problems
  • After that: world domination!

15. March - Thomas Senyk

  • A lot of cleanup
  • Better understanding of what happens when/how/where
  • Code seams to be ok
  • Had successful run of image-decoding … seams to be very unstable … probably some check/wait/additional property missing …
  • couldn't get anything to screen … I hate opengl es 2.0 … need to fix painting, then retry

16. March - Thomas Senyk

  • very quickly fixed the painting … was a stupid error
  • clean up of the code
  • created repository
  • it's seams to work now, but texture doesn't change … I need to reboot everytime I start the application … at the 2nd execute (without reboot) the program hangs on OMX_FillThisBuffer(mRenderHandle, eglBuffer); … the 3rd time it can't even init EGL.

I need help to fix this … going to get in contact with the xbmc guys.

20. March - Thomas Senyk

  • some communication with xbmc guys … but the tips so far didn't help
  • still stuck … no clue how to proceed

21. March - Thomas Senyk