[whatwg] <video> and acceleration

[Ian Fette]

Upgrade the hardware is not an acceptable answer. Video acceleration is
meant to offload work from CPU (especially on constrained devices, e.g.
mobile). You want to be able to do compositing on video card, so that you
don't have to read the video out of the video card's memory, transfer it
over the bus, to the CPU, do some transforms/overlays/..., and then send it
back to the video card for display. Doing that absolutely kills framerate.
As we (browsers) implement <video> I think a lot of us are starting with
software rendering. Certainly we want to be able to do hardware acceleration
at some point. Perhaps some things we will still be able to do in hardware,
e.g. overlays of HTML or certain transforms (if the video device supports
saying "take this, translate it, and composite" and the rendering engine
only needs geometry data.) Other things we might not be able to do in
hardware (e.g. if you have "transparent" flash video on top, and Flash wants
to know what pixels are underneath it, then we would have to read that data
off of the video card, send it to CPU, ...)

I think what would be helpful is for browsers who are implementing <video>
with hardware acceleration to publish information on what would make them
fall back to software rendering. If it turns out that list is roughly
similar across implementations, perhaps it could be added as a note in the
spec that doing the following certain things may cause performance
implications. We're probably not ready to do that yet given that we don't
have enough implementation experience, but that would be my suggestion for
how to move forward.

-Ian

On Tue, Apr 28, 2009 at 5:59 PM, Ian Hickson <ian at hixie.ch> wrote:

> On Sat, 28 Mar 2009, Benjamin M. Schwartz wrote:
> >
> > The <video> tag has great potential to be useful on low-powered
> > computers and computing devices, where current internet video streaming
> > solutions (such as Adobe's Flash) are too computationally expensive.
> > My personal experience is with OLPC XO-1, on which Flash (and Gnash) are
> > terribly slow for any purpose, but Theora+Vorbis playback is quite
> > smooth at reasonable resolutions and bitrates.
> >
> > The <video> standard allows arbitrary manipulations of the video stream
> > within the HTML renderer.  To permit this, the initial implementations
> > (such as the one in Firefox 3.5) will perform all video decoding
> > operations on the CPU, including the tremendously expensive YUV->RGB
> > conversion and scaling.  This is viable only for moderate resolutions
> > and extremely fast processors.
> >
> > Recognizing this, the Firefox developers expect that the decoding
> > process will eventually be accelerated.  However, an accelerated
> > implementation of the <video> spec inevitably requires a 3D GPU, in
> > order to permit transparent video, blended overlays, and arbitrary
> > rotations.
> >
> > Pure software playback of video looks like a slideshow on the XO, or any
> > device with similar CPU power, achieving 1 or 2 fps.  However, these
> > devices typically have a 2D graphics chip that provides "video overlay"
> > acceleration: 1-bit alpha, YUV->RGB, and simple scaling, all in
> > special-purpose hardware. Using the overlay (via XVideo on Linux) allows
> > smooth, full-speed playback.
> >
> > THE QUESTION:
> > What is the recommended way to handle the <video> tag on such hardware?
>
> Upgrade the hardware.
>
>
> > There are two obvious solutions:
> > 0. Implement the spec, and just let it be really slow.
> > 1. Attempt to approximate the correct behavior, given the limitations of
> > the hardware.  Make the video appear where it's supposed to appear, and
> > use the 1-bit alpha (dithered?) to blend static items over it.  Ignore
> > transparency of the video.  Ignore rotations, etc.
> > 2. Ignore the HTML context.  Show the video "in manners more suitable to
> > the user (e.g. full-screen or in an independent resizable window)".
> >
> > Which is preferable?  Is it worth specifying a preferred behavior?
>
> >From HTML's point of view, all are acceptable. From the user's point of
> view, 1 and 2 are preferable, probably at the user's option.
>
> I don't know what else to tell you. :-)
>
> --
> Ian Hickson               U+1047E                )\._.,--....,'``.    fL
> http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
> Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.whatwg.org/pipermail/whatwg-whatwg.org/attachments/20090428/694eb4ac/attachment-0002.htm>