[whatwg] Codecs for <audio> and <video>

[Maciej Stachowiak]

On Jun 30, 2009, at 9:13 PM, Gregory Maxwell wrote:

> On Tue, Jun 30, 2009 at 10:41 PM, Maciej Stachowiak<mjs at apple.com>  
> wrote:
>> I looked into this question with the help of some experts on video  
>> decoding
>> and embedded hardware. H.264 decoders are available in the form of  
>> ASICs,
>> and many high volume devices use ASICs rather than general-purpose
>> programmable DSPs. In particular this is very common for mobile  
>> phones and
>> similar devices - it's not common to use the baseband processor for  
>> video
>> decoding, for instance, as is implied by some material I have seen  
>> on this
>> topic, or to use other fully general DSPs.
>
> Can you please name some specific mobile products? Surely if it's
> common doing so shouldn't be hard.  I don't mean to argue that it
> isn't true or intend to debate you on the merits of any examples…  But
> this is an area which has been subject to a lot of very vague claims
> which add a lot more confusion rather than insight.

For the mobile phones where I have specific knowledge regarding their  
components, I am not at liberty to disclose that information.

However, it's quite clear from even a cursory investigation that H.264  
ASICs are available from multiple vendors. This would not be the case  
if they weren't shipping in high volume products. As I'm sure you  
know, ASICs have fairly high up-front costs so they need volume to be  
cost effective.

> Iphone (of all vintages), and Palm Pre have enough CPU power to do
> Theora decode for 'mobile resolutions' on the main cpu (no comment on
> battery life; but palm pre is OMAP3 and support for that DSP is in the
> works as mentioned).

I can tell you that iPhone does not do H.264 decoding on the CPU.

> I can state this with confidence since the
> horribly slow 400mhz arm4t based SOC in the OpenMoko freerunner is
> able to (just barely) do it with the completely unoptimized (for arm)
> reference libraries (on x86 the assembly optimizations are worth a
> 30-40% performance boost).

No one doubts that software implementations are available. However,  
they are not a substitute for hardware implementations, for many  
applications. I would expect a pure software implementation of video  
decoding on any mobile device would decimate battery life.

>
> Another example I have is the WDTV, a set top media box.  It's often
> described as using a dedicated hardware H.264 decoder, but what it
> actually uses is a SMP8634. Which is a hardware decode engine based on
> general purpose processors which appears to be format-flexible enough
> to decode other formats. (Although the programing details aren't
> freely available so its difficult to make concrete claims).

I would caution against extrapolating from a single example. But even  
here, this seems to be a case of a component that may in theory be  
programmable, but in practice can't be reprogrammed by the device  
vendor.

>
> [snip]
>> As far as I know, there are currently no commercially available  
>> ASICs for
>> Ogg Theora video decoding. (Searching Google for Theora ASIC finds  
>> some
>> claims that technical aspects of the Theora codec would make it  
>> hard to
>> implement in ASIC form and/or difficult to run on popular DSPs, but  
>> I do not
>> have the technical expertise to evaluate the merit of these claims.)
>
> There is, in fact, a synthetically VHDL implementation of the Theora
> decoder backend available at http://svn.xiph.org/trunk/theora-fpga/

I did not mention FPGAs because they are not cost-competitive for  
products that ship in volume.

The points I wanted to make are simply this:

- H.264 decoders are in fact available in ASIC form, this is not a  
case of general-purpose hardware that could be programmed to do any  
codec
- H.264 ASICs are not only available but actually used
- There are no commercially available equivalents for Theora at this  
time

Silvia implied that mass-market products just have general-purpose  
hardware that could easily be used to decode a variety of codecs  
rather than true hardware support for specific codecs, and to the best  
of my knowledge, that is not the case.

Regards,
Maciej