[whatwg] Codecs (was Re: Apple Proposal for Timed Media Elements)

[Maciej Stachowiak]

On Mar 21, 2007, at 6:16 PM, Ian Hickson wrote:

> * I'm concerned about the "type" attribute for content negotiation.
>   Historically, type attributes are very badly implemented and even  
> less
>   reliably used. Conditional fallback in general is badly  
> implemented and
>   bug-prone especially in the context of dynamic changes. In  
> addition, I'm
>   not convinced there is much in the way of multi-codec data on the  
> Web
>   that would be addressed by this.

ON FALLBACK

I think the lack of multi-codec data is in part because it's not easy  
to automatically present the right video stream out of several  
streams using <object>. It's hard enough to write the <object> markup  
to work in all browsers with a single codec!

But I think that having a good mechanism to do this is important.  
Here are some reasons:

- Even if all browsers end up supporting Ogg Theora/Vorbis, these are  
not the best-compression codecs available. So a large-scale video  
content provider that wants to save on bandwidth may wish to provide  
H.264/AAC content to those browsers that can handle it, even if all  
browsers could handle a lower-quality codec as well.

- Many mobile devices cannot practically implement decoding in  
software, and rely on custom hardware which can handle only a fixed  
set of codecs. While hardware decoders for H.264 are widely  
available, I don't think there are any for Ogg Theora. Even in cases  
where the CPU in theory has the power to do some software decoding,  
this will be a much bigger battery drain than hardware decoding. So  
you really want the ability to serve the right codec to such devices.

So while your average blogger may only upload media content in one  
codec, larger scale video service providers may want to take  
advantage of codec-based selection.


I think the fallback mechanism specified avoids some of the pitfalls  
of other fallback mechanisms:

A) It is specified to take the declared type as authoritative for  
fallback purposes, so dynamic fallback and its attendant complexities  
do not have to get involved.

B) It lets you select based on codec and even codec profile, not just  
container format.

C) The <video> syntax itself is simple enough that it won't reduce to  
an incomprehensible jumble like it sometimes does with <object>.


However, it's true that in general you may also want to consider  
issues such as screen size and data rate when choosing from several  
available video options. QuickTime has a concept of selector movies  
than can choose to download one of several separate resources based  
on such criteria, but it makes more sense to do it in markup or CSS.

We discussed the possibility of using CSS media queries to account  
for screen size and data rate. However, this has a couple of issues:

- You probably still want a mechanism for codec-based selection.  
Exposing available codecs to media queries seems like it will be very  
complex, comparing to declaring what codecs you use and letting the  
UA decide.

- To emulate fallback with CSS media queries, you need to make sure  
your queries are mutually exclusive, which generally means query 2  
has to include "not query1 and...", query 3 has to negate both  
queries 1 and 2, and so forth. This seems more complex to author than  
a fallback model.

All that being said, we are not entirely sure what the best approach  
is for screen size and data rate fallback, but "type" seems like a  
good model for format-based fallback.


ON RECOMMENDED OR MANDATED CODECS

We think it is a mistake to require Ogg support, even as a SHOULD- 
level requirement, for several reasons.

- As mentioned above, some devices may have a much harder time  
implementing Ogg than other codecs. Although a SHOULD-level  
requirement would excuse them, I'm not sure it's appropriate to have  
it if it might be invoked often.

- Although the Ogg codecs don't have known patents that aren't RF  
licensed, it's not completely clear that none of the patents out  
there on video/audio encoding apply. Often, parties holding a  
submarine patent wait for a company with very deep pockets (like  
Apple, or Microsoft, or Google) to infringe on the patent before they  
sue. On the other hand, MPEG codecs have been implemented by many  
large corporations already, and no patents have appeared besides the  
ones that can be licensed from MPEG-LA for a fee. So, ironically, for  
a large company that has no problem the patent fees, Ogg may carry  
more patent risk than MPEG.

- Placing requirements on format support would be unprecedented for  
HTML specifications, which generally leave this up to the UA, with de  
facto baseline support being decided by the market.

We are very sympathetic to the desire for interoperability, and we  
would really like there to be a codec that every browser can feel  
comfortable implementing. But we are not sure such a codec exists at  
this time (except for really primitive things, if you want to count  
animated GIF or APNG as video codecs).


Regards,
Maciej