[whatwg] Codecs for <audio> and <video>

[Ian Hickson]

I understand that people are disappointed that we can't require Theora 
support. I am disappointed in the lack of progress on this issue also.


On Tue, 30 Jun 2009, Silvia Pfeiffer wrote:
> On Tue, Jun 30, 2009 at 2:50 PM, Ian Hickson<ian at hixie.ch> wrote:
> > I considered requiring Ogg Theora support in the spec, since we do 
> > have three implementations that are willing to implement it, but it 
> > wouldn't help get us true interoperabiliy, since the people who are 
> > willing to implement it are willing to do so regardless of the spec, 
> > and the people who aren't are not going to be swayed by what the spec 
> > says.
> 
> Inclusion of a required baseline codec into a standard speaks more 
> loudly than you may think. It provides confidence - confidence that an 
> informed choice has been made as to the best solution in a given 
> situation. Confidence to Web developers, confidence to hosting 
> providers, confidence also (but less so, since they are gatekeepers in 
> this situation) to Browser Vendors.

Words in the spec only provide this confidence because that confidence is 
well-placed. If we start making the spec say things that don't represent 
reality -- like "everyone agrees that you should use Theora" -- then that 
confidence will erode in all parts of the spec.


> In my opinion, including a baseline codec requirement into a W3C 
> specification that is not supported by all Browser Vendors is much 
> preferable over an unclear situation

Having a baseline codec requirement that is not supported by all Browser 
Vendors _is_ an unclear situation. Adding the requirement doesn't stop 
that.


> In fact, it is a tradition of HTML to have specifications that are only 
> supported by a limited set of Browser Vendors and only over time 
> increasingly supported by all - e.g. how long did it take for all 
> Browser vendors to accept css2, and many of the smaller features of 
> html4 such as fixed positioning?

CSS2.1 and HTML5 were reactions to the exact problem you describe in CSS2 
and HTML4.


> I firmly believe that making the decision to give up on baseline codecs 
> is repeating a mistake made and repeatedly cited as a mistake on the 
> lack of specification of a baseline format for images - which is one of 
> the reasons why it took years to have two baseline image codecs 
> available in all browsers. We could try the other route for a change and 
> see if standards can actually make a difference to adoption.

We basically tried the idea of requiring something that browser vendors 
didn't agree with with XHTML2. IMHO that is a far bigger mistake than not 
specifying something that people don't agree to implement.


> > Going forward, I see several (not mutually exclusive) possibilities, all
> > of which will take several years:
> >
> >  1. Ogg Theora encoders continue to improve. Off-the-shelf hardware Ogg
> >    Theora decoder chips become available. Google ships support for the
> >    codec for long enough without getting sued that Apple's concern
> >    regarding submarine patents is reduced. => Theora becomes the de facto
> >    codec for the Web.
> 
> This to me is a defeat of the standardisation process. Standards are
> not there to wait for the market to come up with a de-facto standard.
> They are there to provide confidence to the larger market about making
> a choice - no certainty of course, but just that much more confidence
> that it matters.

Actually HTML5 is largely built on the idea of speccing the de-facto 
standards, either long after they were implemented, or in tandem with them 
being implemented. Very little of HTML5 has been ahead of implementations.


> >  2. The remaining H.264 baseline patents owned by companies who are not
> >    willing to license them royalty-free expire, leading to H.264 support
> >    being available without license fees. => H.264 becomes the de facto
> >    codec for the Web.
> 
> That could take many years.

Yup.


> Your main argument against Theora is a recent email stating that YouTube 
> could not be run using Theora.

No; my only argument against Theora is that Apple won't implement it.


> > No, that was just a tactical withdrawal. This e-mail here is the one 
> > that admits defeat. :-)
> 
> First rule of standards: never give up!

My first rule of standards, if I have one, would be "spec reality".


> Including a Theora prescription and having it only partially supported 
> with at least give a large part of the world an interoperable platform 
> and that's all HTML has traditionally been.

The same argument could be made for H.264.


At the end of the day, if the relevant implementors (in this case browser 
vendors) aren't willing to all implement the same codec, then the codecs 
will just have to compete on their own merits.

This isn't HTML5's fight.


On Tue, 30 Jun 2009, Jeff McAdams wrote:
> 
> But "Most people agreed, and one or two vendors objected violently" 
> probably is.  Just because one or two people are really loud, doesn't 
> mean that there isn't concensus.

The WHATWG doesn't work on consensus.


> Also, I find the focus on vendors to the exclusion of other stakeholders 
> a bit concerning.

The relevant implementors (in this case browser vendors) have the ultimate 
control (since they implement the spec) over what happens. It's not that 
other stakeholders are excluded, but that if the browser vendors aren't on 
board, there is no point moving on.


On Tue, 30 Jun 2009, Joshua Brickner wrote:
> 
> IMHO, the fundamental question here is whether or not the spec should be 
> concerned solely about creating a standard that is satisfactory for 
> implementers to follow, or if it should go further and try to make the 
> standard work well for everyone involved including developers and 
> consumers.
> 
> I am certain that most would like to have a standard that best serves 
> the entire community.

It doesn't help the community if it's not implemented.


On Wed, 1 Jul 2009, Anne van Kesteren wrote:
> 
> The "vendor consensus" line of argument seems like a very dangerous 
> slippery slope. It would mean that whenever a vendor refuses to 
> implement something it has to be taken out of the specification. I.e. 
> giving a single vendor veto power over the documentation of the Web 
> Platform. Not good at all in my opinion.

That's exactly what we have. Mozilla won't implement SQL -- SQL is being 
taken out of the Web Storage draft. Apple won't implement Theora -- Theora 
is out of the HTML5 draft.


On Wed, 1 Jul 2009, Anne van Kesteren wrote:
> >
> > My sole goal was to try and point out that the situation with codecs 
> > is not equivalent to past cases where vendors merely _hadn't 
> > implemented_ part of the spec; in this case vendors have _actively 
> > refused_ to implement support for various codecs (Apple with Theora 
> > and Mozilla(/Opera?) with H.264).
> 
> Somehow I doubt that if e.g. Opera vetoed the <video> element it would 
> actually be removed from the specification. And if it that were the case 
> I would consider it to be very bad as I mentioned in my initial email in 
> this thread.

If Opera decided that <video> as a whole is something they would never 
impelment, then it would probably be moved into a separate spec, just like 
SQL is being moved into a separate spec.


On Tue, 30 Jun 2009, Peter Kasting wrote:
>
> * Acid3 was meant to be an illustrative example of a case where the test 
> itself was not intentionally introducing new behavior or attempting to 
> force consensus on unwilling vendors, not a perfect analogy to something

Also, Acid3 wasn't a particularly great example of how to do this. I made 
an number of mistakes in its development. Hopefully we'll do things better 
with Acid4.


On Tue, 30 Jun 2009, Aryeh Gregor wrote:
> 
> *Requiring* Ogg support might be a bit much, I agree, since it implies 
> that nobody would have a legitimate foreseeable reason for not 
> supporting it, and that's ungenerous at best.  But the spec could still 
> *encourage* Ogg support while acknowledging there may be reasons not to 
> have it.  Saying all implementations *should* support Ogg Theora and 
> Vorbis acknowledges that there "may exist valid reasons in particular 
> circumstances to ignore a particular item, but the full implications 
> must be understood and carefully weighed before choosing a different 
> course".  This seems like a perfectly reasonable response to me.

To be honest I don't see much point in a non-normative mention of Theora 
in the spec. It wouldn't have the teeth people want to change reality, and 
it wouldn't help document reality.


On Tue, 30 Jun 2009, Mikko Rantalainen wrote:
> 
> I don't know about Microsoft but Apple has displayed willingness to 
> implement what specifications say (see http://acid3.acidtests.org/ for 
> example). By W3C standards a spec can get REC status if it has at least 
> two implementations and we already have three. The current HTML 5 spec 
> already has stuff not implemented by every vendor, why should <video> be 
> different?

Does HTML5 have anything that any vendor flat-out refuses to implement, 
other than Theora support? If so, that should be removed.


> I'd suggest one of the two choices (I prefer the first one):
> 
> (1) Specify Theora as the baseline codec. Hopefully it will be tested by 
> acid4 test (or by some other popular test) and Apple will either 
> implement it regardless of the assumed patent risks or finds the actual 
> patent owners and acquires the required licenses for Theora to be 
> implemented by Apple. In the future, if Apple implements Theora, then 
> perhaps even Microsoft will do so, too.

Apple have said that they won't do this.


> (2) Specify {Theora or H.264} as the baseline. That way all vendors that 
> have displayed any interest for <video> could implement the spec. 
> Authors would be required to provide the video in both formats to be 
> sure that any spec compliant user agent is able to display the content, 
> but at least there would be some real target set by the spec. However, I 
> think that this just moves the H.264 patent licensing issue from browser 
> vendors to content authors: if you believe that you cannot decode H.264 
> without proper patent license there's no way you could encode H.264 
> content without the very same license. As a result, many authors will 
> not be able to provide H.264 variant -- and as a result the Theora would 
> become de facto standard in the future.

I don't think this would be any difference, in practice, than what we have 
now.



On Tue, 30 Jun 2009, Jeff McAdams wrote:
> 
> Yes, clearly publishing the spec with a baseline codec specified isn't 
> *sufficient* for definitively "get[ting] us true interoperabiliy[sic]", 
> but it certain does *help* get us true interoperability, in two ways 
> that I can think of off the top of my head.
> 
> First, there is some inherent pressure for implementing the spec.

I think you overestimate this pressure. (Just look at XHTML2.)


> Again, some parties have indicated that it is not enough to get them to 
> do so, but that eliminates their ability to claim adherence to this 
> standard when others are doing so.

Given that nobody has ever correctly and fully implemented HTML4, and yet 
people have claimed to have implemented HTML4, I think that's demonstrably 
false.


> Second, it gives us (people like me) an extra tool to go back to vendors and
> say, "Hey, please support HTML5, its important to me, and the <video> tag,
> with the correct baseline codec support, is important to me."

In my experience your requests would actually have more power on Apple if 
they were _not_ backed by the spec. That is, you will have more effect on 
Apple if HTML5 doesn't say which codec to support and you _still_ go to 
Apple and ask for Theora support, than if the spec says Theora and you ask 
for Theora. In the latter case, they are much more likely to dismiss your 
request as "just someone checking that we support the spec", in the former 
case, they are more likely to actually believe you need Theora support.


> You've said a couple of things that I perceive as contradictory, here. 
> You've said that you want to help bring about interoperability, but then 
> you've also said that you're only documenting what it is that the 
> browser makers are implementing.  There is room in the standards bodies 
> world for both goals, and both goals, at times are valid and beneficial.  
> But, if your intent is to help bring about interoperability, *real* 
> interoperability, then I think its pretty clear that the way forward 
> involves specifying a baseline codec.

The way forward involves getting to the point where we can specify a 
baseline codec, yes. But we do that _after_ everyone agrees on a baseline 
codec. While it may appear that I write stuff in the spec and the browser 
vendors then agree to it, in practice it's usually the other way around.


> Leaving such an important point of interoperability completely up to the 
> whims of "people out there" seems unwise here (I look at MS's latest 
> attempt at supporting ODF as a great example of how interoperability can 
> actually be harmed, even by a "complying" implementation, when important 
> parts of guidelines to interoperability are left out...there are plenty 
> more examples).

This seems unaffected by whether the spec requires this or not.


On Tue, 30 Jun 2009, Dr. Markus Walther wrote:
> > 
> > Having removed everything else in these sections, I figured there 
> > wasn't that much value in requiring PCM-in-Wave support. However, I 
> > will continue to work with browser vendors directly and try to get a 
> > common codec at least for audio, even if that is just PCM-in-Wave.
> 
> Please, please do so - I was shocked to read that PCM-in-Wave as the 
> minimal 'consensus' container for audio is under threat of removal, too.

The consensus isn't removed, only the text documenting it.


> Frankly, I don't understand why audio was drawn into this. Is there any 
> patent issue with PCM-in-Wave? If not, then IMHO the decision should be 
> orthogonal to video.

I just don't see much value in calling this one minor codec amongst all 
the other formats and codecs that browsers have to support (CSS, JS, HTTP, 
PNG, GIF, JPEG, SVG, etc).


On Tue, 30 Jun 2009, Manuel Amador (Rudd-O) wrote:
> 
> The objecting vendors could have taken a number of approaches to band 
> together and protect themselves from the supposed "patent threats" 
> against free formats, but they did not.  They could have lobbied against 
> software patents; they didn't.  They could have agreed to countersue -- 
> with their immense patent portfolio -- anyone who sued any of them for 
> patent infringement; they didn't.  Of course, why would the objecting 
> vendors do that? Most of the competitive advantages they have stem from 
> being able to leverage government force against potential competitors, 
> so doing any of the above would have undermined that.

Actually, I can attest from first-hand knowledge that the "objecting 
vendors" did attempt to find solutions to this. Though this was not done 
in public, plenty of efforts have been made, and continue to be made, to 
find other solutions. The vendors involved really do appear to be acting 
in good faith.

On Thu, 2 Jul 2009, Charles Pritchard wrote:
> 
> Can the standard simply address video containers (OGG, MKV, AVI) ?
> Each container is fairly easy to implement and codecs can be identified within
> the container.
> Vendors can decide on their own what to do with that information.

The spec does document how to distinguish containers via MIME type. Beyond 
that I'm not sure what we can do.

<video> does support fallback, so in practice you can just use Theora and 
H.264 and cover all bases.

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'