[whatwg] Codecs for <audio> and <video>

[Charles Pritchard]

Ian Hickson wrote:
> On Thu, 2 Jul 2009, Charles Pritchard wrote:
>   
>> I'd like to see <canvas> support added to the <video> tag (it's as 
>> natural as <img>).
>>     
>
> <video> elements can be painted onto <canvas> elements already; did you 
> have something more in mind?
>   
This is sufficient. <video> can be used with the drawImage tag,
and original video element can be hidden.

>> and enable the <audio> tag to accept raw data lpcm),
>> just as the <canvas> tag accepts raw data (bitmap).
>>     
>
> This is on the list of things to consider in a future version. At this 
> point I don't really want to add new features yet because otherwise we'll 
> never get the browser vendors caught up to implementing the same spec. :-)
>   
Consider a programmable <audio> element as a priority.

Apple has said they will not carry the Vorbis codec in their distributions.
I don't see any objection in them carrying a programmable <audio> element.

With a raw data api, Vorbis enthusiasts may share their codec.
Otherwise, HTML 5 is codec naive.

Programmable audio has support in Flash and Java, and their wide audience.
WebIDL is described in ECMAScript and Java.

I believe this is an achievable compromise.

It's already available in ActionScript and Java,
and their associated VMs installed on well over 90% of web browsers,
with open source development tools and compilers.

>> And add an event handler when subtitles are enabled / disabled.
>>     
>
> This is on the list for the next version. (Generally speaking, control 
> over subtitles is one of the very next features that will be added.)
>
>   
Meanwhile, using a <canvas> tag, drawImage(HTMLVideoElement) and fillText
are sufficient.

>> I'd think that FLAC would make more sense than PCM-in-Wave, as a PNG 
>> analog.
>>     
>
> I encourage you to suggest this straight to the relevant vendors. As it 
> stands, HTML5 is codec-neutral.
>   
Perhaps it's not necessary.

There's no need to standardize on an audio codec.
It can be handled in a device independent manner,
provided the player for that codec can be expressed in
ECMAScript+Java (see: WebIDL).


>> I'd like to see a font analog in audio as well. Canvas supports the font 
>> attribute, audio could certainly support sound fonts. Use a generated 
>> pitch if your platform can't or doesn't store sound fonts.
>>     
>
> This seems like an issue for the CSS or SVG working groups, who are 
> working on font-related technologies.
>   
Should it gather any momentum, this would be an attribute which 
HTMLMediaElement tags should
be aware of. For example: <audio style="-ext-audio-font: 
tone|url(arbitrary.url)" />.

Should the audio source be a container format, such as midi, it would 
interpret the css property.


>> "User agents should provide controls to enable the manual selection of 
>> fallback content."
>>     
>
> There's no reason the UA couldn't provide an override mechanism to select 
> an alternative source if the UA vendor so desires.
>   
I was considering the current dead-lock about the situation of h.264 and 
theora codecs.
An explicit statement encourages UA-developers to adopt good practices.
>> Many non-technical users will want to know why there is a black screen 
>> (or still image), even though they can hear the audio.
>>     
This section works well:

"User agents that cannot render the video may instead make the element 
represent a link to an external video playback utility or to the video 
data itself."