[whatwg] On implementing videos with multiple tracks in HTML5

[Silvia Pfeiffer]

On Tue, Aug 24, 2010 at 1:07 AM, Eric Carlson <eric.carlson at apple.com>wrote:

>
> On Aug 20, 2010, at 5:53 PM, Silvia Pfeiffer wrote:
>
> On Sat, Aug 21, 2010 at 10:03 AM, Eric Carlson <eric.carlson at apple.com>wrote:
>
>>
>> On Aug 19, 2010, at 5:23 PM, Silvia Pfeiffer wrote:
>>
>> >
>> > * Whether to include a multiplexed download functionality in browsers
>> for media resources, where the browser would do the multiplexing of the
>> active media resource with all the active text, audio and video tracks? This
>> could be a context menu functionality, so is probably not so much a need to
>> include in the HTML5 spec, but it's something that browsers can consider to
>> provide. And since muxing isn't quite as difficult a functionality as e.g.
>> decoding video, it could actually be fairly cheap to implement.
>> >
>>
>>   I don't understand what you mean here, can you explain?
>>
>>
>>
>
> Sure. What I mean is: you get a video resource through the <video> element
> and a list of text resources through the <track> element. If I as a user
> want to take away (i.e. download and share with friends) the video file with
> the text tracks that I have activated and am currently watching, then I'd
> want a download feature that allows me to download a single multiplexed
> video file with all the text tracks inside. Something like a MPEG-4 file
> with the <track> resources encoded into, say, 3GPP-TT. Or a WebM with WebSRT
> encoded (if there will be such a mapping). Or a Ogg file with WebSRT - maybe
> encoded in Kate or natively.
>
> The simplest implementation of such a functionality is of course where the
> external text track totally matches the format used in the media resource
> for encoding text. Assuming WebM will have such a thing as a WebSRT track,
> the "download" functionality would then consist of multiplexing a new WebM
> resource by re-using the original WebM resource and including the WebSRT
> tracks into that. It wouldn't require new video and audio encoding, since
> it's just a matter of a different multiplexed container. If transcoding to
> the text format in the native container is required, then it's a bit more
> complex, but no less so than what we need to do for extracting such data
> into a Web page for the JavaScript API (it's in fact the inverse of that
> operation).
>
> So, I wouldn't think it's a very complex functionality, but it certainly
> seems to be outside the HTML spec and a browser feature, possibly at first
> even a browser plugin. Sorry if this is now off topic. :-)
>
>   Even in the hypothetical case where the external text track is already in
> a format supported by the media container file, saving will require the UA
> to regenerate the movie's "table of contents" (eg. the 'moov' atom in MPEG-4
> or QuickTime files, Meta Seek Information in a WebM file) as well as muxing
> the text track with the other media data.
>
>   As you note transcoding is "a bit more complex", especially in the case
> where a feature in the text track format is not supported by the text format
> of the native container.
>
>   Further, what should a UA do in the case where the native container
> format doesn't support any form of text track - eg. mp3, WAVE, etc?
>


Well, for those you cannot obviously expect a single download file. Maybe it
makes sense to just have a download functionality that allows downloading
the text tracks as well. I was just following on from some of the user
requirements raised in this thread.



>   I disagree that it is not a complex feature, but I do agree that it is
> outside of the scope of the HTML spec.
>


I guess the complexity really depends on the format in use. For Ogg there is
plenty of software available to demux and remux bitstreams, which is the
main functionality here. The second part is the encoding of the text track,
which again for Ogg has separate tools and libraries. After downloading the
text tracks and the Ogg file separately, I would find it very easy to create
a new multiplexed file using all these tools. That's where my judgement of
"simple" came from. But you are probably right and it's a lot more
complicated for other formats.

Enough off-topic brainstorming. ;-) I think we have more important things to
solve right now.

Cheers,
Silvia.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.whatwg.org/pipermail/whatwg-whatwg.org/attachments/20100824/ff0ad2a0/attachment-0002.htm>