[html5] About possible formats for multiple audio and subtitle streams for HTML5

[Carlos Andrés Solís]

Hello, I've been writing lately in the WHATWG and WebM mail-lists and would
like to hear your opinion on the following idea.
Imagine a hypothetical website that delivers videos in multiple languages.
Like on a DVD, where you can choose your audio and subtitles language. And
also imagine there is the possibility of downloading a file with the video,
along with either the chosen audio/sub tracks, or all of them at once. Right
now, though, there's no way to deliver multiple audio and subtitle streams
on HTML5 and WebM. Since the latter supports only one audio and one video
track, with no embedded subtitles, creating a file with multiple tracks is
impossible, unless using full Matroska instead of WebM - save for the fact
that the standard proposed is WebM and not Matroska.
A solution could be to stream the full Matroska with all tracks embedded.
This, though, would be inefficient, since the user often will select only
one language to view the video, and there's no way yet to stream only the
selected tracks to the user. I have thought of two solutions for this:
* Solution 1: Server-side demuxing. The video with all tracks is stored as a
Matroska file. The server demuxes the file, generates a new one with the
chosen tracks, and streams only the tracks chosen by the user. When the user
chooses to download the full video, the full Matroska file is downloaded
with no overhead. The downside is the server-side demuxing and remuxing;
fortunately most users only need to choose once. Also, there's the problem
of having to download the full file instead of a file with only the tracks
wanted; this could be solved by even more muxing.
* Solution 2: User-side muxing. Each track (video, audio, subtitles) is
stored in standalone files. The server streams the tracks chosen by the
user, and the web browser muxes them back. When the user chooses to download
the video, the generation of the file can be done either server-side or
client-side. This can be very dynamic but will force content providers to
use extra coding inside of the pages.

Now for part two, there is still an issue on which format should be used for
subtitling. As you might know, there are basic subtitle formats that are
formed by timed plain text and little else (like SRT or the proposed
WebSRT), and there are full-blown subtitle formats that allow for extreme
formatting and typesetting (like Advanced SubStation Alpha). The basic
subtitles have the advantage of being easily editable by hand, but
sacrificing capabilities that advanced formats allow with the cost of
harder-to-understand syntax. It would be a shame to drop advanced subtitles
from the HTML5 specs, but it would be bothersome if everybody is forced to
use a complex-to-write format. So a middle ground could be handy: allowing
WebSRT for the simple tasks, and using another format for advanced
typesetting. To put an example, ASSA allows to modify the text font, size,
border, shadow, scale, rotation, position, and some other properties; it
also allows movement of text, text animation, karaoke, and even some
vectorial graphing. But all of that could be achieved with HTML5 programming
on top of the WebSRT format (or whichever gets chosen). This, of course,
causes a pair of problems.
* The first one is that there would be no tools to edit HTML5 subtitles
specifically, forcing to make a type of subset which would have to be
standardized, plus an editor to be able to create such subtitles without
having to learn how to create a full-blown website.
* The second one is that media players that wanted to use such subtitles
would be forced to ship an HTML5 decoder. Most media players are NOT web
browsers, though, or based on one either. The only exceptions I remember are
media players built on top of XUL, like Songbird or Nightingale. But players
like WMP, WinAmp, VLC, Xine family, GStreamer family or MPlayer family would
be left out, since they have no need (and no time) to plug in a web browser
in a program that hasn't needed it.

And that would be the end of my brainstorming. Any ideas or suggestions?
- Carlos Solís
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.whatwg.org/pipermail/help-whatwg.org/attachments/20100522/3c6b6fdb/attachment-0001.htm>