[whatwg] Subtitles, captions, and other tracks augmenting video or audio

[Ian Hickson]

On Tue, 20 Apr 2010, Silvia Pfeiffer wrote:
> 
> I spent some time today filling that page and when I came back to it 
> just now it seems you have moved most of the use cases elsewhere, namely 
> to 
> http://wiki.whatwg.org/wiki/Use_cases_for_API-level_access_to_timed_tracks 
> .
> 
> IIUC the idea is to focus on the core problem at hand which right now 
> are captions and subtitles. That is fair enough, but I think you might 
> want to reconsider this for lyrics and chapter markers. I'm ok with 
> moving the others to a later stage.

The one page was starting to cover more topics than I could fit in my head 
at once. :-)

The two pages are both things I think we need to support in the first 
version. The first page (timed_tracks) is stuff that needs in-video 
rendering, and that therefore affects whatever formatting model we end up 
using. The stuff on the second page (use_cases_for_API-level_access_to_ 
timed_tracks) is stuff that only affects the UI or APIs but that I 
wouldn't expect the browser itself to render natively without script.


> Firstly about the Lyrics. I think they are just the same as captions and 
> should go back into the first document. In particular since we are 
> talking about captions and subtitles for both the <video> and the 
> <audio> element and this shows some good examples of how lyrics are 
> being displayed as time-aligned text with audio resources. Most of these 
> examples are widgets used on the Web, so I think they are totally 
> relevant.

All the examples are off-video lyrics except one which I presume everyone 
would agree should require script or at least SMIL+SVG. This suggests to 
me that they are all things we should handle as part of an API, and not as 
a native built-in feature.

Clearly though, lyrics, along with subtitles and karaoke, should be 
supported by whatever format(s) we use to expressed timed track data.


> Lyrics (LRC) files typically look like this:
> 
> [ti:Can't Buy Me Love]
> [ar:Beatles, The]
> [au:Lennon & McCartney]
> [al:Beatles 1 - 27 #1 Singles]
> [by:Wooden Ghost]
> [re:A2 Media Player V2.2 lrc format]
> [ve:V2.20]
> [00:00.45]Can't <00:00.75>buy <00:00.95>me <00:01.40>love,
> <00:02.60>love<00:03.30>, <00:03.95>love, <00:05.30>love<00:05.60>
> [00:05.70]<00:05.90>Can't <00:06.20>buy <00:06.40>me <00:06.70>love,
> <00:08.00>love<00:08.90>
> 
> There is some metadata at the start and then there are time fragments,
> possibly overloaded with explicit subtiming for individual works in
> karaoke-style. This is not very different from SRT and in fact should
> fit with your Karaoke use case.

Indeed.


> I'm also confused about the removal of the chapter tracks. These are
> also time-aligned text files and again look very similar to SRT. Here
> is an extract of a QTtext chapter track example:
> 
> {QTtext} {size:16} {font:Lucida Grande} {width:320} {height:42}
> {language:0} {textColor:65535,65535,65535} {backColor:0,0,0}
> {doNotAutoScale:off} {timeScale:100} {timeStamps:absolute}
> {justify:center}
> [00:00:09.30]
> Chocolate Rain
> [00:00:12.00]
> Some stay dry and others feel the pain
> [00:00:16.00]
> Chocolate Rain
> [00:00:18.00]
> A baby born will die before the sin

Indeed, chapter markers are clearly something that fits into the same 
model and should be supported.

(Those are some damn short chapters. Are you sure that's not a lyric 
track?)


> So, while I can understand that you currently want to focus on just 
> solving captions and subtitles, I think it is important to keep other 
> time-aligned text applications that can be solved in the exact same way 
> part of the design to keep an open mind about general time-aligned text 
> use cases.

The split was not intended to indicate a lack of desire to support these 
use cases in whatever format we eventually pick or adapt (or invent, if 
necessary, though I'd really like to avoid that). Sorry for any confusion.

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'