[whatwg] Comments about the track element

[Cyril Concolato]

Hi Silvia,

Le 7/25/2012 4:24 AM, Silvia Pfeiffer a écrit :
> Expanding a bit on what Anne said...
>
> On Tue, Jul 24, 2012 at 11:18 PM, Cyril Concolato
> <cyril.concolato at telecom-paristech.fr>  wrote:
>> Dear WhatWG,
>>
>> During the ongoing SVG F2F meeting, the SVG WG discussed the use case of
>> displaying SVG graphics on top of a video, in a synchronous manner.
>>
>> The SVG WG believes that for such use case, it is necessary to indicate to
>> the browser that the SVG and video content should stay synchronized (no
>> matter what happens to the video playback), and to let the browser handle
>> the synchronization internally. The SVG WG resolved to include such
>> indication as part of the Web Animation specification, for instance using
>> the HTML mediagroup attribute or the MediaController API.
>>
>> However, the SVG WG thinks it would also be interesting to leverage the
>> native UI controls of the video element to select (or deactivate) the
>> display of the SVG content on top of the video, in a similar manner to a
>> subtitle track. Obviously, the HTML 5 track element would be a suitable
>> option for that. However, currently it only allows text tracks. So, the SVG
>> WG would like HTML to allow the track element's URL to identify an SVG
>> resource, and in that case the track kind would be 'graphics'. There would
>> be a need to define how the graphics are displayed on top of the video,
> Right now it is fully defined how data in a TextTrack (of the defined
> kinds) is displayed on top of the video. As this is as yet unclear for
> SVG resources,
I wouldn't say it's unclear, I'd say it needs to be specified ;) meaning 
that it probably doesn't require much specification. I was thinking that 
we could use the CSS box of the video element to position the SVG, as if 
the SVG was put in a div.
> I would suggest using the @metadata tack kind for now
> and providing the SVG as markup in a TextTrackCue (either from WebVTT
> cues
I've tried this option but I'm facing several problems (Tested with 
Chrome Version 22.0.1216.0 canary).

The first problem is how to embed SVG in a cue? Should the '<', '>' and 
other characters be escaped or not? According to Anne's validator, they 
should be. But if I use them, then the parsing of the escaped string 
returns 'empty document' 
(http://perso.telecom-paristech.fr/~concolat/html5_tests/getcueasSVG-escaped.html 
<http://perso.telecom-paristech.fr/%7Econcolat/html5_tests/getcueasSVG-escaped.html>). 
However, if I don't escape them, the parsing doesn't fail and returns an 
SVG document 
(http://perso.telecom-paristech.fr/~concolat/html5_tests/getcueasSVG.html <http://perso.telecom-paristech.fr/%7Econcolat/html5_tests/getcueasSVG.html>). 
In any case, I think embedding the SVG in WEBVTT does not really make 
sense.

An other problem is in terms of design. SVG has a timing model (similar 
to TTML), WebVTT another. For instance, SVG can express things like 
repetitions of animations that WebVTT cannot. Are you saying that TTML 
should be carried in a WebVTT file?
Similarly, in terms of design, embedding SVG in cues requires repeating 
a lot of SVG content at each cue (see 
http://perso.telecom-paristech.fr/~concolat/html5_tests/svg.vtt 
<http://perso.telecom-paristech.fr/%7Econcolat/html5_tests/svg.vtt>), as 
this approach requires parsing an entire document at each cue. You could 
probably envisage overlapping cues but that would require a lot of 
overhead. Leveraging the progressive loading of SVG cannot be done this 
way either.
In general, I think it would make sense to leverage the browsers' 
support for SVG and not stack different technologies.

Another problem is that I don't know if it's possible to display the SVG 
content in a layer between the video and the UI controls. Currently, I 
display the SVG on top of the video element, therefore the UI controls 
are not accessible for clicks. Having to embed my own UI controls for 
that is a bit of a pain. And, semantically, when reading the spec, 
'metadata' tracks say " Not displayed by the user agent. " so I think 
this might be a bit confusing for users/authors.

The third problem is performance-wise. In my example, the blue line (in 
SVG), when synchronized with the video, should be aligned with the 
moving (white-gray) edge of the pie. As you can see, this is not the 
case. Only 4-5 cuechange events seems to be processed properly. I 
noticed the same problem with 'timeupdate' events. Also, I've noticed 
that even though my WebVTT file is designed to have only one active cue 
at a time, for some cuechange events, there are 2. This might be an 
implementation issue but this might be a problem of reentrant code (the 
cuechange callback being called while it's not finished), but in 
general, I'm not sure it's a good idea to go through the Javascript 
engine to do that, for the processing overhead.

> or from JavaScript calls to addTextTrack()).
Can you elaborate on this one? However, I suspect it'll have the same 
processing overhead.

> The
> timing/synchronization is provided by the TextTrackCue, which seems to
> be all you are asking for right now. The rendering is then done by
> JavaScript, which seems the most flexible approach. You can even use
> getCueAsHTML() to simply hand the SVG to HTML for rendering.
>
> @metadata tracks can be part of the native video controls and the
> @label on the text track would provide a description of what it is,
> e.g. "Graphic overlays".
>
>> for
>> instance reusing the viewport/viewbox negotiation phase. There would also be
>> a need to make a more generic Track API or to replace the TextTrack API by
>> the SVG API when the track is of kind 'graphics'.
> I don't understand this requirement. What API needs are there aside
> from the synchronization? Trying to replicate SVG APIs through the
> TextTrack API seems like a repetition of the API and thus fragile.
Sorry for the confusion here. I didn't mean to replicate the SVG APIs 
here but I just meant that the TextTrack API is very specific to 'pure' 
text tracks (and even to WebVTT text tracks). You might want to expose 
the SVG API when SVG content is used for the overlay to control it.

Regards,
Cyril

-- 
Cyril Concolato
Maître de Conférences/Associate Professor
Groupe Multimedia/Multimedia Group
Telecom ParisTech
46 rue Barrault
75 013 Paris, France
http://concolato.wp.mines-telecom.fr/