[whatwg] Comments about the track element

[Cyril Concolato]

Hi Silvia,

[trimming a bit the text to keep the email short]

Le 7/27/2012 12:35 AM, Silvia Pfeiffer a écrit :
>
> There is the SVG viewbox and there is the video viewbox. It is not
> immediately clear how they relate to each other. What I meant was: how
> to position the SVG viewbox within the boundaries of the video
> viewbox. It could fully cover it, but it may not need to. For example
> in your example with the clock, it could be positioned by coordinates
> of the video, e.g. left: 70%, top:30% or something like it. Then the
> SVG can be much smaller and it is possible to overlay other elements,
> too.
I understand. What I'm saying is: when SVG is used as track, make it 
simple, both viewboxes are the same.

> You could, however, put SVG in WebVTT e.g. to provide overlay graphics
> that are non-moving or are in a loop for a certain duration of the
> video. E.g. an animated character (like your Rhino) could be rendered
> in a loop on top of a video for the first 3 minutes of the video.
Agreed, why not.

>
> I don't want to take this discussion off track, but it is news to me
> that TTML can express frame-based animations.
> I indeed wouldn't mingle WebVTT and TTML layering since they satisfy
> the same use cases.
I've seen examples like that, used to carry DVB subtitles, urgh!


> How does the browser support constructing SVG progressively right now?
> If there is a SVG-internal solution, that should be used. In this
> case, @mediagroup synchronization would again make the most sense. Or
> you just do everything in SVG.
Browsers construct SVG progressively right now as they do for HTML. The 
parser parses the data as it receives it.
@mediagroup is indeed a solution but the track has other advantages. 
Controlling the time when the data is received for instance with inband 
stored track is one.

> No, not really. What I meant was to draw the blue handle on top of the
> video not through cues, but directly in the browser. Then, the WebVTT
> file only delivers the according position changes for that particular
> time and all you need to do in JavaScript is to change the handle
> position in the SVG. That makes the WebVTT slimmer and not contain any
> SVG at all.
Right, but that again requires JS, and hence incurs some processing 
penalty. And also, this requires a dedicated authoring while using an 
SVG track, you could just use any existing tool.

>> The basic Track interface would be almost the same as the VideoTrack or
>> AudioTrack. The GraphicsDocumentTrack interface would be used for tracks
>> which have an underlying document (TTML, SVG, SMIL?, HTML?...) with a visual
>> representation and not necessarily based on cues. For these documents, you
>> are not interested in cues or cue changes (and it might not make sense to
>> define cues). For these, you're only interested in:
>> - the dispatch of the track content to the parser being done automatically
>> by the browser (no need to use a JS DOMParser);
>> - the rendering of the underlying document being synchronized (natively) by
>> the browser, i.e. the timeline of the underlying document should be locked
>> with the timeline of the video (or audio). No need to monitor cue changes to
>> render the right SVG.
>> You could discriminate between a TextTrack and a GraphicsDocumentTrack by a
>> mime type or the inBandMetadataTrackDispatchType (not sure...). When the
>> track carries SVG, the trackDocument object could be an SVGDocument. This
>> would allow for controlling the SVG as if it was embedded in the HTML but
>> for the synchronization done by the browser. What do you think?
> Why does it have to be a track at all? Video and audio can be
> synchronized to each other without one needing to be a track of the
> other. To use @mediagroup, you might need to consider what an SVG
> graphic has to provide for the MediaController [1]. There is no
> need to consider cues and tracks - we seem to agree on that. :-)
We seem to agree on many things but not all ;)
We agree that there are use cases for SVG graphics synchronize and 
overlayed on top of a video and that @mediagroup is a solution.
The track mechanism (without cue) has several advantages:
- synchronization equivalent to the @mediagroup solution
- reuse of SVG native behavior (progressive loading) equivalent to the 
@mediagroup solution
- selectability of the track using the default UI controls, just like 
with subtitles, without needing specific controls.
- rendering on top of the video (just like subtitles) without 
interfering with the UI controls.
- unified handling of out-of-band and in-band SVG tracks.
Note, that this would be perfectly applicable to HTML/CSS animations on 
top of a video.
It is just enlarging the scope of tracks, currently restricted (except 
for video and audio) to cue-based formats and text only.

Regards,
Cyril

-- 
Cyril Concolato
Maître de Conférences/Associate Professor
Groupe Multimedia/Multimedia Group
Telecom ParisTech
46 rue Barrault
75 013 Paris, France
http://concolato.wp.mines-telecom.fr/