[whatwg] How to handle multitrack media resources in HTML

Thu Feb 17 08:55:16 PST 2011

Silvia,

MPEG transport streams, as used for commercial TV, will often contain multiple types of metadata: content advisories, ad insertion opportunities, interactive TV application triggers, etc.  If we were getting this information out-of-band we would, as you suggest, know how to deal with it.  We would use multiple @kind=metadata tracks, with the correct handler associated with each track.  In our case, however, this information is all coming in-band. 

There is information within the MPEG transport stream that identifies the types of metadata being carried.  This lets the video player know, for example, that the stream has a particular track with application triggers, and another one with content advisories.  To be consistent with the out-of-band tracks, we envision the player creating separate TimedTrack elements for each type of metadata, and adding the associated data as cues.  But there isn't a clear way for the player to indicate the type of metadata it's putting into each of these TimedTrack cues.

Which brings us to the mime types.  I have an event handler on the <video> tag that fires when the player creates a new metadata track, and this handler tries to figure out what to do with the track.  Without a type on the track, I have to set another handler on the track that fires when the player creates a cue, and tries to figure out what to do from the cue.  As there is no type on the cue either, I have to examine the cue location/text to see if it contains metadata I'm able to handle.

This all works, but it requires event handlers on tracks that may have no interest to the application.  On the player side, it depends on the player tagging the metadata in a consistent ad-hoc way, as well as requiring the player to create separate metadata tracks.   (We also considered starting the cue's text with a mime type, but this has the same basic issues.)

Clear as mud, right?

Thanks,

Eric Winkelman
---
CableLabs

> -----Original Message-----
> From: Silvia Pfeiffer [mailto:silviapfeiffer1 at gmail.com]
> Sent: Wednesday, February 16, 2011 1:34 PM
> To: Eric Winkelman
> Cc: WHAT Working Group
> Subject: Re: [whatwg] How to handle multitrack media resources in HTML
> 
> Hi Eric,
> 
> I'm curious: if you are using @kind=metadata - which is not generically
> applicable, but only has application-specific data in it - then this implies that
> the web page knows what type of data is in the track's cues and knows how
> to parse it. Why do you need a mime type on the cues then? Is it because
> MPEG has metadata cue tracks that can contain different types of structured
> content? Can you clarify?
> 
> Cheers,
> Silvia.
> 
> On Thu, Feb 17, 2011 at 6:44 AM, Eric Winkelman
> <E.Winkelman at cablelabs.com> wrote:
> > Silvia, all,
> >
> > We're working with multitrack MPEG transport streams, and have an
> implementation of the TimedTrack interface integrating with in-band
> metadata tracks.  Our prototype uses the Metadata Cues to synchronize a
> JavaScript application with a video stream using the stream's embedded EISS
> signaling.  This approach is working very well so far.
> >
> > The biggest issue we've faced is that there isn't an obvious way to tell the
> browser application what type of information is contained within the
> metadata track/cues.  The Cues can contain arbitrary text, but neither the
> Cue, nor the associated TimedTrack, has functionality for specifying the
> format/meaning of that text.
> >
> > Our current implementation uses the Cue's @identifier for a MIME type,
> and puts the associated metadata into the Cue's text field using XML.  This
> works, but requires the JavaScript browser application to examine the cues
> to see if they contain information it understands.  It also requires the video
> player to follow this convention for Metadata TimedTracks.
> >
> > Adding a @type attribute to the Cues would certainly help, though it would
> still require the browser application to examine individual cues to see if they
> were useful.
> >
> > An alternate approach would be to add a @type attribute to the <track>
> tag/TimedTrack that would specify the mime type for the associated
> cues.  This would allow a browser application to determine from the
> TimedTrack whether  or not it needed to process the associated cues.
> >
> > Eric
> > ---
> > Eric Winkelman
> > CableLabs
> >
> >> -----Original Message-----
> >> From: whatwg-bounces at lists.whatwg.org [mailto:whatwg-
> >> bounces at lists.whatwg.org] On Behalf Of Silvia Pfeiffer
> >> Sent: Wednesday, February 09, 2011 5:41 PM
> >> To: WHAT Working Group
> >> Subject: [whatwg] How to handle multitrack media resources in HTML
> >>
> >> Hi all,
> >>
> >> One particular issue that hasn't had much discussion here yet is the
> >> issue of how to deal with multitrack media resources or media
> >> resources that have associated synchronized audio and video resources.
> >> I'm concretely referring to such things as audio descriptions, sign
> >> language video, and dubbed audio tracks.
> >>
> >> We require an API that can expose such extra tracks to the user and
> >> to JavaScript. This should be independent of whether the tracks are
> >> actually inside the media resource or are given as separate
> >> resources, but should be linked to the main media resource through
> markup.
> >>
> >> I am bringing this up now because solutions may have an influence on
> >> the inner workings of TimedTrack and the <track> element, so before
> >> we have any implementations of <track>, we should be very certain
> >> that we are happy with the way in which it works - in particular that
> >> <track> continues to stay an empty element.
> >>
> >> We've had some preliminary discussions about this in the W3C
> >> Accessibility Task Force and the alternatives that we could think
> >> about are captured in
> >> http://www.w3.org/WAI/PF/HTML/wiki/Media_Multitrack_Media_API .
> This
> >> may not be the complete list of possible solutions, but it provides ideas
> for the different approaches that can be taken.
> >>
> >> I'd like to see what people's opinions are about them.
> >>
> >> Note there are also discussion threads about this at the W3C both in
> >> the Accessibility TF [1] and the HTML Working Group [2], but I am
> >> curious about input from the wider community.
> >>
> >> So check out
> >> http://www.w3.org/WAI/PF/HTML/wiki/Media_Multitrack_Media_API
> >> and share your opinions.
> >>
> >> Cheers,
> >> Silvia.
> >>
> >> [1]
> >> http://lists.w3.org/Archives/Public/public-html-a11y/2011Feb/0057.htm
> >> l [2]
> >> http://lists.w3.org/Archives/Public/public-html/2011Feb/0205.html
> >