[whatwg] How to handle multitrack media resources in HTML

[Eric Winkelman]

On Friday, April 08, 2011, Ian Hickson wrote:
> On Thu, 17 Feb 2011, Eric Winkelman wrote:
> >
> > MPEG transport streams, as used for commercial TV, will often contain
> > multiple types of metadata: content advisories, ad insertion
> > opportunities, interactive TV application triggers, etc.  If we were
> > getting this information out-of-band we would, as you suggest, know how
> > to deal with it.  We would use multiple @kind=metadata tracks, with the
> > correct handler associated with each track.  In our case, however, this
> > information is all coming in-band.
> >
> > There is information within the MPEG transport stream that identifies
> > the types of metadata being carried.  This lets the video player know,
> > for example, that the stream has a particular track with application
> > triggers, and another one with content advisories.  To be consistent
> > with the out-of-band tracks, we envision the player creating separate
> > TimedTrack elements for each type of metadata, and adding the associated
> > data as cues.  But there isn't a clear way for the player to indicate
> > the type of metadata it's putting into each of these TimedTrack cues.
> >
> > Which brings us to the mime types.  I have an event handler on the
> > <video> tag that fires when the player creates a new metadata track, and
> > this handler tries to figure out what to do with the track.  Without a
> > type on the track, I have to set another handler on the track that fires
> > when the player creates a cue, and tries to figure out what to do from
> > the cue.  As there is no type on the cue either, I have to examine the
> > cue location/text to see if it contains metadata I'm able to handle.
> >
> > This all works, but it requires event handlers on tracks that may have
> > no interest to the application.  On the player side, it depends on the
> > player tagging the metadata in a consistent ad-hoc way, as well as
> > requiring the player to create separate metadata tracks.  (We also
> > considered starting the cue's text with a mime type, but this has the
> > same basic issues.)
> 
> This is an interesting problem.
> 
> What is the way that the MPEG streams identify these various metadata
> streams? Is it a MIME type? Some other identifier? Is this identifier
> separate from the track's label, or is it the track's label?

The streams contain a Program Map Table (PMT) which contains a list of tuples (program id (PID) and a standard numeric "type") for the program's tracks. This is how the user agent knows about this metadata and what is contained in it. We're envisioning that the combination of transport, e.g. MPEG-2 TS, and PMT "type" would be used by the UA to select a MIME type. We're proposing that this MIME type would be the track's "label". We think it would be better if there were a "type" attribute for the track to use instead of the "label", but using the "label" would work.

Thanks,
Eric
---
e.winkelman at cablelabs.com