[whatwg] Video feedback

[Silvia Pfeiffer]

On Mon, Jun 20, 2011 at 7:31 PM, Mark Watson <watsonm at netflix.com> wrote:
>>
>>> The TrackList object has an onchanged event, which I assumed would fire when
>>> any of the information in the TrackList changes (e.g. tracks added or
>>> removed). But actually the spec doesn't state when this event fires (as far
>>> as I could tell - unless it is implied by some general definition of events
>>> called onchanged).
>>>
>>> Should there be some clarification here ?
>>
>> I understood that to relate to a change of cues only, since it is on
>> the tracklist. I.e. it's an aggregate event from the oncuechange event
>> of a cue inside the track. I didn't think it would relate to a change
>> of existence of that track.
>>
>> Note that the even is attached to the TrackList, not the TrackList[],
>> so it cannot be raised when a track is added or removed, only when
>> something inside the TrackList changes.
>
> Are we talking about the same thing ? There is no TrackList array and
> TrackList is only used for audio/video, not text, so I don't understand the
> comment about cues.
> I'm talking
> about http://www.whatwg.org/specs/web-apps/current-work/multipage/the-iframe-element.html#tracklist which
> is the base class for MultipleTrackList and ExclusiveTrackList used to
> represent all the audio and video tracks (respectively). One instance of the
> object represents all the tracks, so I would assume that a change in the
> number of tracks is a change to this object.

Ah yes, you're right: I got confused.

It says "Whenever the selected track is changed, the user agent must
queue a task to fire a simple event named change at the
MultipleTrackList object." This means it fires when the selectedIndex
is changed, i.e. the user chooses a different track for rendering. I
still don't think it relates to changes in the composition of tracks
of a resource. That should be something different and should probably
be on the MediaElement and not on the track list to also cover changes
in text tracks.


>>> Also, as Eric (C) pointed out, one of the things which can change is which
>>> of several available versions of the content is being rendered (for adaptive
>>> bitrate cases). This doesn't necessarily change any of the metadata
>>> currently exposed on the video element, but nevertheless it's information
>>> that the application may need. It would be nice to expose some kind of
>>> identifier for the currently rendered stream and have an event when this
>>> changes. I think that a stream-format-supplied identifier would be
>>> sufficient.
>>
>> I don't know about the adaptive streaming situation. I think that is
>> more about statistics/metrics rather than about change of resource.
>> All the alternatives in an adaptive streaming "resource" should
>> provide the same number of tracks and the same video dimensions, just
>> at different bitrate/quality, no?
>
> I think of the different adaptive versions on a per-track basis (i.e. the
> alternatives are *within* each track), not a bunch of alternatives each of
> which contains several tracks. Both are possible, of course.
>
> It's certainly possible (indeed common) for different bitrate video
> encodings to have different resolutions - there are video encoding reasons
> to do this. Of course the aspect ratio should not change and nor should the
> dimensions on the screen (both would be a little peculiar for the user).
>
> Now, the videoWidth and videoHeight attributes of HTMLVideoElement are not
> the same as the resolution (for a start, they are in CSS pixels, which are
> square), but I think it quite likely that if the resolution of the video
> changes than the videoWidth and videoHeight might change. I'd be interested
> to hear how existing implementations relate resolution to videoWidth and
> videoHeight.

Well, if videoWidth and videoHeight change and no dimensions on the
video are provided through CSS, then surely the video will change size
and the display will shrink. That would be a terrible user experience.
For that reason I would suggest that such a change not be made in
alternative adaptive streams.


>> Different video dimensions should be
>> provided through the <source> element and @media attribute, but within
>> an adaptive stream, the alternatives should be consistent because the
>> target device won't change. I guess this is a discussion for another
>> thread... :-)
>
> Possibly ;-) The device knows much better than the page author what
> capabilities it has and so what resolutions are suitable for the device. So
> it is better to provide all the alternatives as a single resource and have
> the device work out which subset it can support. Or at least, the list
> should be provided all at the same level - there is no rationale for a
> hierarchy of alternatives.

The way in which HTML deals with different devices and their different
capabilities is through media queries. As a author you provide your
content with different versions of media-dependent style sheets and
content, so that when you view the page with a different device, the
capabilities of the device select the right style sheet and content
for display on that device. Opera has an example on how to use this
here: http://dev.opera.com/articles/view/everything-you-need-to-know-about-html5-video-and-audio/
(search for "Media Query").

I believe that this mechanism should also work for adaptive streaming,
such that you provide multiple alternative media resources through the
<source> element, each of which has a @media attribute that says what
device capabilities that particular resource is adequate for. Except
that the "media resource" provides alternative bitrate files for that
case. I do not see a need to move this functionality into the adaptive
streaming file.

Nice to get started on this discussion about adaptive streaming. ;-)


Cheers,
Silvia.