[whatwg] HTML5 video: frame accuracy / SMPTE

[Kevin Marks]

Returning to this discussion, I think it is lacking in use cases.

Consider the controllers we are used to - they tend to have frame step,
chapter step and some kind of scrub bar.

Frame stepping is used when you want to mark an accurate in or our point, or
catch a still frame. This needs to be accurate, and it is always local.

Chapter stepping means 'move me to the next meaningful break point in this
media. There is a very natural structure for this in almost all professional
media, and it is definitely worth getting this right. This is a long range
jump, but it is likely to be a key frame or start of new file segment.

Scrubbing is when you are dragging the bar back and forth to find a
particular point. It is intermediate in resolution between the previous two,
but it needs to be responsive to work - the lag between moving the bar and
showing something. In many cases decoding only key frames in this state
makes sense, as this is most responsive, and also likely to catch scene
boundaries, which are commonly key frames anyway.

The degenerate case of scrubbing is 'fast-forwarding', where the stream is
fetched faster than realtime, but again only keyframes are shown.

Are we sure all of these use cases are represented by the options mentioned
below?

On Mon, Jan 24, 2011 at 12:49 PM, Robert O'Callahan <robert at ocallahan.org>wrote:

> On Tue, Jan 25, 2011 at 9:34 AM, Philip Jägenstedt <philipj at opera.com
> >wrote:
>
> > On Mon, 24 Jan 2011 21:10:21 +0100, Robert O'Callahan <
> > robert at ocallahan.org> wrote:
> >
> >>
> >> Interesting. It doesn't in Firefox; script always sees a snapshot of a
> >> consistent state until it returns to the event loop or does something
> >> modal
> >> (although audio, and soon video, will continue to play while script
> runs).
> >> I'm not sure if the spec should require that ... overall our APIs try
> >> pretty
> >> hard not to expose races to JS.
> >>
> >
> > How does that work? Do you take a copy of all properties that could
> > possibly change during script execution, including ones that create a new
> > object, like buffered and seekable?
>
>
> All script-accessible state exists on the main thread (the thread that runs
> the event loop), and is updated via asynchronous messages from decoder and
> playback threads as necessary. 'buffered' is always in sync since data
> arrival and eviction from the media data cache happen on the main thread.
> (That cache can be read from other threads though.)
>
> If you instead only make a copy on the first read, isn't it still possible
> > to get an inconsistent state, e.g. where currentTime isn't in the
> buffered
> > ranges?
> >
>
> No, this wouldn't happen, although it might be possible for currentTime to
> be outside the buffered ranges for other reasons.
>
> How about HTMLImageElement.complete, which the spec explicitly says can
> > change during script execution?
> >
>
> Interesting, I didn't know about that.
>
> In any case, it sounds like either HTMLMediaElement is underspecified or
> one
> > of us has interpreted in incorrectly, some interop on this point would be
> > nice.
>
>
> Maybe. If the spec is clarified to allow races when accessing media element
> state, I guess it won't be the end of the world, although I predict interop
> difficulties. But that's always an easy prediction! :-)
>
> The biggest use case is clicking a seek bar and ending up somewhere close
> > enough, but yes, being able to do fast relative seeking is a nice bonus.
> > Maybe we should do what many media frameworks do and use a "reference"
> > parameter, defining what the seek is relative to. Usually you can seek
> > relative to the beginning, end and current position, but perhaps we could
> > reduce that to just "absolute" and "relative". That's a bit less magic
> than
> > inspecting currentTime when the method is called.
> >
> > So far:
> >
> > seek(t, ref, how);
> >
> > ref is "absolute" (default) or "relative"
> >
> > how is "accurate" (default) or "fast"
> >
> > (or numeric enums, if that's what DOM interfaces usually have)
>
>
> That works.
>
> Rob
> --
> "Now the Bereans were of more noble character than the Thessalonians, for
> they received the message with great eagerness and examined the Scriptures
> every day to see if what Paul said was true." [Acts 17:11]
>