[whatwg] HTML5 video: frame accuracy / SMPTE

[Rob Coenen]

This is a very interesting observation, I've been thinking what this means
in terms of UI design for scrubbars/players and there's a design limitation
with (traditional) seekbars:
Seekbars tend to have a physical length in pixels, meaning that there's a
physical limitation how precise one can move the scrubbar.
I'd think that only when the amount of frames of a movie is lower or equal
to than the amount of pixels on the scrubbar one can do a frame-accurate
seek using that scrubbar.
(note: the only non-traditional example I  know of is Apple's gesture based
scrubbing on ipad/iPhone where one can move the finger that drags the
scrubbar up/down to change the precision of seeking. However, it does still
not allow to get to frame-accurate level in most cases.)

In all other cases, and that's probably every movie longer than a few
seconds, the scrubbar will physically limit the user to perform any such
thing as frame accurate seeking. This will always result in a situation
where 'frame accurate seeking' is not required.

so, it's indeed at API level where we really wan to to have frame accurate
seeking and the default there should then ofcourse be 'yes, seek frame
accurate, unless specified otherwise'

-Rob

On Sat, Jan 22, 2011 at 5:18 PM, Michael Dale <dale at ucsc.edu> wrote:

> It seems the needs for a seek bar are different from the api usage of
> setting video.currentTime. As people mention rules of 'least surprise'
> are important. If you set currentTime to 3.453 you expect it to return
> something very close to 3.453 ..
>
> Why not have a separate api for "seek bars" that just seek to a
> percentage of the video stream. Letting implementations do what's
> fastest and consistent, but without guarantees for a given play time
> being set. This way the different needs are addressed and we don't
> conflate the currentTime api with parameters for accuracy.
>
> In terms of real world use cases, we use accurate seeks in an html5
> video editor. Its especially important when flattening with firefogg
> that as we can accurately set a given time in source videos per our
> output frame interval.
> http://commons.wikimedia.org/wiki/Commons:Sequencer
>
> --michael
>
>
> On 01/22/2011 03:41 AM, Philip Jägenstedt wrote:
> > On Sat, 22 Jan 2011 02:19:10 +0100, Chris Pearce <chris at pearce.org.nz>
> > wrote:
> >
> >>
> >> On 22/01/2011 7:31 a.m., Gregory Maxwell wrote:
> >>> It's usually the decoding, not the file access that kill you.  Firefox
> >>> seeking on low resolution clips is very snappy index or not. But try a
> >>> 1080p clip encoded with a 10 second maximum keyframe interval...
> >>
> >> This is true. If you want fast frame accurate seeking, particularly
> >> over the internet, it's best to not encode with such a large keyframe
> >> interval. This is a problem caused by a webdev's inappropriate
> >> encoding choice, not by a bad API choice.
> >>
> >> If seeking is slow when you encode with a large keyframe interval,
> >> don't encode with a large keyframe interval!
> >>
> >> What if the browser's controls by default seeked to the previous
> >> keyframe, if the user had enough precision in the controls and wanted
> >> to seek to 9 seconds after a keyframe (1 second before the subsequent
> >> keyframe), then they could not.
> >
> > Simple heuristics could solve this:
> >
> > HTMLMediaElement.prototype.magicSeek = function(t) {
> >   if (Math.abs(t - this.currentTime) < 10)
> >     this.seek(t, "accurate");
> >   else
> >     this.seek(t, "fast");
> > }
> >
> > Or some refinement thereof...
> >
> >> On 22/01/2011 10:04 a.m., Philip Jägenstedt wrote:
> >>> Since, as you say, the behavior is currently inconsistent, there is
> >>> still time to agree on something that makes sense and have everyone
> >>> implement that. I think the best default is keyframe seeking and
> >>> haven't seen any strong arguments for accurate seeking as the
> >>> default yet.
> >>
> >> I disagree. The default should be exact, with "approximate" seeking
> >> to the nearest keyframe.  When you call video.seek(X.xx), you've
> >> specified an exact time, and would likely expect an exact time, so
> >> the media should seek to that exact time. Another reason to make
> >> exact the default, is that if the media is seekable, it can always be
> >> seeked exactly, whereas media without an index may not be able to be
> >> seeked approximately.
> >
> > Nitpick: If you can't seek approximately then simply not doing it
> > would be fine, there's no requirement for a fast seek to always miss
> > the target.
> >
> >> We've already implemented frame accurate seeking in Firefox. I'd be
> >> happy for us to implement approximate seeking, it would be useful for
> >> seeking into areas of the media which are unbuffered. We may change
> >> our controls to use approximate seeking when seeking into unbuffered
> >> areas, but we wouldn't use approximate seeking when seeking into
> >> buffered areas. Videos are usually short enough that you want
> >> accuracy higher than keyframe granularity when seeking using the
> >> default controls (at least in buffered areas).
> >
> > OK, having the feature is more important than the default.
> >
> >> On 22/01/2011 10:50 a.m., Roger Hågensen wrote:
> >>> Hmm. I think the default (nothing) should be synonymous with
> >>> "best-effort" (or "best") and leave it to the
> >>> browser/os/codec/format/etc. as to what "best effort" actually is.
> >>
> >> We should specify the default, otherwise by default webdevs will
> >> always need to specify their seek accuracy level.
> >
> > Indeed.
> >
> >> On 22/01/2011 11:25 a.m., Silvia Pfeiffer wrote:
> >>>>> * KEYFRAME is keyframe-accurate seeking, so to the previous keyframe
> >>>> What does this mean when a seekable stream doesn't have interior
> >>>> keyframes? Should the client always seek to the beginning? Why is this
> >>>> valuable over a "fast" option?
> >>> Where no keyframes are available, this seek option simply doesn't do
> >>> anything, since obviously there are not keyframes. The point is that
> >>> where this concept exists and people want to take advantage of it,
> >>> this seek should be possible.
> >>>
> >>
> >> Some media format's index for audio as well, so where there are no
> >> keyframes, you could seek to the nearest indexed point from the
> >> index. Or (probably a better idea) have some way of reporting whether
> >> keyframe level approximate seeking is available.
> >>
> >> Exact seeking is always available if the media is seekable (so it
> >> makes a good default...), and script can determine whether faster but
> >> approximate seeking is available and can choose to use it.
> >
> > Do we really need to make this feature testable? Why not just let them
> > request fast seeking and give them what's possible?
> >
>
>