[whatwg] HTML5 video seeking

[Ian Hickson]

On Mon, 14 Nov 2011, Aaron Colwell wrote:
> 
> I was looking at the seeking algorithm and had a question about step 10.
> 
># 10. Wait until the user agent has established whether or not the media 
># data for the new playback position is available, and, if it is, until 
># it has decoded enough data to play back that position.
> 
> Does this mean the user agent must resume playback at the exact location 
> specified?

Since you asked this question, the spec has been updated to explicitly 
support, in certain cases, approximate seeking. The answer, therefore, is 
now "yes, unless the approximate-for-speed flag is set". The flag gets set 
by the fastSeek() method, and optionally by native seek bars.


> What if the nearest keyframe is several seconds away? Is the UA expected 
> to decode and toss the frames instead of starting playback at the 
> nearest keyframe?

Yes.


> On desktop machines I don't think this would be a problem, but on mobile 
> devices it might be since the hardware may not be able to decode 
> significantly faster than realtime. What is the intended behavior for 
> such constrained devices?

Same as for desktop.

Having said that:

On Mon, 14 Nov 2011, Ralph Giles wrote:
> 
> Maybe you can muck with the 'media.seekable' TimeRanges object to only 
> show keyframes?

This would indeed be a plausible solution.


On Tue, 15 Nov 2011, Aaron Colwell wrote:
> 
> Yeah I thought about this, but it seems a little hacky and not what the 
> spec intended for seekable.

Well, it's really intended for cases where the format simply has no way to 
seek to the given position, but if you don't want to implement seeking to 
a particular position, then you don't have to, so long as you don't 
include it in seekable.

This is a quality of implementation feature. The idea UA would cache all 
incoming data and implement instantaneous arbitrary seeking and still 
miraculously use no memory, but real implementations have to make 
tradeoffs. The media element feature exposes a number of ways for authors 
to detect those tradeoffs; one of them is "seekable".


On Wed, 16 Nov 2011, Ralph Giles wrote:
> 
> Video splicing/mixing was a use case we wanted to support, and such 
> applications aren't really possible without frame-accurate seeking. 
> Thus, it's better to require it up front and possibly allow applications 
> to relax it later, as with Frank and Philip's 'strict' attribute, than 
> to disallow such applications by leaving this to implementation 
> variance.

Right, exactly. The seekable attribute can be used to indicate that this 
isn't supported, so that the author can detect such implementations, but I 
would indeed not expect authors to like such an implementation.


On Thu, 17 Nov 2011, Philip Jägenstedt wrote:
> On Wed, 16 Nov 2011 19:36:26 +0100, Frank Galligan 
> <fgalligan at google.com> wrote:
> > Here is another link where Ian mentions that the seek should be the 
> > exact frame. 
> > http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2008-November/017225.html 
> > This came up from 2 years ago, where I wanted to add an optional 
> > "strict" parameter to the seek call, which would default to false. 
> > Exactly the same as what Philip suggested. I'm all for it.
> > 
> > I think Chrome added frame accurate seeking in version 7.
> 
> I filed http://www.w3.org/Bugs/Public/show_bug.cgi?id=14851 to see if we 
> can make this happen.

This ended up resulting in fastSeek();


On Fri, 13 Jan 2012, Hugh Guiney wrote:
>
> I'm currently creating a custom HTML video player UI. What I'm trying to 
> achieve is a seek bar control—a horizontal slider that allows the user 
> to seek to a specific point in time. Right now the basic functionality 
> of this can be achieved using a range input with a max value set to the 
> video's duration, and which sets the video's currentTime onchange. 
> However, there are two problems with this:
> 
> The first is that I'd like to be able to show the user, before selecting 
> a point in the range, what point they are about to seek to. At present, 
> even if there is an element that exposes currentTime to the user, 
> there's no way to show them what currentTime *will* be until they 
> actually select a point on the range. Although I realize range inputs 
> are meant to be imprecise, it's often helpful to be able to see more 
> precise values one at a time, so that if there is a value they're aiming 
> for, they can find it or get close enough with less trial and error than 
> blind selection.
>
> It has become a ubiquitous UI convention on the Web to provide time code 
> and/or frame previews in a tooltip as the user scans the seek bar (but 
> before seeking). This can be seen for example on YouTube, Vimeo, Hulu, 
> blip.tv, Brightcove, Firefox, and many JS library players. And although 
> the effect is most common in video players, it would be helpful in audio 
> players, or any other situation where a user wants to browse available 
> options within a range without committing to any right away and 
> triggering an unwanted action.
>
> The second problem is that at present, range inputs are not quite
> sufficient semantically for marking up seek bars. Seek bars typically
> display not only a thumb, but also a progress bar that displays what
> percentage of the video is loaded. Obviously, we have <progress> for this,
> and so theoretically one could use both a range input and a progress bar
> together, this falls short for a few reasons: [...]

There's basically two ways we can address this:

1. Extend <input type=range> and <progress> to support more complicated 
UIs, e.g. including data in the range control to visualise buffering, 
playback, etc; having a way to provide a tooltip; having a way to have 
more careful positioning like YouTube's seek bar on long videos; having a 
way to get information about the position the user is hovering over before 
the user clicks; etc.

2. Have something like Web Components, enabling authors to create custom 
widgets that support whatever they want.

I expect we'll do both, on the long term. However, on the short term, I 
think we should delay #1 until we have much better implementations of the 
current widgets in all the major browsers. #2 is ongoing, but might take a 
long time also.

So in the meantime, you're kind of stuck without a good solution. Sorry. I 
would recommend going to the old standby, <div>s and lots of CSS, script, 
and ARIA attributes (this is what ARIA is useful for!).

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'