[whatwg] <video> feedback

[Brian Campbell]

On Feb 9, 2010, at 9:03 PM, Ian Hickson wrote:

> On Sat, 31 Oct 2009, Brian Campbell wrote:
>> 
>> As a multimedia developer, I am wondering about the purpose of the timeupdate
>> event on media elements.
> 
> It's primary use is keeping the UIs updated (specifically the timers and 
> the scrubber bars).
> 
> 
>> On first glance, it would appear that this event would be useful for 
>> synchronizing animations, bullets, captions, UI, and the like.
> 
> Synchronising accompanying slides and animations won't work that well with 
> an event, since you can't guarantee the timing of the event or anything 
> like that. For anything where we want reliable synchronisation of multiple 
> media, I think we need a more serious solution -- either something like 
> SMIL, or the SMIL subset found in SVG, or some other solution.

Yes, but that doesn't exist at the moment, so our current choices are to use timeupdate and to use setInterval().

>> At 4 timeupdate events per second, it isn't all that useful. I can 
>> replace it with setInterval, at whatever rate I want, query the time, 
>> and get the synchronization I need, but that makes the timeupdate event 
>> seem to be redundant.
> 
> The important thing with timeupdate is that it also fires whenever the 
> time changes in a significant way, e.g. immediately after a seek, or when 
> reaching the end of the resource, etc. Also, the user agent can start 
> lowering the rate in the face of high CPU load, which makes it more 
> user-friendly than setInterval().

I agree, it is important to be able to reduce the rate in the face of high CPU load, but as currently implemented in WebKit, if you use timeupdate to keep anything in sync with the video, it feels fairly laggy and jerky. This means that for higher quality synchronization, you need to use setInterval, which defeats the purpose of making timeupdate more user friendly.

Perhaps this is just a bug I should file to WebKit, as they are choosing an update interval at the extreme end of the allowed range for their default behavior; but I figured that it might make sense to mention a reasonable default value (such as 30 times per second, or once per frame displayed) in the spec, to give some guidance to browser vendors about what authors will be expecting.

> On Thu, 5 Nov 2009, Brian Campbell wrote:
>>> 
>>> Would something like <video> firing events for every frame rendered 
>>> help you out?  This would help also fix the <canvas> over/under 
>>> painting issue and improve synchronization.
>> 
>> Yes, this would be considerably better than what is currently specced.
> 
> There surely is a better solution than copying data from the <video> 
> element to a <canvas> on every frame for whatever the problem that that 
> solves is. What is the actual use case where you'd do that?

This was not my use case (my use case was just synchronizing bullets, slide transitions, and animations to video), but an example I can think of is using this to composite video. Most (if not all) video formats supported by <video> in the various browsers do not store alpha channel information. In order to composite video against a dynamic background, authors may copy video data to a canvas, then paint transparent to all pixels matching a given color.

This use case would clearly be better served by video formats that include alpha information, and implementations that support compositing video over other content, but given that we're having trouble finding any video format at all that the browsers can agree on, this seems to be a long way off, so stop-gap measures may be useful in the interim.

Compositing video over dynamic content is actually an extremely important use case for rich, interactive multimedia, which I would like to encourage browser vendors to implement, but I'm not even sure where to start, given the situation on formats and codecs. I believe I've seen this discussed in Theora, but never went anywhere, and I don't have any idea how I'd even start getting involved in the MPEG standardization process.

> On Thu, 5 Nov 2009, Andrew Scherkus wrote:
>> 
>> I'll see if we can do something for WebKit based browsers, because today 
>> it literally is hardcoded to 250ms for all ports. 
>> http://trac.webkit.org/browser/trunk/WebCore/html/HTMLMediaElement.cpp#L1254
>> 
>> Maybe we'll end up firing events based on frame updates for video, and 
>> something arbitrary for audio (as it is today).
> 
> I strongly recommend making the ontimeupdate rate be sensitive to system 
> load, and no faster than one frame per second.

I'm assuming that you mean "no faster than once per frame"?

> On Fri, 6 Nov 2009, Philip Jägenstedt wrote:
>> 
>> We've considered firing it for each frame, but there is one problem. If 
>> people expect that it fires once per frame they will probably write 
>> scripts which do frame-based animations by moving things n pixels per 
>> frame or similar. Some animations are just easier to do this way, so 
>> there's no reason to think that people won't do it. This will break 
>> horribly if a browser is ever forced to drop a frame, which is going to 
>> happen on slower machines. In balance this may or may not be a risk 
>> worth taking.
> 
> I strongly agree with this.

Anyone who depends on a certain number of events being fired instead of paying attention to the actual time, whether via timeupdate or setInterval or any other method, will be burnt by dropped frames, GC pauses, or any other form of system slowdown. I'm not sure that making timeupdate less useful by not updating often enough will really help people avoid bugs.

> On Sat, 7 Nov 2009, Jonas Sicking wrote:
>> 
>> When timeupdate was added, the stated goal was actually as a battery 
>> saving feature for for example mobile devices. The idea was that the 
>> implementation could scale back how often it fired the event in order to 
>> save battery.
> 
> Indeed.
> 
>> Now that we have implementation experience, is timeupdate fulfilling 
>> this goal? If not, is it fulfilling any other goals making it worth 
>> keeping?

I think that it can only work for this purpose if it is good enough that people will actually use it. If an author gets much better results with setInterval than with timeupdate (or a combination that feeds off of both, to get good results when scrubbing and in normal playback), she will likely use that approach and defeat the purpose of scaling based on CPU load (unless, of course, setInterval does the same thing).

> On Sat, 7 Nov 2009, Justin Dolske wrote:
>> 
>> FWIW, I felt that having Firefox's default video controls update their 
>> state for every frame was excessive (and could lead to competing for the 
>> CPU with the video itself). So, the controls basically ignore timeupdate 
>> events that occur within .333 seconds of the last timeupdate position... 
>> Which leads to having a bit of complication to deal with edge cases like 
>> having the video end less than .333 seconds after the last timeupdate 
>> event (otherwise the UI might look like stuck shortly before the end of 
>> the video).
>> 
>> At least for my needs, having an event fire at ~3 Hz (and when special 
>> things happen, like a seek or the video ending) would be somewhat 
>> simpler and more efficient.
> 
> 3Hz seems a little slow for the timer -- you'd want at least 10Hz so you 
> can show a tenths-of-a-second timer. More than that seems pointless 
> though.

You don't get 10Hz with the current timeupdate, you get 4Hz.

> On Sat, 7 Nov 2009, Silvia Pfeiffer wrote:
>> 
>> I use timeupdate to register a callback that will update 
>> captions/subtitles.
> 
> That's only a temporary situation, though, so it shouldn't inform our 
> decision. We should in due course develop much better solutions for 
> captions and time-synchronised animations.

The problem is, due to the slow pace of standards and browser development, we can sometimes be stuck with a temporary feature for many years. How long until enough IE users support HTML6 (or whatever standard includes a time-synchronization feature) for it to be usable? 10, 15 years?

> On Fri, 6 Nov 2009, Brian Campbell wrote:
>> 
>> Our major use case is actually synchronizing bullets, slide changes, and 
>> the like with video, in educational multimedia produced with high 
>> production values.
> 
> For this timeupdate is terrible. You need something like the old cuerange 
> interface, and we'll introduce something for this in the next version for 
> sure, along with captions support. All we're waiting for is for 
> implementations to be of high enough quality that the existing spec can be 
> reliably used by authors.

Yes, you're absolutely right, something like the previously suggested cuerange interface would be much better. I'm interested in developing content before that is specced, implemented, and widely enough available in browsers to depend upon, however. I was hoping to use timeupdate for this purpose, but had to fall back to setInterval to get a good experience.

> On Mon, 1 Feb 2010, Brian Campbell wrote:
>> 
>> I think it would be best to immediately go as full screen as possible 
>> (so, full window if permission hasn't yet been given), and then resize 
>> to full screen if permission is granted. This will avoid content authors 
>> having to duplicate that same functionality themselves for their users 
>> that don't ever give or deny permission.
> 
> We can do that with an API that just does page-wide fullscreen -- when the 
> page requests fullscreen mode, it makes the relevant bit take the full 
> width of the page, and then only if the user agrees to fullscreen does the 
> window actually go fullscreen.

Yep, that's what I'm looking for.

>> Resizing when in full screen mode will need to be implemented anyhow, to 
>> support devices like the iPhone or iPad which can change orientation and 
>> will need to reshape the screen.
> 
> Indeed. Generally this is free (CSS will just handle it automatically).

Right, though sometimes you need to do modifications to your interface that CSS can't handle. This should be covered by the element resize event work that is ongoing.

>> No, you can't stop someone who is truly dedicated from guessing based on 
>> the exact size. My concern is more with authors who feel that their 
>> content is best displayed in full screen, and so may simply refuse to 
>> play it until they've gotten the fullscreen event or have the fullscreen 
>> pseudoclass. That would be pretty easy to implement, if you have that 
>> functionality available to you. I know my previous director would have 
>> requested it; he is very particular about content being displayed in 
>> full screen, and while I would argue that we shouldn't lock people out 
>> who don't want to be in full screen mode, I may have been overruled if 
>> such functionality were available and so easy to use.
> 
> Yeah... it might be ok to have only the "exit full screen" event and have 
> it trigger just when the user declines or exits? That way if the user does 
> nothing, the page can't know, and it'll just render "full window" rather 
> than "full screen".

Yeah, that might be reasonable. I suppose if you always enter a full screen like mode when the API is called, you don't need the event for entering fullscreen. Would you want it anyhow for cases in which the user enters full screen by using the existing browser UI, or is that going to be considered an entirely different mode?

-- Brian