[whatwg] <video> feedback

[Brian Campbell]

On Feb 10, 2010, at 1:37 PM, Eric Carlson wrote:

> 
> On Feb 10, 2010, at 8:01 AM, Brian Campbell wrote:
> 
>> On Feb 9, 2010, at 9:03 PM, Ian Hickson wrote:
>> 
>>> On Sat, 31 Oct 2009, Brian Campbell wrote:
>>>> 
>>>> At 4 timeupdate events per second, it isn't all that useful. I can 
>>>> replace it with setInterval, at whatever rate I want, query the time, 
>>>> and get the synchronization I need, but that makes the timeupdate event 
>>>> seem to be redundant.
>>> 
>>> The important thing with timeupdate is that it also fires whenever the 
>>> time changes in a significant way, e.g. immediately after a seek, or when 
>>> reaching the end of the resource, etc. Also, the user agent can start 
>>> lowering the rate in the face of high CPU load, which makes it more 
>>> user-friendly than setInterval().
>> 
>> I agree, it is important to be able to reduce the rate in the face of high CPU load, but as currently implemented in WebKit, if you use timeupdate to keep anything in sync with the video, it feels fairly laggy and jerky. This means that for higher quality synchronization, you need to use setInterval, which defeats the purpose of making timeupdate more user friendly.
>> 
>> Perhaps this is just a bug I should file to WebKit, as they are choosing an update interval at the extreme end of the allowed range for their default behavior; but I figured that it might make sense to mention a reasonable default value (such as 30 times per second, or once per frame displayed) in the spec, to give some guidance to browser vendors about what authors will be expecting.
>> 
> I disagree that 30 times per second is a "reasonable default". I understand that it would be useful for what you want to do, but your use case is not a typical. I think most pages won't listen for 'timeupdate' events at all so instead of making every page incur the extra overhead of waking up, allocating, queueing, and firing an event 30 times per second, WebKit sticks with  the minimum frequency the spec mandates figuring that people like you that need something more can roll their own.

Do browsers fire events for which there are no listeners? It seems like it would be easiest to just not fire these events if no one is listening to them.

And as Ian pointed out, just basic video UI can be better served by having at least 10 updates per second, if you want to show time at a resolution of tenths of a second.

>>> On Thu, 5 Nov 2009, Brian Campbell wrote:
>>>>> 
>>>>> Would something like <video> firing events for every frame rendered 
>>>>> help you out?  This would help also fix the <canvas> over/under 
>>>>> painting issue and improve synchronization.
>>>> 
>>>> Yes, this would be considerably better than what is currently specced.
>>> 
>>> There surely is a better solution than copying data from the <video> 
>>> element to a <canvas> on every frame for whatever the problem that that 
>>> solves is. What is the actual use case where you'd do that?
>> 
>> This was not my use case (my use case was just synchronizing bullets, slide transitions, and animations to video), but an example I can think of is using this to composite video. Most (if not all) video formats supported by <video> in the various browsers do not store alpha channel information. In order to composite video against a dynamic background, authors may copy video data to a canvas, then paint transparent to all pixels matching a given color.
>> 
>> This use case would clearly be better served by video formats that include alpha information, and implementations that support compositing video over other content, but given that we're having trouble finding any video format at all that the browsers can agree on, this seems to be a long way off, so stop-gap measures may be useful in the interim.
>> 
>> Compositing video over dynamic content is actually an extremely important use case for rich, interactive multimedia, which I would like to encourage browser vendors to implement, but I'm not even sure where to start, given the situation on formats and codecs. I believe I've seen this discussed in Theora, but never went anywhere, and I don't have any idea how I'd even start getting involved in the MPEG standardization process.
>> 
> Have you actually tried this? Rendering video frames to a canvas and processing every pixel from script is *extremely* processor intensive, you are unlikely to get reasonable frame rate. 

Mozilla has a demo of this working, in Firefox only:

https://developer.mozilla.org/samples/video/chroma-key/index.xhtml

But no, this isn't something I would consider to be production quality. But perhaps if the WebGL typed arrays catch on, and start being used in more places, you might be able to start doing this with reasonable performance.

> The H.262 does support alpha (see AVC spec 2nd edition, section 7.3.2.1.2 Sequence parameter set extension), but we do not support it correctly in WebKit at the moment. *Please* file bugs against WebKit if you would like to see this properly supported. QuickTime movies support alpha for a number of video formats (eg. png, animation, lossless, etc), you might give that a try.

Well, I'll give this a shot. This is clearly far preferable to rolling this functionality by hand in JS, so I would like to encourage it. This doesn't help for those browser which only support Theora, as I don't think there's a standardized way for doing this in Theora at the moment.

-- Brian