[whatwg] <video> feedback

[Jer Noble]

On Sep 17, 2012, at 12:43 PM, Ian Hickson <ian at hixie.ch> wrote:

> On Mon, 9 Jul 2012, adam k wrote:
>> 
>> i have a 25fps video, h264, with a burned in timecode.  it seems to be 
>> off by 1 frame when i compare the burned in timecode to the calculated 
>> timecode.  i'm using rob coenen's test app at 
>> http://www.massive-interactive.nl/html5_video/smpte_test_universal.html 
>> to load my own video.
>> 
>> what's the process here to report issues?  please let me know whatever 
>> formal or informal steps are required and i'll gladly follow them.
> 
> Depends on the browser. Which browser?
> 
> 
>> i'm aware that crooked framerates (i.e. the notorious 29.97) were not 
>> supported when frame accuracy was implemented.  in my tests, 29.97DF 
>> timecodes were incorrect by 1 to 3 frames at any given point.
>> 
>> will there ever be support for crooked framerate accuracy?  i would be 
>> more than happy to contribute whatever i can to help test it and make it 
>> possible.  can someone comment on this?
> 
> This is a Quality of Implementation issue, basically. I believe there's 
> nothing inherently in the API that would make accuracy to such timecodes 
> impossible.

TLDR; for precise navigation, you need to use a a rational time class, rather than a float value.

The nature of floating point math makes precise frame navigation difficult, if not impossible.  Rob's test is especially hairy, given that each frame has a timing bound of [startTime, endTime), and his test attempts to navigate directly to the startTime of a given frame, a value which gives approximately zero room for error.

I'm most familiar with MPEG containers, but I believe the following is also true of the WebM container: times are represented by a rational number, timeValue / timeScale, where both numerator and denominator are unsigned integers.  To seek to a particular media time, we must convert a floating-point time value into this rational time format (e.g. when calculating the 4th frame's start time, from "3 * 1/29.97" to "3 * 1001/30000").  If there is a floating-point error in the wrong direction (e.g., as above, a numerator of 3002 vs 3003), the end result will not be the frame's startTime, but one timeScale before it. 

We've fixed some frame accuracy bugs in WebKit (and Chromium) by carefully rounding the incoming floating point time value, taking into account the media's time scale, and rounding to the nearest 1/timeScale value.  This fixes Rob's precision test, but at the expense of precision. (I.e. in a 30 fps movie, "currentTime = 0.999999 / 30" will navigate to the second frame, not the first, due to rounding, which is technically incorrect.)

This is a common problem, and Apple media frameworks (for example) therefore provide rational time classes which provide enough accuracy for precise navigation (e.g. QTTime, CMTime). Using a floating point number to represent time with any precision is not generally accepted as good practice when these rational time classes are available.

-Jer