[whatwg] Video : Slow motion, fast forward effects

[Ian Hickson]

Regarding whether to play audio while rewinding:

Leaving this be something that is optional doesn't make sense to me. If we 
want interoperability, we need to either have all the browsers play sound 
while moving backwards through a sound or video file, or not play sound. 
Leaving things optional means that one author will test with one browser 
and expect one behavior, and then when the user uses that page on another 
browser, he will get unexpected behavior.

For example, if Alice uses Browser A, and finds that when rewinding the 
sound isn't played, and thus does a hack that fakes the sound playback by 
having a separate hidden <video> element while rewinding, in which she 
seeks, plays forward for a bit, seeks, plays, etc, then when Bob loads 
that page in Browser B, which does play audio when rewinding, he'll get 
a horrible double audio effect.

Or similarly, if Alice, in an attempt to get a cool effect, plays a bunch 
of videos backwards, and when the user selects one, plays it forward, then 
Bob will find that all the audio from all the videos plays simultaneously 
instead of all being muted except the video that Alice is expecting to 
play.

So. We have to either require that backwards-playing audio be muted, or 
require that it play. I don't mind which. I've required that it be muted 
because implementors said playing would be hard.


On Tue, 14 Oct 2008, Dave Singer wrote:
> > > 
> > > I think that the spec. should say that degraded playback (e.g. I 
> > > frames only) or no playback (non-reversed audio) may occur...
> > 
> > I think that's a quality of implementation issue, I don't really see 
> > what the spec can say about it.
> 
> It's a warning to authors and a permission to developers;  a helpful 
> note, that's all.

How would you phrase it?


> > > I'm sorry if I wasn't clear:  I agree.  If you want your 
> > > implementation to shine, or be used heavily for audio scrubbing, or 
> > > something, go ahead and implement But it should not be required. 
> > > ("For extra credit")
> > 
> > We don't want some to do it and some not to do it, because then we get 
> > all kinds of interoperability problems (e.g. someone who hides his 
> > video and rewinds it at a particular rate for some reason or other, 
> > and doesn't hear audio in one UA, deploys, and finds his users get 
> > audio on another UA).
> 
> I don't think I agree.  I can't think why you would do such a rewind 
> (media files are not usually that much like physical tape), and if you 
> want to do it and be muted, you should mute.

You are applying logic to author behavior, which we have repeatedly seen 
doesn't work in reality. Authors act irrationally, we have to plan around 
that.


> The spec. can be perfectly clear that at non-unitary playback rates, 
> including negative ones, media may be presented degraded or, 
> particularly for audio, not at all.

This isn't going to stop authors from relying on broken behavior.


> If muting is needed, then it should be explicitly requested.

We can get this by requiring that the user agent play _some_ audio, even 
if severely degraded, at all speeds including negative speeds. I'm happy 
to require that if people are ok with it.


> Working around this prohibition for the case of an implementation that 
> could do it and a site that *wants* audio scrubbing, would be a pain in 
> the neck.

I agree that it would be good to provide audio scrubbing, and not require 
authors to implement it themselves. Are UAs willing to support this?


On Tue, 14 Oct 2008, Eric Carlson wrote:
>
> Some media formats and/or engines may not support reverse playback, but 
> I think it is a mistake for the spec to mandate this behavior. Why is 
> reverse playback different from other situations described in the spec 
> where different UAs/ media formats will result in different behavior, 
> eg. pitch adjusted audio, negotiation with a server to achieve the 
> appropriate playback rate, etc?

Because muted vs not muted is a feature difference, whereas pitch 
adjustment is a quality difference. Authors will rely on feature 
differences, they won't generally rely on quality differences. (Though 
they might, and we might end up having to spec the pitch differences more 
precisely as well.)

Negotiation with a server to achieve the appropriate playback rate is 
something that shouldn't be optional nor left to the UA, but it is out of 
scope for this spec as we don't define the network protocol.


> I think the current sentence that talks about audio playback rate:
> 
> When the playbackRate is so low or so high that the user agent cannot 
> play audio usefully, the corresponding audio must not play.
> 
> could be modified to include reverse playback as well:
> 
> When the playbackRate is such that the user agent cannot play audio 
> usefully (eg. too low, too high, negative when the format or engine does 
> not support reverse playback), the corresponding audio must not play.

For the reasons described above, I'd rather change this to say that the 
audio _must_ play _despite_ the problems, and that authors should set 
muted to false if they don't actually want the audio. That way we won't 
end up inadvertently setting authors up for relying on a particular 
de-facto standard behavior.


On Tue, 14 Oct 2008, Peter Kasting wrote:
> 
> Mandating silence during reverse playback seems bizarre in the abstract, 
> unnecessary if authors have a way to mute, and potentially detrimental 
> to future applications which may _want_ to be able to do this in a 
> controlled fashion (e.g. a virtual turntable application).

On Wed, 15 Oct 2008, Silvia Pfeiffer wrote:
> 
> Also in the case of a caption/subtitle authoring application: being able 
> to scrub through the audio forwards/backwards is a very important means 
> for captioners to identify the time points to start/end a caption 
> segment. I disagree with mandating silence and would rather deal with an 
> intermediate inconsistent browser landscape for lack of the audio 
> scrubbing feature in some browsers.

I don't think anyone is disagreeing on the importance of allowing this. 
The question is whether we should require it to be supported today, or add 
an explicit attribute to support this later.


On Tue, 14 Oct 2008, WeBMartians wrote:
>
> Yes, one must ALWAYS remember NEVER to use words like "always" and 
> "never" ... and "required" and "forbidden" and... 

We're writing a spec. Requiring and forbidding things is the name of the 
game.

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'