[whatwg] When to stop <video> elements from playing

[Ian Hickson]

On Fri, 19 Oct 2007, Robert O'Callahan wrote:
>
> Suppose a script creates a <video> element, adds it to the document, 
> starts it playing, then removes the element from the document and drops 
> all references to it. When should the element stop playing?

I've made the spec say that it stops playing when you remove it from the 
document. However, that doesn't really mean much, it can still play when 
it's out of the document.


> -- when all JS references to the element have been dropped (and garbage
> collection runs)?
> -- when the user leaves the page?
> -- when the end of the stream is reached? (which could be "never")

I've explicitly addressed all of these in the spec.


On Thu, 18 Oct 2007, Maciej Stachowiak wrote:
> 
> However, you'd probably want <audio> elements to be able to play outside 
> the document.

Indeed, that's an important use case.


On Fri, 19 Oct 2007, timeless wrote:
> 
> I can easily have a bookmarklet that goes through the dom tree and stops 
> (or stops and deletes) all audio elements. I can't have such a 
> bookmarklet for things that are only reachable via JS scope (and I don't 
> mean JS object graph, I mean function call chains, since that's the best 
> way to be evil, site's don't need to let people have access via object 
> properties).

Bookmarklets are effectively browser-specific and thus interoperability 
isn't a concern for them. They can use UA-specific extensions if needed.


On Fri, 19 Oct 2007, timeless wrote:
> On 10/19/07, Anne van Kesteren <annevk at opera.com> wrote:
> >
> > This doesn't seem like something a typical end user would do.
> 
> the how perhaps not, but i think the goal is.

If a user wants to mute all audio, a "mute" key on the keyboard or in the 
UA's chrome seems like the more likely solution, both of which are 
possible within the constraints of the spec.


On Sat, 20 Oct 2007, dev wrote:
>
> As a user , I would really prefer that the no Sound comes from the 
> browser, unless and until I can see a play/pause button inside the page. 
> So if the element is removed the sound and/or the video should stop.

This is what the spec now requires.


On Fri, 19 Oct 2007, Robert O'Callahan wrote:
>
> A related question is whether display:none audio and video elements 
> should produce sound.

Yes. The sound from <audio> and <video> elements isn't stylistic; it's 
asynchronously played content, and thus not affected by CSS properties.


> My current opinion is that <audio> and <video> elements should behave 
> like <img> and load/play whether or not they're in a document, but they 
> should only emit sound if they're in a document with a presentation and 
> are not in a display:none subtree. Then to play a sound you'd have to 
> insert the element into your document somewhere with size 0x0 (which 
> should be the default for <audio>).
> 
> But it's a rather tangled issue.

Having to insert audio elements in the tree to have them play seems 
artificial, especially since they wouldn't play where you put them -- the 
tree position would be basically meaningless. A typical use case is the:

   var a = new Audio('fx.wav');
   a.autoplay = true;

...or similar.


On Fri, 19 Oct 2007, Anne van Kesteren wrote:
>
> I think you want to do:
> 
>   var soundeffect = new Audio("sound.wav")
>   soundeffect.onload = function() { this.play() }
> 
> which is what was possible with the old Audio API (became <audio>) Opera 
> implemented.

Right, that too.


On Fri, 19 Oct 2007, Robert O'Callahan wrote:
>
> When would the Opera stop playing that sound? When the user left the 
> page? Only when the sound finished?

Per spec, at the earliest of the two. (Though in the former case, if the 
user returned to the page and the bfcache hadn't flushed the message away, 
the sound would resume playing.)


On Fri, 19 Oct 2007, Jonas Sicking wrote:
> > 
> > No. The time at which garbage collection will reclaim an object is 
> > unpredictable, so relying on garbage collection for behavior is a bad 
> > idea.
> > 
> > Agreed.
> 
> If we let <video>s or <audio>s play even when out of the document I 
> don't see how we could avoid this.

The spec requires audible elements not to be garbage collected.


> Or do you mean by 'play' only move forward in their time-position, not 
> emit sound?

Sound emission.


> What happens in opera is if you do:
> 
> myAudio = new Audio("foo.wav");
> myAudio.onload = function () {
>   this.play();
>   myAudio = null;
> }
> 
> When will the audio stop playing?

Per spec, the element will play through to the end (or until the browsing 
context is navigated away).


On Fri, 19 Oct 2007, Jonas Sicking wrote:
> 
> Opps, forgot to answer this one. I strongly feel that both <audio> and 
> <video> should make sound for display:none elements. The CSS display 
> property should only affect the visual appearance of the node.
> 
> Additionally, it's more efficient for us, and simpler for the author, if 
> hidden-but-played audio/video elements don't create a frame and don't 
> affect layout.

Agreed.


On Fri, 19 Oct 2007, fantasai wrote:
> > 
> > A related question is whether display:none audio and video elements 
> > should produce sound.
> 
> No. "display: none" is defined to affect all media, and that certainly 
> should not change for <audio> and <video>.

No, 'display:none' affects the render tree, of which the audio component 
is not part. The video component would get removed, but the audio 
component plays asynchronously and thus there is nothing to remove when 
you do 'display:none'.


On Tue, 23 Oct 2007, Jonas Sicking wrote:
> 
> I think this is different than screen readers not speaking display:none 
> text. Both hiding layout frames and silencing screen readers only affect 
> the 'rendering' of the contained text, it doesn't otherwise deactivate 
> the contained display:none elements:
> 
> <style> elements that are display:none still have their stylesheets 
> applied to the document.
> <script> elements still execute.
> Formcontrols are still submitted.
> <iframe>s are still loaded.
> 
> I think these examples are much more similar to the <video> element and 
> its produced sound.

Agreed.


On Thu, 25 Oct 2007, Jonas Sicking wrote:
> 
> Yes, so I think that a screen reader should not say whatever it would 
> normally say when hitting a <video> that is display:none.
> 
> However I see that as very different from silencing the audio stream 
> coming from the video. The audio stream in a video is not what the 
> screen reader would normally say when encountering a <video>, so I don't 
> think that neither the voice-volume nor the display property should 
> affect it.

Agreed.


On Fri, 19 Oct 2007, Maciej Stachowiak wrote:
> 
> I think it would be ok to do a one-time stop of a playing <video> and 
> <audio> element at the time it is removed from the document, but still 
> allow at least <audio> (and maybe <video> too for consistency, though it 
> seems gratuitous) to play play if is outside the document already when 
> it starts. In any case, like <img>, I think loading should be able to 
> proceed when outside the document or set to display: none.

Agreed, and made so.


> Stopping at remove time ensures that sounds and videos end at a 
> deterministic time when parts of the document are replaced, say with 
> innerHTML. Without this, if you replace a piece of the document that may 
> contain <video> or <audio> elements you will have to search for them 
> first and make sure to stop them. However, if the element can still 
> start to play when outside the document, that addresses the use case for 
> programmatic audio for things like game sound effects.

Good call.


On Fri, 19 Oct 2007, Jonas Sicking wrote:
> 
> Yeah, it makes a lot of sense to me to stop playing when a node is moved 
> out of the document. I'd even say that it should never autostart based 
> on its DOM attributes unless it is in the document. In other words, the 
> only way you could get it to play when outside the document would be 
> through an explicit call to play().

Well, the load won't start unless you do one of the following:

   * insert a media element (<audio> or <video>) into a document
     while it has a src= or <source>

   * set a src= on a media element already in a document

   * add a <source> to a media element already in a document

   * you call .load() directly

   * you use the new Audio(url) constructor and specify a URL.

...and nothing plays until it's loaded (and it won't start playing unless 
you've set autoplay, even then).

So this seems covered.


> This does leave the problem of what happens if the element is garbage 
> collected while it's playing sound. It'd be nice to hear what experience 
> opera has here.

The spec requires that it not be.


> I would be a little hesitant to say that elements that play are never 
> garbage collected. That would make it easy for the page author to 
> accidentally leak <video>s without sound and muted-but-playing 
> <audio>/<video>s for the duration of the page. This sucks up both CPU 
> cycles and memory.

The spec allows the optimisation that videos/audio be garbage collected 
before they end, if they are silent.


On Sat, 20 Oct 2007, Robert O'Callahan wrote:
>
> After some discussion on IRC, I think Jonas, Hixie and I agreed that 
> we're OK with the following approach:
> -- sound is produced for display:none elements and elements not in the 
> DOM
> -- removing an element from the DOM automatically calls stop() on that 
> element
> -- tearing down the owner document always stops the element playing (so 
> navigating away from the page always stops sound)

Agreed (s/stop/pause/, and actually not invoking the method directly for 
various reasons but the effect is almost identical).


On Fri, 19 Oct 2007, Jonas Sicking wrote:
> 
> This also brings up the point of what should happen for <video> elements 
> living in documents created through XMLHttpRequest. Currently firefox 
> doesn't do any network loads, not even for images, from such documents. 
> I would argue that that should be the case here too.

Such documents aren't ever in browsing contexts, and thus can't ever be 
active documents, and thus can never advance their playback position, per 
the spec (they can still end up loading the contents, currently, though 
since the spec allows UAs to arbitrarily limit bandwidth usage to zero for 
media elements, that doesn't really mean much).


On Mon, 22 Oct 2007, Geoffrey Garen wrote:
> >
> > -- sound is produced for display:none elements and elements not in the 
> > DOM
> 
> Does this statement imply that elements producing sound must be 
> explicitly protected from garbage collection?

Yes.


On Tue, 23 Oct 2007, Jonas Sicking wrote:
> 
> One thing that I was thinking of that might be beneficial both for the 
> greasemonkey case, and for debugging purposes is to have an API where 
> you can enumerate all currently sounding audio/video elements. This 
> could something like:
> 
> interface Document {
> 
>   ...
>   readonly NodeList soundingMediaElements;
>   ...
> }
> 
> The UA would most likely have to keep a list like this anyway in order 
> to protect these elements from getting garbage collected.
> 
> One argument against this is that it might make it harder for 
> implementations to optimize what elements it's keeping alive. For 
> example, an implementation could remove an element from this list if it 
> knows that a <video> contains no audio from the current play position to 
> the end, even if earlier parts did contain audio.

I think we should wait for v2 before adding this, but it's an interesting 
idea. We'll see if there is demand.

Cheers,
-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'