[whatwg] <video> loading algorithms

[Ian Hickson]

Silvia wrote:
>
> [...] All other browsers set the @networkState to NETWORK_EMPTY at the 
> start of loading a new media resource (Firefox, Safari and Chrome). I 
> was going to discuss this with you, since Opera interprets the spec 
> different here - which is understandable, since the first step in the 
> resource selection algorithm says to:
>
> 1. Set the networkState to NETWORK_NO_SOURCE.

The other browsers implemented the first draft, which was then changed 
based on implementation feedback. Hopefully the other browsers will update 
their implementation now to fix the bugs that were present in the original 
version of the spec, leading to everyone doing the same thing. :-)


> OTOH however, the description for the [NETWORK_EMPTY] state is as 
> follows:
>
> NETWORK_EMPTY (numeric value 0)
> The element has not yet been initialized. All attributes are in their 
> initial states.
>
> and for NETWORK_NO_SOURCE is:
>
> NETWORK_NO_SOURCE (numeric value 3)
> The element's resource selection algorithm is active, but it has failed 
> to find a resource to use.
> 
> Thus, I believe this may be a spec bug and really the first step in the 
> resource selection algorithm should say
>
>    1. Set the networkState to NETWORK_EMPTY.
>
> which is more consistent with the implementations of the other browsers 
> and also consistent with the meaning of NETWORK_EMPTY and 
> NETWORK_NO_SOURCE.

The definitions were a bit confusing. I've fixed them. The algorithm was 
correct, however.


On Sat, 24 Jul 2010, Chris Pearce wrote:
> 
> Thanks, Firefox is not setting the networkState to NETWORK_NO_SOURCE 
> when the synchronous part of the resource selection algorithm returns. 
> We've not updated our load algorithm implementation in about a year, 
> there will no doubt be other places where we're non-compliant. We've got 
> an existing bug on file to update the load algorithm:
> 
> https://bugzilla.mozilla.org/show_bug.cgi?id=485288
> 
> Walking on water and developing software from a specification are easy 
> if both are frozen.

Hear hear!


On Sat, 24 Jul 2010, Silvia Pfeiffer wrote:
>
> There is definitely a spec bug, because different locations of the spec 
> say diverging things.

For the record, when the spec contradicts itself, and one part is 
normative (such as the algorithm here) and another is non-normative (such 
as the definitions here) then always assume the normative part is wrong. I 
often forget to update the non-normative parts.


On Mon, 26 Jul 2010, Silvia Pfeiffer wrote:
> 
> I now wonder about the intention of play() (and also of pause()). As I 
> understood it, they are both meant to reload the media resource if 
> @currentSrc has changed, similar to what load() is supposed to do.

I do not believe that has ever been the intent.


> Also, I wonder what NETWORK_EMPTY is used for when not at the beginning 
> of parsing the media resource. I haven't managed to catch Opera in that 
> state.

NETWORK_EMPTY is the state you'll find immediately after creating a 
<video> element with no src="" element:

   document.createElement('video').readyState == NETWORK_EMPTY


On Sat, 24 Jul 2010, Maciej Stachowiak wrote:
>
> [...] should we ensure that any time you call play(), it will cause the 
> media resource to start playing once loaded? That seems like the real 
> spec bug.

As far as I can tell, the spec already guarantees that.


On Mon, 26 Jul 2010, Philip Jägenstedt wrote:
> 
> My solution:
> 
>    <video controls width="400px">

Note that the "px" here is invalid (and ignored).

>    </video>
>    <script type="text/javascript">
>      var video = document.querySelector("video");
>      var exts = ["mp4", "webm", "ogv"];
>      exts.forEach(function(ext) {
>        var source = document.createElement("source");
>        source.src = "HelloWorld."+ext;
>        source.type = "video/"+ext;
>        video.appendChild(source);
>      });
>      video.play();
>    </script>
> 
> Of course, there must be some good reason to use scripts to begin with, 
> as the result of this is always the same, so you might just as well use 
> static markup.

It's not clear what this would solve. The above is a really complicated 
way of doing things as far as I can tell.


On Tue, 27 Jul 2010, Silvia Pfeiffer wrote:
> 
> Sure, but this is only a snippet of an actual application. If, e.g., you 
> want to step through a list of videos (maybe an automated playlist) 
> using script and you need to provide at least two different formats with 
> <source>, you'd want to run this algorithm frequently.

Just have a bunch of <video>s in the markup, and when one ends, hide it 
and show the next one. Don't start dynamically manipulating <source> 
elements, that's just asking for pain.

If you really must do it all using script, just use canPlayType and the 
<video src=""> attribute, don't mess around with <source>.


On Mon, 26 Jul 2010, Philip Jägenstedt wrote:
> 
> Looking again at the resource selection algorithm, the second step is to 
> await a stable state, i.e. wait until the current script has finished. 

Not necessarily finished. The idea is just that the resource selection 
algorithm might run on another thread, and the way the spec is written 
allows the UA to synchronise at very predictable points (e.g. when 
showModalDialag() is invoked, or when the HTML parser isn't inserting 
elements, or while waiting for a style sheet to load so that a <script> 
can be run).


> Given that, it wouldn't be a big problem to let modification of src 
> attributes on source elements trigger resource selection, you won't get 
> the 3-2-1 problem I mentioned earlier.

I don't really understand what use case this would solve.


On Tue, 27 Jul 2010, Silvia Pfeiffer wrote:
> 
> Right, so it works if you create the <source> elements newly, but it 
> still doesn't work when you have previously created the <source> element 
> just with an empty @src attribute (which I think is legal).

It's not.


> Both of these work in all the other browsers, btw.

They're buggy.


On Tue, 27 Jul 2010, Philip Jägenstedt wrote:
> 
> All attributes are mutable via setAttribute. The reason that src, type 
> and media are also reflected as properties on HTMLSourceElement is so 
> that it's easier to write code like this:
> 
> var v = document.querySelector('video');
> var s = document.createElement('source');
> s.src = 'foo.webm';
> s.type = 'video/webm';
> v.appendChild(s);
> 
> [..] clarification of that would be helpful, not least for authors. This 
> isn't the actual reason for the current incompatible behavior though, 
> the reason is that only Opera uses the NETWORK_NO_SOURCE state as per 
> the spec.

I've added a note saying not to bother dynamically updating <source>.


On Wed, 28 Jul 2010, Silvia Pfeiffer wrote:
> 
> I believe the spec should specify what authors should reasonably expect 
> to work. I think this is a case that should reasonably be expected to 
> work. If the spec doesn't currently allow it, but browsers have 
> implemented it - so it is possible to make it work - then I don't see a 
> reason why the spec shouldn't be adapted to work as implementations work 
> and as users/authors expect it to work.

What's the use case for modifying <source>? It just seems like a much more 
complicated way of doing things that necessary. Just use <video src=""> 
and your life will be way easier.


On Wed, 28 Jul 2010, Philip Jägenstedt wrote:
> 
> I'm just saying that it shouldn't work if implementations follow the 
> spec. I don't think that it *not* working is a feature, but I also don't 
> think that making it work is very important. In the example the markup 
> was invalid, so if the spec doesn't make it work perfectly I'm OK with 
> that. Furthermore, once it stops working in all browsers authors will 
> learn to add a call to load() after they are done modifying the 
> attributes of source elements. If it turns out that this is something 
> many authors get stuck on then maybe we can add more triggers for src, 
> type and media attributes on source elements.

Indeed.


On Tue, 3 Aug 2010, Philip Jägenstedt wrote:
> 
> For the record, here's how I interpreted "await a stable state":
> 
> The only state that is not stable is a running script.

Any running task is "not stable", not just running script. For example, 
you shouldn't run a synchronous section while the parser is adding 
elements to the DOM.


> The only step in any video-related algorithm one can reach from a script 
> is step 2 of the resource selection algorithm. Therefore, when reaching 
> that step, if the resource selection algorithm was triggered by a 
> script, wait until that script has finished and then continue. The only 
> somewhat tricky part is that if we are in an event handler triggered by 
> script, we should wait until the script that triggered the event handler 
> has finished. The only way I know of triggering this corner case is by 
> invoking a synchronous event handler from script, e.g. by calling 
> click().
>
> All other occurrences of "await a stable state" I've ignored as we can't 
> not be in a stable state when reaching them.

Sure you can. The rest of the algorithm is asynchronous, so a script could 
have started when you reach them. Or the parser could be adding elements, 
or you could be handling incoming network traffic in a way that affects 
the DOM (e.g. EventSource could be about to fire an event), etc.


On Tue, 3 Aug 2010, Boris Zbarsky wrote:
> 
> Per spec as currently written, this will give the wrong behavior.  For 
> example, if the script in question calls showModalDialog, this is 
> supposed to interrupt the script task and spin the event loop, and so 
> your synchronous section would need to run while the modal dialog is up.  

Indeed, that's another point where the event loop will run the synchronous 
sections.


On Thu, 5 Aug 2010, Boris Zbarsky wrote:
> 
> No, I meant what I said.  Async XHR is clearly going to go back to the 
> event loop, etc.  In Gecko, sync XHR does as well, by manually spinning 
> the event loop under the Send() call (and blocking delivery of certain 
> classes of events to the web page that triggered the send).

I'll let Anne worry about the synchronous XHR issues.


> This isn't at all like the "pause" state the spec describes, which I 
> also mentioned.
> 
> On the other hand, I think the spec's "pause" state can have some really 
> undesirable interactions whereby a sync XHR on one page can affect 
> behavior of other pages, which is why I have problems with it...

Certainly "pause" is not great.

Currently the things I'm aware of that trigger "pause" are:

 - the onbeforeunload prompt
 - alert()
 - confirm()
 - prompt()
 - printing (including print() and user-triggered printing)
 - getting or setting document.cookie*
 - manipulating the localStorage object*

(Cases marked * are side-effects of needing to obtain the storage mutex.)

I don't see how to get around needing to pause for any of these. For 
example, consider alert(): if you don't pause, then timeouts could run, 
and a timeout that fires alert() could result in infinite alerts. You 
could spin the event loop but blocking task sources that could end up 
running script, but I don't think that would leave you with much 
(possibly nothing) so I don't see the point.


On Fri, 6 Aug 2010, Philip Jägenstedt wrote:
> 
> The most important things is that the following always alert 3 
> (NETWORK_NO_SOURCE):
> 
> <!doctype html>
> <video>
> <source>
> <script>alert(document.querySelector('video').networkState)</script>
> 
> In other words, the synchronous section must at the latest be run just 
> before running an inline script.

Right.


On Fri, 6 Aug 2010, Boris Zbarsky wrote:
>
> I still think the behavior of pause (which you're proposing for 
> showModalDialog as well, I think) makes no sense; I'd much rather have 
> the showModalDialog behavior for things like sync XHR, alerts, etc.

I don't see how we could.


> OK, let's focus on the issue at hand; I agree that we're more likely to 
> get somewhere with that than with generalities.  What about this case:
> 
> <script>
>   var v = document.createElement("video");
>   var s = document.createElement("source");
>   v.appendChild(s);
>   document.body.appendChild(v);
>   alert(v.networkState);
> </script>
> 
> Should this also run the synchronous section before returning from the 
> appendChild call, even though there's a script active?

That should always alert 3; the synchronous section doesn't run before the 
alert.


On Mon, 9 Aug 2010, Philip Jägenstedt wrote:
> 
> Whether it's the parser or a script that triggers the resource selection 
> algorithm, it is run up to step 2 where it's supposed to await a stable 
> state. Actually, my example is bad because networkState is set to 
> NETWORK_NO_SOURCE is step 1. What's actually interesting is when one can 
> see the side-effects of the synchronous section running, e.g. by 
> networkState being set to NETWORK_LOADING.
> 
> In the case of a script triggering it, it's important that the 
> synchronous section not be run until the script has finished modifying 
> the DOM.

This is also important in the parser-triggering case.


> If that's accomplished by waiting until the thread has finished or by 
> means of the event loop isn't important as long as it's implementable 
> and the result is 100% predictable. Modulo the effects of modal dialogs 
> and other corner-cases, networkState should remain at NETWORK_NO_SOURCE 
> for the remainder of the script that triggered resource selection.
> 
> Parsing then. I don't know the parser section very well, but it doesn't 
> mention "task", so I'm guessing that running it isn't considered a task 
> itself.

It is. Each task that the networking task source places on the task queue 
while the fetching algorithm runs must then fill the parser's input stream 
with the fetched bytes and cause the HTML parser to perform the 
appropriate processing of the input stream.


> I can't find anything to suggest that the parser inserting a script 
> element would cause synchronous sections to run before the script is 
> executed.

Can cause. Step 3 of the Otherwise clause at the end of the "An end tag 
whose tag name is "script"" section of The "text" insertion mode.


> What then, does "await a stable" state amount to? Just running it the 
> next time the event loop spins would appear to give rather random 
> results, right?

How so?


> The parser example above alerts 0 or 3 depending on whether the sync 
> section has run.

It should always alert 3. The networkState is set _before_ the sync 
section runs.


> Opera alerts 3, as the sync section is run at the same time the source 
> element is inserted by the parser. If that's not the right thing to do, 
> what is?

The sync section runs whenever the parser decides it's time to run it -- 
conceptually, you can treat any sequence of bytes processed by the parser 
in an uninterrupted fashion as a task.


On Mon, 9 Aug 2010, Philip Jägenstedt wrote:
> 
> The general idea of waiting is that since the following steps of the 
> resource selection algorithm inspects video elements src attribute and 
> source element children, those should be in a consistent state before 
> the checking is done.

Right. We don't want the list of <source> elements changing while the 
algorithm is looking at the DOM.


On Mon, 9 Aug 2010, Boris Zbarsky wrote:
> 
> OK, but then why are we not imposing such a requirement for the case 
> when the <video> is being created by the parser?

We do.


On Tue, 10 Aug 2010, Boris Zbarsky wrote:
> 
> Why does the algorithm not just reevaluate any sources after the 
> newly-inserted source instead?

Because evaluating a <source> can involve a multi-megabyte network load. 
It would also mean that you could get very unpredictable behaviour, e.g. 
if earlier <source>s were to suddenly become available again or change 
type (e.g. as might happen if the page was loaded from cache while the 
user was logging into a wireless network with a captive portal).


On Tue, 10 Aug 2010, Boris Zbarsky wrote:
> 
> Well, the problem is that it introduces hysteresis into the DOM. [...] 
> why do we only consider sources inserted after the |pointer| instead of 
> all newly inserted sources?

What exactly are you proposing we do instead?

I couldn't come up with a mechanism that didn't have some hysteresis in 
the face of pathological DOM manipulation. I figured if we were going to 
have some, I'd just make it so that the spec was the simplest necessary to 
handle the normal use cases, and had the most optimistic performance 
characteristics (least built-in delays).

However, if you can come up with a mechanism that doesn't exihibit 
hysteresis -- i.e. has the same behaviour regardless of the speed of the 
network or of the implementation, and independent of the order in which 
nodes are added to the DOM, without requiring that the UA wait beyond 
parsing the first <source> to start downloading resources -- then I'm all 
for adopting that.


On Mon, 9 Aug 2010, Philip Jägenstedt wrote
>
> So, what I'm tentatively suggesting is:
> 
> 1. Remove the "await a stable state" concept and just continue running 
> the steps that follow it. (This is what Opera does now when resource 
> selection is triggered by the parser, as I have no idea how long to wait 
> otherwise.)

This would introduce race conditions, so that's not really a good idea.


On Mon, 9 Aug 2010, Philip Jägenstedt wrote:
> 
> 2. Instead of calling the resource fetch algorithm in step 6/9, queue a 
> task to call it and then return. The failure steps that follow can be 
> called explicitly from the resource fetch algorithm.

That would potentially make things much slower. What's the advantag?


On Wed, 11 Aug 2010, Philip Jägenstedt wrote:
> 
> [...] would result in disasters like this:
> 
> <!doctype html>
> <video src="video.webm"></video>
> <!-- network packet boundary or lag? -->
> <script>alert(document.querySelector('video').networkState)</script>
> 
> The result will be 0 (NETWORK_EMPTY) or 2 (NETWORK_LOADING) depending on 
> whether or not the parser happened to return to the event loop before 
> the script.

What it returns depends entirely on what happened when you fetched 
video.webm, so it's not really possible to tell. This isn't a disaster, 
you can do the same with, e.g., <img src=""> -- will its .complete be 
true or not?

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'