[whatwg] Adding and removing media source elements

[Philip Jägenstedt]

On Wed, 04 Feb 2009 00:59:12 +0100, Calogero Alex Baldacchino  
<alex.baldacchino at email.it> wrote:

> Philip Jägenstedt ha scritto:
>> On Tue, 03 Feb 2009 05:44:07 +0100, Ian Hickson <ian at hixie.ch> wrote:
>>
>>> On Tue, 3 Feb 2009, Chris Pearce wrote:
>>>>
>>>> (2) Why don't we invoke load() whenever a media element's src  
>>>> attribute
>>>> or <source> children are changed, regardless of networkState? That way
>>>> changes to the media's src/source other than the first change would  
>>>> have
>>>> the same effect as first change, i.e. they'd have an immediate effect,
>>>> causing load() to be invoked.
>>>
>>> Doing this would cause the first file to be downloaded multiple times  
>>> in a
>>> row, leading to excessive network usage.
>>>
>>
>> Surely this can't be the only reason? User agents are free to  
>> speculatively keep the current source loading when src/source changes  
>> and to stop loading it only if the "current media resource" does  
>> change. That, and caching, should be enough.
>>
>> I have always imagined that the reason for the conditioned load() is to  
>> not interrupt playback by fiddling with the DOM or doing something like  
>> *v.src=v.src* (although I'm quite sure that doesn't count as changing  
>> the attribute). However, now I can't convince myself that this makes  
>> any sense, since surely if you change src/source you actually do want  
>> to change the effective source (and load() is scheduled to run after  
>> the current script, so there's no risk of it being run too early).
>
> Doing the same with a script element can cause the script to be  
> re-fetched and re-executed on some browsers, so I think there is a  
> concrete chance to find the same behaviour for videos, and the spec have  
> to say when the load is allowed (or, at least, when it should not  
> happen). I'm not sure that every changes to the effective source should  
> take place, for instance, changing it (through the dom) after playback  
> has already started might not be very usable and should be avoided,  
> therefore, any such attempt should be ignored/aborted (eventually with  
> an exception) after playback start and until its end or an explicit stop  
> (by the user or by a script, so to encourage programmers to check the  
> state of the playback before taking any action).

I also had this "avoid accidental reloads" theory before, but it doesn't  
strike me as very reasonable after thinking more about it. Can anyone give  
an example of a use case where the DOM src attribute or <source> elements  
are added/changed "accidentally" so that it would cause an unwanted reload?

> Also, scheduling the load "after the current script" could not solve the  
> whole problem: any changes to the video may happen through an event  
> handler, therefore by different scripts, thus I think that it could be  
> helpful to allow a script to freeze (or revert) ongoing operations (as  
> well as the video interface) but playback (if yet started), so to (try  
> and) ensure (somehow) that any dynamic changes can be performed without  
> bothering the user, or are disallowed otherwise.

Just so that there is no misunderstanding, load() already is scheduled to  
run after the current script has finished. This was rather explicit in an  
earlier version of the spec and the new wording with "queue a task" should  
be taken to mean the same I assume. Perhaps it should be made explicit  
though.

> (what for? I'm considering the (maybe edge) case of a dynamic update of  
> a video source, for instance when a different/better source (higher  
> quality or with a more appropriate translation) is available, or for any  
> other reason (e.g. the complete list of available sources might be  
> streamed as a sequence of remote events for an immediate update and a  
> deferred/repeated playback), but if the current source is being played  
> it might not make sense to stop it and change it with a different one,  
> eventually restarting from the beginning, because it may be annoying for  
> users).

Can you give an example of when a script would want to change src/source  
without intending for it to cause a change of the effective source? It  
sounds like an edge case indeed and something that the spec doesn't really  
need to care about. If you really wanted to do it, construct/copy a  
HTMLMediaElement in the DOM over time (in however many event handlers you  
want) and insert/replace it into the Document when done.

>
>>
>> Related, since load() is async it depends on timing whether or not
>>
>>    <video id=v></video>
>>    <script>
>>     v = document.getElementById('v');
>>     v.src = 'test';
>>    </script>
>>
>> causes the source 'test' to be loaded, as the network state may not be  
>> NETWORK_EMPTY when the src attribute is set. The same goes for adding  
>> source child elements of course. Yes, this is the same issue as  
>> http://lists.w3.org/Archives/Public/public-html/2009Jan/0103.html and  
>> would be resolved by calling load() unconditionally.
>>
> Or checking the network state to choose if it's the case to call load()  
> explicitely; however, due to its asynchronous nature, that might cause a  
> double invocation (depending on implementations), or similar problems.  
> Perhaps, the load() method should leave the network state unchanged  
> (NETWORK_EMPTY in this case) or revert it to a previous value whenever  
> the method fails to choose a candidate (e.g. because there is no  
> valid/new source, a yet chosen source is being played and cannot be  
> changed before it's stopped, and so on), and successive changes could be  
> scheduled for an evaluation as soon as possible (e.g. as soon as the  
> network state returns to be NETWORK_EMPTY, or becomes NETWORK_LOADED  
> and/or the playback ended or has been stopped - if appropriate in this  
> case), possibly being collapsed into a single task.

Certainly it's possible to work around the current quirks in the spec by  
adding an explicit load(), but wouldn't it be better to remove the quirks  
unless there's some very compelling reason for them to be there?  
Scheduling load() after playback has ended and this type of thing just  
seems to add to the unpredictable asynchronousness of it all.

It's a big problem that the script above can cause the source 'test' to  
sometimes be loaded and sometimes not, depending on implementation and  
speed of the network. If the network stops right after the <video> tag the  
asynchronous part of load() will finish and return networkState to  
NETWORK_EMPTY. However, if the network is fast you'll likely load the  
script and run it before the asynchronous part of load() has been run,  
while networkState is NETWORK_LOADING. It's completely un-obvious to the  
casual observer that a load() might be needed in the example above, and  
script authors will make mistakes since it will work in some browsers some  
times.

> This way, a load evaluation preceeding the script execution, in your  
> example, would fail and revert the network state to be empty, triggering  
> a new invocation after the script has been executed; an evaluation  
> following the script would work as expected; an evaluation invoked while  
> the script is executing would cause the new v.src value to be scheduled  
> for a later check, (the overall mechanism would result in an  
> unconditioned scheduling of conditioned load() invocations, collapsed  
> into one single entry until a call to .load() is made, which I think  
> should be more performant than calling load() unconditionally - but I'm  
> not sure, it may depend on both the scheduler and the load method  
> implementations).
>
> Source elements might require a different treatment from the "src"  
> attribute. For instance, adding such an element while a list of  
> candidates is being generated or before entering the Candidate Loop  
> could cause the list to be re-checked, otherwise a delayed load is  
> scheduled (but this could add unneeded complexity, thus just scheduling  
> the change for a later evaluation could be enough).

The DOM can't change while the candidate list is being generated as that's  
in the synchronous part of load(). But yes, one could keep adding criteria  
for scheduling new delayed load()s and perhaps solve some of the problems.

However...

There seems to be no practical benefit from the current behavior. To the  
contrary, it (sometimes, depending on random factors) cause some rather  
unexpected behavior that requires in-depth reading of the spec to  
understand. This is our implementor experience. Instead of making the spec  
more complicated to compensate, I suggest making load() unconditional. I  
am confident that we can handle the caching issues just fine and that edge  
cases of changing src/source when you don't really want to change the  
effective source are 100% solvable by other means. In fact, there would be  
little need for an explicit load() function at all, but that's another  
issue...

-- 
Philip Jägenstedt
Opera Software