[whatwg] Script preloading

[Jake Archibald]

On Wednesday, 10 July 2013, Kyle Simpson wrote:
>
> You know, I keep relying on the fact that the body of work on this topic for almost 3 years … I've spent more time over the last 4+ years obsessing on script loading than any other developer … I am saying the same things I've been saying for 3 years.

This is the open web, length of service does not excuse anyone from
reasoning and evidence.

>
> But sure, I'll say them, AGAIN, because now someone wants to hear them again.

If you feel you're repeating content from elsewhere, you could have
linked to it (hurrah for the web)! If you hadn't compiled your
use-cases & requirements in one place before, then excellent, you have
now. You can link to this in future.

>
> I doubt anyone is going to read this crazy long message and actually read all these, but I'll put them here nonetheless.

I am reading this, and I will show how Hixie's #1 solution (plus the
minor additions I suggested) meet your use-cases. For others reading
I'll also detail the use-case in a complete but succinct way.

>
> 1. Premise: I'm the author of a popular and wide-spread used script loader. It's a general utility that's used in tens of thousands of different sites, under a myriad of conditions and in different ways, and in a huge swath of different browsers and devices. I need the ability inside this general utility to do consistent, 100% reliable, predictable script loading for sites, without making ANY assumptions about the site/markup/environment itself. I need to be as unintrusive as possible. It needs to be totally agnostic to where it's used.

Use-case: Script loaders such as LabJS should continue to work at
least as well as they do now.

As you've stated previously, LabJS is complete in that it continues to
work without much development effort. LabJS could either improve by
using the new feature, or not, and continue as is.

>
> 2. Premise: I need a solution for script (pre)loading that works not JUST in markup at page-load time, but in on-demand scenarios long after page-load, where markup is irrelevant. Markup-only solutions that ignore on-demand loading are insufficient, because I have cases where I load stuff on-demand. Lots of cases. Bookmarklets, third-party widgets, on-demand loading of heavy resources that I only want to pay the download penalty for if the user actually goes to a part of the page that needs it (like a tab set, for instance). In fact, most of the code I write ends up in the on-demand world. That's why I care so much about it.

Use-case: I want to preload scripts that execute straight away, such
as social media scripts, and defer their execution. The need for the
script may be determined by script (eg feature detection), so you need
to be able to trigger preload via script. Executing the script should
be optional (user may not interact with the button).

(this is actually many of the use-cases in this email rolled into one,
to save on reading and repetition)

Anyway, here's how you'd preload two scripts and have them execute in
order some time later (but not be held up by other scripts like
async=false). If the script have more flexibility in terms of
execution order, you can specify that and get better performance.

<link rel="subresource" href="path/to/script.js" class="preload">
<link rel="subresource" href="path/to/another-script.js" class="preload">
<script>
  // scripts are preloading at this point
  function loadScripts(done) {
    var toLoad = document.querySelectorAll('.preload');
    var script;
    for (var i = 0, len = toLoad.length; i < len; i++) {
      script = document.createElement('script');
      // depend on the previous script
      if (i) script.dependencies = 'script[src="' + toLoad[i-1].href + '"]';
      script.src = toLoad[i].href;
      document.head.appendChild(script);
    }
    script.onload = done;
  }

  loadScripts(function() {
    // scripts are ready!
  });
</script>

The without-markup solution is the same as above, but the
link[rel=subresource] elements are created conditionally with JS.

link[rel=subresource] is the right solution for preloading, it's what
it's for and it works on more than just script. If there are issues
with this & cache headers, there shouldn't be, let's fix that.

> 3. Premise: this is NOT just about deferring parsing. Some people have argued that parsing is the expensive part. Maybe it is (on mobile), maybe not. Frankly, I don't care. What I care about is deferring EXECUTION, not parsing (parsing can happen after-preload or before-execution, or anywhere in between, matters not to me). Why? Because there's still lots of legacy content on the web that has side-effects when it runs. I need a way to prevent those side effects through my script loading, NOT just hoping someday they rewrite their code so that it has no side effects upon execution.

Use-case: this is just the previous use-case with more words.

See above. Although most libraries etc defer major execution to
function calls anyway, this use-case is catered for.

>
> NOTE: there ARE people who care about the expense of parsing. Gmail-mobile (at one point, anyway) was doing the /* here's my code */ comment-execute trick to defer parsing

I'm unconvinced that modern browsers (which the proposed feature will
be present in) suffer in terms of JS parsing. However, it is catered
for. I've contacted the gmail team to get their take on it.

> 4. Use-case: I am dynamically loading one of those social widgets that, upon load, automatically scans a page and renders social buttons. I need to be able to preload that script so it's ready to execute, but decide when I want it to run against the page. I don't want to wait for true on-demand loading, like when my user clicks a button, because of the loading delay that will be visible to the user, so I want to pre-load that script and have it waiting, ready at a moment's notice to say "it's ok to execute, do it now! now! now!".

Use-case: As above, but a single script file & no ordering.

Pretty much the same but we can simplify due to lack of dependencies.

<link rel="subresource" href="path/to/script.js" class="preload">
<script>
  function loadScript() {
    var script = document.createElement('script');
    script.src = document.querySelector('.preload').href;
    document.head.appendChild(script);
  }
</script>

>
> <link rel="subresource">… almost 50% of scripts … sent without proper caching headers. If the browser is doing what it should do, it won't cache those

This is not how link[rel=subresource] should work, the first request
for the subresource should pick up from where the link left off. If it
doesn't do that, let's fix it, then we get the feature working
properly for more than just scripts.

>
> 6. Use-case: I want to preload a script which is hosted somewhere that I don't control caching headers, and to my dismay, I discover that they are serving the script with incorrect/busted/missing caching headers. If I use a cache-preload technique, it will fail to work as I had hoped. I will pay the double-download penalty because the browser didn't actually cache it the first time, and my user will pay the extra UX penalty of having to wait longer for the second load, when my whole goal was to remove that UX visible delay.

Use-case: (as above) I should be able to preload scripts served by a
third party with no cache headers

link[rel=subresource] has you covered and should be fixed if it isn't working.

>
> 7. Premise: my script loader might be used in one centralized and coordinated location on the page, or it might be used many times indepedently in many parts of the page, such as many different CMS plugins requesting their own sets of scripts to load. Those plugins are ignorant of anything else going on in the page, and as far as they're concerned, they'll expect their script loading to be independent of any other script loading.

Use-case: X groups of Y scripts are loaded (each group is an
independent page feature), they must execute in order within their
group but execution within one group must not be held back by scripts
outside that group.

The first code example covers this. The scripts execution is ordered
within each group.

>
> 8. Use-case: One CMS plugin wants to load "A.js" and "B.js", where B relies on A. Both need to load in parallel (for performance), but A must execute before B executes. I don't control A and B, so changing them is not an option. This CMS plugin doesn't want to auto-execute its code, however. It wants to wait for some user-interaction, such as a button click, before executing the code. We don't want there to be any big network-loading delay visible to the user between their click of the button and the running of that plugin's code. Preloading is desired to minimize the delays.
>
> …snipped due to the 40k WHATWG limit…
>
>
> So any solution which relies on these sorts of things MIGHT fail, and is thus unreliable and insufficient. Full stop.

Use-case: A CMS plugin script group (as above) depends on another
script group which may not be present, but if it is present it
shouldn't be loaded and/or executed a second time.

This is very much the domain of module loaders such as LabJS, Require
and eventually ES6 modules.

The framework would need to provide some abstraction to load the core
script group, else any plugin system that depends on URLs for
dependencies is fragile to changes in the shape of the core script
group.

With fragile URLs:

<script>
  var dependencyUrl = new URL('/core.js', window.location.href).href;
  var dependencyExists = document.findAll('script').some(function(script) {
    if (new URL(script.src).href == dependencyUrl) {
      script.classList.add('core');
      return true;
    }
  });

  if (!dependencyExists) {
    let script = document.createElement('script');
    script.src = dependencyUrl;
    script.className = 'core';
    document.head.appendChild(script);
  }

  var script = document.createElement('script');
  script.src = dependencyUrl;
  script.dependencies = '.core';
  document.head.appendChild(script);
</script>

The above would be less fragile if the core script group had a class
name, and a method to add them to the page if they're not already
there. This would allow the framework to change the shape and url of
its core scripts without breaking plugins.

However, I still consider the above a hack around an absent module
loading system.

The example uses http://url.spec.whatwg.org/ to resolve URLs. I'm not
convinced it caters for <base>, if not we should fix this in the URL
spec so more use-cases can benefit from it.

>
> 9: Premise: sometimes you load more scripts than you actually use. While that can be considered a bad thing in many cases, there are cases where it's a desired thing. Any suitable solution needs to be able to preload a script but not be required to actually ever execute it.

Use-case: A social media button script is preloaded for speed of
execution, but the user doesn't trigger execution. Execution of
preloaded scripts should not be required.

This is part of the social media use-case, I've rolled it into the
first example. It's catered for, just don't call the loadScript(s)
function.

>
> 10. Use-case: I have two different calendar widgets. I want to pop one of them up when a user clicks a button. The user may never click the button, in which case I don't want the calendar widget to have ever executed to render. I don't just want the calendar widget rendered but hidden, I don't want it rendered into the DOM at all if the user doesn't click the button.
>
> …snipped due to the 50k WHATWG limit…
>
>
> Any solution which forces a script that is preloaded to eventually be executed is insufficient. Full stop.

Use-case: as above but with lots of words.

Note: typing "full stop." after a full stop is unnecessary and without
care could result in recursion, be careful! :)

>
> 11. Use-case: I have a set of script "A.js", "B.js", and "C.js". B relies on A, and C relies on B. So they need to execute strictly in that order.
>
>
> But, I want to ensure that there's as little delay between A executing and B executing and C executing as possible. For instance, imagine they progressively render different parts of a widget. I don't want A to run just because it's done loading, if B and C are not ready to execute "immediately" thereafter. In other words, I only want to execute A, B and C once all 3 are preloaded and ready to go. It's not just about their order preservation, but also about minimizing delays between them, for performance PERCEPTION.
>
> …snipped due to the 50k WHATWG limit…
>
> Any solution that relies on something having been executed (`onload`) or being marked eligible for auto-execution (`markNeeded()`) earlier than I want it to be … It's insufficient. Full stop.

Use-case: Prevent the user seeing the page enhance script by script
(this is ugly in some edge cases), and instead enhance all at once.

Enhancing the page script by script should be the default as it's
getting behaviour and potentially content to the user faster, but
avoiding this is certainly possible.

The vast majority of scripts don't do anything until a function is
called. In which case the solution is:

<script src="1.js" class="core"></script>
<script src="2.js" class="enhancement" dependencies=".core"></script>
<script src="3.js" class="enhancement" dependencies=".core"></script>
<script src="4.js" class="enhancement" dependencies=".core"></script>
<script src="5.js" class="enhancement" dependencies=".core"></script>
<script dependencies=".enhancement">
  enhanceThis();
  enhanceThat();
  enhanceAllTheThings();
</script>

The final script could be in its own file too if it's repeated in many
places. As you can see from earlier examples, this is possible using
script only too.

However, if you have scripts that enhance stuff as soon as they
execute, you can work around this too:

<style>
  .before-enhancement .thing-to-enhance {
    visibility: hidden;
  }
</style>
<script>
  document.documentElement.classList.add('before-enhancement');
</script>
<script src="1.js" class="core"></script>
<script src="2.js" class="enhancement" dependencies=".core"></script>
<script src="3.js" class="enhancement" dependencies=".core"></script>
<script src="4.js" class="enhancement" dependencies=".core"></script>
<script src="5.js" class="enhancement" dependencies=".core"></script>
<script dependencies=".enhancement">
  document.documentElement.classList.remove('before-enhancement');
</script>

>
> OK, deep breath. Sigh. That was way more than I wanted to write, and probably way more than anyone wanted to read.

Yes, as I've shown by my summaries there's a lot of repetition and
things could be a lot more brief and get the same point across. I try
to read through my emails before I send them and eliminate repetition
& extraneous language to ensure I'm making my point with the minimum
reading required.

>
> But that should cover all my big concerns. Hopefully I don't have to recount those use cases over and over again, now.

Well, you can simply link to your email in future. It's something you
can cite in discussions rather than expect others to exhaustively
crawl the internet for all relevant material and history.

>
> I noted a whole bunch of places where I see shortcomings of the other proposals compared to the nuances of those use-cases.

I am interested to see how the above use-cases would be met in your
counter proposal(s) to see if it would be simpler/faster. If LabJS is
a requirement, it must be factored in as a unit of complexity and
load-step.

Please do this rather than declare anything to be insufficient without
reasoning.

>
> But nevertheless, I hope that list is exhaustive enough (certainly exhausting to write/read) to show that there's a lot more than just "I really want this". There's real complexities to these use-cases.

It's a good set of more complex cases, yes.

>
> As far as I'm concerned, the only "silver bullet" that solves every single one, no matter how complex, stated here or not, is real preloading like my proposal suggests.
>

I've provided examples that the simpler of Hixie's proposals solves
each with very little code and no external libraries.

If there are issues with link[rel=subresource] and/or the URL spec,
lets fix those to the benefit of use-cases that aren't exclusively
script loading and the whole web wins.

Cheers,
Jake.