[whatwg] AppCache-related e-mails

[Michael Nordman]

A common request that maybe we can agree upon is the ability to list the
manifests that are cached and to delete them via script. Something like...
  String[] window.applicationCache.getManifests();  // returns appcache
manifest for the origin
  void window.applicationCache.deleteManifest(manifestUrl);

I think it's clear from this discussion (and others) that the overall
appcache feature set leaves something to be desired, but it's less clear how
to best satisfy the "desirements". Until there is some clarity, it's hard to
see how the community is going to make progress. Personally, I think whats
needed to move things forward is for browser vendors to do some independent
innovating to see what works and what doesn't work.

>> @Hixie... any idea when the appcache feature set will be up for a growth
>> spurt? I think there's an appetite for another round of features in the
>> offline app developers that i communicate with. There's been some recent
>> interest here in pursuing a means of programatically producing a
>> response instead of just returning static content.
>
> Who implements it currently? Is there a test suite? Those are the main
> things that would gate a dramatic addition of new features.

Well, nobody yet; but I have a roadmap in mind that builds up to that. Much
of the discussion in this thread has been on the second item. Mobile
developers are particularly interested 2 to avoid HTTP cache churn and the
cost of HTTP cache validation. In this roadmap, you can see that it would
also allow pages vended from servers to make use of "executable intercept
handlers".


-1. [DONE] Support for cross-origin HTTPS resources.
http://code.google.com/p/chromium/issues/detail?id=69594

0. [DONE] A means of not invoking the fallback resource for some error
responses that would generally result in the fallback resource being
returned. An additional response header would suite they're needs...
something like...
x-chromium-appcache-fallback-override: disallow-fallback
If a response header is present with that value, the fallback response would
not be returned.
http://code.google.com/p/chromium/issues/detail?id=82066

1. [UNDER CONFUSING DISCUSSION] Allow a syntax to associate a page with an
application cache, but does
not add that page to the cache. A common feature request also mentioned on
the whatwg list, but it's not getting any engagement from other browser
vendors or the spec writer (which is kind of frustrating). The premise is to
allow pages vended from a server to take advantage of the resources in an
application cache when loading subresources. A perfectly reasonable
request, <http useManifest='x'>.

2. Introduce a new manifest file section to INTERCEPT requests into a prefix
matched url namespace and satisfy them with a cached resource. The resulting
page would be free to interpret the location url and act accordingly based
on the path and query elements beyond the prefix matched url string. This
section would be similar to the FALLBACK section in that prefix matching is
involved, but different in that instead of being used only in the case of a
network/server error, the cached INTERCEPT resource would be used
immediately w/o first going to the server.
  INTERCEPT:
  urlprefix redirect newlocationurl
  urlprefix return cachedresourceurl

Here's where the INTERCEPT namespace could fit into the changes to the
network model.
   if (url is EXPLICITLY_CACHED)  // exact match
     return cached_response;
   if (url is in NETWORK namespace) // prefix match
     return network_response_as_usual;
   if (url is in INTERCEPT namespace) // prefix match <---- this is the new
section
     return handle_intercepted_request_accordingly
   if (url is in FALLBACK namespace) // prefix match
     return network_response_but_fallback_where_needed;
   if (ONLINE_WILDCARD)
     return network_response;
   otherwise
     return synthesized_error_response;

3. Allow an INTERCEPT cached resources to be "executable". Instead of simply
returning the cached resource or redirect in response to the request, load
it into a
background worker context (if not already loaded) and invoke a function in
that context to asynchronously compute response headers and body based on
the request headers (including cookie) and body. The background worker would
have access to various local storage facilities (fileSystem, indexed/sqlDBs)
as well as the ability to make network requests via XHR.
  INTERCEPT:
  urlprefix execute cachedexecutableresourceurl

4. Create a syntax to allow FALLBACK resources to be similarly executable in
a background worker context.

5. Some kind of auto-update policy where the appcache is refreshed w/o the
app running.

There are a couple of features that are not on this list that I want to call
out:

* The ability to add(url) and remove(url) the appcache is not on the list.
FileSystem urls cover a lot of this already, and the ability to cache adhoc
resources and later load them via http urls could be composed out of the
filesystem and executable handlers. Although having add(url) and remove(url)
could probably make for a friendlier API, so maybe this should be squeezed
somewhere inbetween 1 and 3. Or maybe an intercept handler directive could
bind a url prefix to a directory in the origins FileSystem?
   INTERCEPT:
   urlprefix filesystem_temp /path/to/dir/in/temp/web_file_system
   urlprefix filesystem_pers /path/to/dir/in/persistent/...

* A Gears-like 'querymatching' capability is not on the list, executable
handlers subsume that in a much better way.



On Thu, Jun 30, 2011 at 8:22 PM, Felix Halim <felix.halim at gmail.com> wrote:

> On Fri, Jul 1, 2011 at 12:40 AM, timeless <timeless at gmail.com> wrote:
> > It's possible to build a main page so that it can update its content
> > using a subresource. You can use iframes, javascript (including json),
> > xmlhttprequests, or other things to do this.
>
> Those are another option besides using localStorage.
> Again, those things requires restructuring your website.
> I'm looking for a solution that doesn't require modifying anything
> except adding a manifest.
>
>
> > Nothing requires you to have a monolythic main page which is incapable
> > of dynamically updating itself. ... If I visit your page on May 1st
> > and sit there for two months, does your page really just want to
> > continue to show me the same content when I glance at it on July 1st?
> > It can show other content if it wants to, and in order to save
> > bandwidth costs, it should avoid resending the framework which
> > shouldn't be changing. Once your page works well for this case, it
> > should work well for app-cache.
>
> As I said before, separating dynamic from the static will work,
> however, if we don't have "pageStorage", even we have a clean dynamic
> separation, it will quickly run out of space if we use "localStorage"
> since the localStorage quota is per domain.
>
> Let's see an example:
>
> I have a dynamic page with this url:
>
> http://bla/page?id=10
>
> The content inside is changing very frequently, lets say every hour.
> Of course, I want the browser to cache the latest version.
> So, it seemed that AppCache is a perfect fit...
>
> I then add the manifest to enable the App Cache, and what do I get?
>
> Everytime I open that URL every hour, I ALWAYS see the STALE version
> (the 1 hour late version). Then few seconds (or minutes) later (depend
> on when the AppCache gets updated), I refresh, then I got the latest
> content. Annoying, right?
>
> In this case, I better off NOT to use App Cache, since it brings the
> old content everytime.
>
> This is why most people says please "DON'T cache the main page".
>
>
>
> Now, let see the alternative: I build a framework to separate the
> dynamic from the static.
> I have to make it so that only ONE MAIN PAGE get cached by the app cache.
> So, my URL can NO LONGER BE:
>
> http://bla/page?id=10
>
> But it has to change to:
>
> http://bla/page#!id=10
>
> Why do I have to do this? it's because if I DON'T, then each page will
> be stored on different App Cache, and the "stale by one" still occurs!
> That is,
>
> http://bla/page?id=10
>
> and
>
> http://bla/page?id=11
>
> Will be on DIFFERENT AppCache!
>
> In that case, my cleanly separated static and dynamic will have no effect!
> Because all the statics get duplicated for each App Cache.
> It will be the same as if I don't have the framework!
>
> So, to make the AppCache only cache one static framework, I have to
> make my page such that it is served under ONE url:
>
> http://bla/page
>
> Then take the "#!id=10" as non url (or ajax bookmark). This way, the
> AppCache will only cache ONE of my static framework, and MANY dynamic
> content inside it.
>
> Guess what? All the incoming links from other blogs are now broken!
> Of course I can make a redirect, but redirect is AGAINST making the web
> faster!
>
> I think Facebook did the #! thing a while ago, then they abandoned it, why?
>
> Ok now I'm happy with my framework and the redirect, and guess what?
> Soon, I have other pages with #!id=11, #!id=12, ...,  #!id=10000.
> All of them are important and I wan't to cache them and I uses the
> localStorage (or indexedDB) to cache the dynamic content of those
> pages.
> Note that even though the dynamic content is "dynamic" it doesn't mean
> that:
>
> http://bla/page?id=10
>
> has "shared" data with
>
> http://bla/page?id=11
>
> It can be totally different unrelated dynamic content. id=10 dynamic
> content is entirely different from id=11 dynamic content. However,
> since I use localStorage to cache the dynamic content, ALL OF THEM are
> limited to the quota of my domain. My 5MB localStorage domain quota
> will quickly run out of space.
>
> If only I can store the dynamic content into a pageStorage (assuming
> different URL -> including the shebang bookmark has different
> pageStorage), then I won't be running out of storage if I keep one
> page within 5MB. So
>
> http://bla/page#!id=10
>
> has 5 MB pageStorage quota, and
>
> http://bla/page#!id=11
>
> also has 5 MB pageStorage quota, etc...
>
> Then I would be very happy with the new framework.
> Since it will store very compact static App and very compact dynamic
> content.
> It's a win win for everyone, nothing is wasted.
>
> But, if I don't have pageStorage quota, my beautifully separated the
> dynamic from the static framework will be useless since the
> localStorage DOMAIN QUOTA will kill me.
>
>
> So, we have seen how the AppCache fails to satisfy certain usecase and
> how pageStorage is needed to make the alternative solution works.
>
> Here, I propose a solution:  AppCache should COMPLEMENT HTTP Cache so
> that "the main page is not cached" (you know this is not literally
> what it means).
>
> With that solution, I don't have to do ANYTHING on my original site to
> make it work (except adding a manifest to my original page). I can
> still use the old url:
>
> http://bla/page?id=10
>
> It will cache AND SHOW the latest page to the users (just like normal
> web page with HTTP Cache). Then will show the latest cache if the
> network or the server is offline (and perhaps give notification
> header). That's ALL I NEED. This guarantees that my website will still
> be available when the user goes offline.
>
>
> All these discussions only begs to add one feature to AppCache:
> - Only show the cache when the network / server is offline, otherwise,
> show the online version of the page.
>
> The current AppCache doesn't care whether the network/server is online
> or offline, it BLINDLY shows the cache everytime. This is good for the
> default, however, we should HAVE an option to not show the cache if we
> are ONLINE (this is what people meant when they say "DON"T CACHE THE
> MAIN PAGE").
>
> Felix Halim
>