[whatwg] Web Storage: apparent contradiction in spec

[Michael Nordman]

> At a minimum the HTML 5 spec should be silent on how user agents implement
local storage policies.
+1 linus

The sort of 'policies' being discussed are feeling 'out-of-scope' for the
HTML5 spec. The practical realities are that eviction needs to be there in
some form w/o an explicit user 'installation' step.

Something that could be appropriately specified is the grain size of local
data eviction...

* Can an individual <key,value> pair for an origin be removed from local
storage while leaving other local data in place?

* Can an individual Database be deleted for an origin while leaving other
 local data in place?

* Can an individual Manifest be deleted for an origin while leaving other
local data in place?

* Or should an origin's data be subject to all-or-none eviction.

I would prefer to see the spec clarify questions along those lines. That
would be useful.

A mandate from on high that says 'shall store forever and ever' will be
promptly ignored because its impossible to make that guarantee.



On Wed, Aug 26, 2009 at 4:01 PM, Linus Upson <linus at google.com> wrote:

> Not convinced. :)
>
> 1. Analogies
>
> The analogy was made comparing a user agent that purges local storage to an
> OS throwing out files without explicit user action. This is misleading since
> most files arrive on your computer's disk via explicit user action. You copy
> files to your disk by downloading them from the internet, copying from a
> network drive, from a floppy, your camera, etc. You put them on your disk
> and you are responsible for removing them to reclaim space.
>
> There are apps that create files in hidden places such as:
>
> C:\Documents and Settings\linus\Local Settings\Application
> Data\Google\Chrome\User Data
>
> If those apps do not manage their space carefully, users get annoyed. If
> such an app filled the user's disk they would have no idea what consumed the
> space or how to reclaim it. They didn't put the files there. How are they
> supposed to know to remove them? Most users have no idea that Local Settings
> exists (it is hidden), much less how to correctly manage any files they
> find.
>
> A better analogy would be, "What if watching TV caused 0-5MB size files to
> silently be created from time to time in a hidden folder on your computer,
> and when your disk filled up both your TV and computer stopped working?"
>
> Lengthy discussion on cleaning up hidden resources (persistent background
> content) here:
> http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2009-July/021421.html
>
> 2. Attack
>
> Without automatic space management the local storage consumed will grow
> without bound. I'm concerned that even without an intentional DOS attack
> users are going to be unhappy about their shrinking disks and not know what
> to do about it. The problem is worse on phones.
>
> Things get worse still if a griefer wants to make a point about the
> importance of keeping web browsers logically stateless. Here's how such an
> attack could be carried out:
>
> 2a. Acquire a bunch of unrelated domains from a bunch of registrars using
> stolen credit cards. Skip this step if UAs don't group subdomains under the
> same storage quota. For extra credit pick names that are similar to
> legitimate sites that use local storage.
>
> 2b. Start up some web hosting accounts. Host your attack code here. If they
> aren't free, use stolen credit cards.
>
> 2c. Buy ads from a network that subsyndicates from a network that
> subsyndicates from a major ad network that allows 3rd party ad serving.
> There are lots to choose from. No money? Stolen credit cards. Serve the ads
> from your previously acquired hosting accounts.
>
> 2d. Giggle. The user will be faced with the choice of writing off the
> space, deleting everything including their precious data, or carefully
> picking though tens of thousands of entries to find the few domains that
> hold precious content. User gets really unhappy if the attack managed to
> fill the disk.
>
> 3. Ingcognito / Private Browsing
>
> Chrome's Incognito mode creates a temporary, in-memory profile. Local
> storage operations will work, but nothing will be saved after the Incognito
> window is closed. Safari takes a different approach and causes local storage
> operations to fail when in Private Browsing mode. Some sites won't work in
> Private Browsing. I don't recall what Firefox or IE do. Pick your poison.
>
> Lengthy discussion here:
> http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2009-April/019238.html
>
> 4. Cache Eviction Algorithms
>
> At a minimum the HTML 5 spec should be silent on how user agents implement
> local storage policies. I would prefer the spec to make it clear that local
> storage is a cache, domains can use up to 5MB of space without interrupting
> the user, and that UAs were free to implement varying cache eviction
> algorithms.
>
> Some browsers may provide interface to allow users to specify precious
> local storage, some may not. Eviction policies for installed extensions may
> be different than those for web pages. Quotas for extensions may be
> different than that for web pages. Non-browser UAs such as Dashboard, AIR,
> etc. may have different policies.
>
> If the spec requires UAs to maintain local storage as 'precious' it will be
> the first such feature in HTML 5. Everything else in the spec is treated as
> volatile.
>
> Linus
>
>
> On Tue, Aug 25, 2009 at 4:36 PM, Jeremy Orlow <jorlow at chromium.org> wrote:
>
>> On Tue, Aug 25, 2009 at 4:18 PM, Brady Eidson <beidson at apple.com> wrote:
>>
>>>
>>> On Aug 25, 2009, at 3:51 PM, Jeremy Orlow wrote:
>>>
>>> On Tue, Aug 25, 2009 at 3:19 PM, Aaron Boodman <aa at google.com> wrote:
>>>
>>>> On Tue, Aug 25, 2009 at 2:44 PM, Jeremy Orlow<jorlow at chromium.org>
>>>> wrote:
>>>>
>>>> Extensions are an example of an application that is less cloud-based.
>>>> It would be unfortunate and weird for extension developers to have to
>>>> worry about their storage getting tossed because the UA is running out
>>>> of disk space.
>>>
>>>
>>> Extensions are pretty far out of scope of the spec (at least for now),
>>> right?  (Within Chrome, we can of course special case this.)
>>>
>>>
>>> The current spec is about "Web Applications" of all forms, including
>>> those that are offline, and others that hope to break from from the
>>> *required* chain to the cloud.
>>>
>>> Extensions based on web technology are just one form of this.
>>>  Widgets/gadgets are another.  Stand alone web applications are yet another.
>>>  Native applications that integrate HTML for their UI are another still.
>>>
>>> On Tue, Aug 25, 2009 at 3:09 PM, Jens Alfke <snej at google.com> wrote:
>>>
>>>> Interesting comments. Linus and Jeremy appear to be coming at this from
>>>> a pure "cloud" perspective, where any important or persistent data is kept
>>>> on a remote server and the browser, so local storage can be treated as
>>>> merely a cache. That's definitely a valid position, but from my perspective,
>>>> much of the impetus for having local storage is to be able to support *
>>>> other* application models, where important data is stored locally. If
>>>> browsers are free to dispose HTML5 local storage without the user's informed
>>>> consent, such applications become dangerously unreliable.
>>>> For example, Linus wrote:
>>>>
>>>> User agents need to be free to garbage collect any local state. If they
>>>> can't then attackers (or the merely lazy) will be able to fill up the user's
>>>> disk. We can't expect web sites or users to do the chore of taking out the
>>>> garbage.
>>>>
>>>>
>>>> Replace "user agent" -> "operating system" and "local state" -> "user
>>>> files", and you have an argument that, when the hard disk in my MacBook gets
>>>> too full, the OS should be free to start randomly deleting my local files to
>>>> make room. This would be a really bad idea.
>>>>
>>>
>>> Well, it's certainly different from what we're used to.  I'm not
>>> convinced it's wrong though.  The web has gotten by pretty well with such a
>>> model so far.
>>>
>>>
>>> But behind the scenes, developers have shoehorned their own data storage
>>> solutions in to place because there hasn't been a good solution in place.
>>>
>>> Why should an app that is largely about client side experience have to
>>> store user preferences in cookies and hope they won't be purged, or load a
>>> plug-in that has reliable local storage, or sync preferences over the cloud
>>> to a server?
>>>
>>>
>>>  Similar analogies —
>>>> • If the SD card in my Wii fills up, should the system automatically
>>>> start deleting saved games?
>>>> • If my iPhone's Flash disk gets full, should it start deleting photos?
>>>> What if I haven't synced those photos to my iTunes yet?
>>>>
>>>> In each of those cases, what the device actually does is warns you about
>>>> the lack of free space, and lets you choose what to get rid of.
>>>>
>>>
>>> It's worth noting that today, OSs do a pretty poor job of helping you
>>> with this task.  (I don't see any reason why the spec will prohibit UAs from
>>> creating a good UI for this, though.)
>>>
>>>
>>> I completely agree OSs do a pretty poor job of helping with the task.
>>>  Browsers might be an innovating space here.  I challenge you to come up
>>> with a great UI for this that shows up in a UA.  I challenge the WHATWG to
>>> not decide that deleting user data is okay because it's the easiest way
>>> out.
>>>
>>>
>>>
>>>> Local storage is different from cloud storage. The HTML5 storage API can
>>>> be used for both, so it shouldn't be limited to what's convenient for just
>>>> one of them.
>>>>
>>>
>>> I still don't understand what use local storage has outside of 'cloud
>>> storage'.  Even in the extensions use case (which I think is out of scope
>>> for this spec), there's no reason you can't sync user preferences and such
>>> to the cloud.
>>>
>>>
>>> Once thing I think that HTML5 has made clear is that "web technologies"
>>> are no longer exclusively about "web sites" that exist solely "in the
>>> cloud."  Widgets/gadgets, html-based extensions, offline web applications,
>>> and native applications that use HTML/JS/CSS to embed parts of their UI are
>>> *all* covered by HTML5, and I don't think requiring the cloud for any of
>>> them is necessary.
>>>
>>> Also - and I don't mean this to be flippant, I raise it as a serious
>>> point - not all web application developers are Google or Apple with access
>>> to a server infrastructure.  To many web developers, just "throwing data up
>>> on a server somewhere" is outside the constraints of their resources or
>>> their design.
>>>
>>> The cloud is within the scope of web technologies, but web technologies
>>> should not rely on the cloud.
>>>
>>> By the way, in case it's not clear, my position is not that UAs should
>>> take deleting user information lightly, my position is 1) this behavior
>>> should be left up to the UA and 2) when possible, developers should keep
>>> information in the cloud not local storage.
>>>
>>>
>>> Don't take my hyperboles too seriously - I don't *seriously* think that
>>> anyone is suggested browsers should be light hearted about deleting user
>>> data.
>>> I think our positions are all pretty clear, and it's coming down to this
>>> differing philosophy.
>>>
>>> But my position is:
>>> 1) If this behavior is left up to the UA, then developers who are using
>>> web technologies to write applications and they don't want to have to worry
>>> about "the cloud" for their data are out of luck, because *one* browser that
>>> is willing to delete data behind the scenes ruins their reliable picture of
>>> the technology.
>>> 2) Developers should not be *forced* into using the cloud when it is not
>>> within the scope of what they're trying to develop.
>>>
>>
>> I still think there are non-trivial downsides to treating local storage
>> (and presumable database) data as "sacred", but I guess I'm convinced that
>> it's the correct way to go.  I hate throwing UI at problems, but it really
>> does make a good deal of sense in this case.  Heuristics will still be
>> helpful in that we can suggest what to delete to the user, but I guess the
>> user should always make the final decision.
>>
>> Linus, are you convinced?
>>
>> J
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.whatwg.org/pipermail/whatwg-whatwg.org/attachments/20090826/a71958b2/attachment-0002.htm>