<div class="gmail_quote">On Thu, Sep 24, 2009 at 11:57 PM, Jonas Sicking <span dir="ltr"><jonas@sicking.cc></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div><div></div><div class="h5">On Thu, Sep 24, 2009 at 9:04 PM, Darin Fisher <<a href="mailto:darin@chromium.org">darin@chromium.org</a>> wrote:<br>
> On Thu, Sep 24, 2009 at 4:43 PM, Jonas Sicking <jonas@sicking.cc> wrote:<br>
>><br>
>> On Thu, Sep 24, 2009 at 10:52 AM, Darin Fisher <<a href="mailto:darin@chromium.org">darin@chromium.org</a>> wrote:<br>
>> > On Thu, Sep 24, 2009 at 10:40 AM, Jonas Sicking <jonas@sicking.cc><br>
>> > wrote:<br>
>> >><br>
>> >> On Thu, Sep 24, 2009 at 1:17 AM, Darin Fisher <<a href="mailto:darin@chromium.org">darin@chromium.org</a>><br>
>> >> wrote:<br>
>> >> > On Thu, Sep 24, 2009 at 12:20 AM, Jonas Sicking <jonas@sicking.cc><br>
>> >> > wrote:<br>
>> >> >><br>
>> >> >> On Wed, Sep 23, 2009 at 10:19 PM, Darin Fisher <<a href="mailto:darin@chromium.org">darin@chromium.org</a>><br>
>> >> >> wrote:</div></div></blockquote><div><br></div><div>... snip ...</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div><div class="h5">
>> >> >> > multi-core is the future. what's the opposite of fine-grained<br>
>> >> >> > locking?<br>
>> >> >> > it's not good ;-)<br>
>> >> >> > the implicit locking mechanism as spec'd is super lame.<br>
>> >> >> > implicitly<br>
>> >> >> > unlocking under<br>
>> >> >> > mysterious-to-the-developer circumstances! how can that be a good<br>
>> >> >> > thing?<br>
>> >> >> > storage.setItem("y",<br>
>> >> >> > function_involving_implicit_unlocking(storage.getItem("x")));<br>
>> >> >><br>
>> >> >> I totally agree on all points. The current API has big<br>
>> >> >> imperfections.<br>
>> >> >> However I haven't seen any workable counter proposals so far, and I<br>
>> >> >> honestly don't believe there are any as long as our goals are:<br>
>> >> >><br>
>> >> >> * Don't break existing users of the current implementations.<br>
>> >> >> * Don't expose race conditions to the web.<br>
>> >> >> * Don't rely on authors getting explicit locking mechanisms right.<br>
>> >> >><br>
>> >> ><br>
>> >> > The current API exposes race conditions to the web. The implicit<br>
>> >> > dropping of the storage lock is that. In Chrome, we'll have to drop<br>
>> >> > an existing lock whenever a new lock is acquired. That can happen<br>
>> >> > due to a variety of really odd cases (usually related to nested loops<br>
>> >> > or nested JS execution), which will be difficult for developers to<br>
>> >> > predict, especially if they are relying on third-party JS libraries.<br>
>> >> > This issue seems to be discounted for reasons I do not understand.<br>
>> >><br>
>> >> I don't believe we've heard about this before, so that would be the<br>
>> >> reason it hasn't been taken into account.<br>
>> >><br>
>> >> So you're saying that chrome would be unable implement the current<br>
>> >> storage mutex as specified in spec? I.e. one that is only released at<br>
>> >> the explicit points that the spec defines? That seems like a huge<br>
>> >> problem.<br>
>> ><br>
>> > No, no... my point is that to the application developer, those<br>
>> > "explicit"<br>
>> > points will appear quite implicit and mysterious. This is why I called<br>
>> > out third-party JS libraries. One day, a function that you are using<br>
>> > might transition to scripting a plugin, which might cause a nested<br>
>> > loop, which could then force the lock to be released. As a programmer,<br>
>> > the unlocking is not explicit or predictable.<br>
>><br>
>> Ah, indeed, this is a problem. However the unfortunate fact remains<br>
>> that so far no other workable solution has been proposed.<br>
><br>
> OK, so we agree that the current solution doesn't meet the goals you<br>
> stated above :-(<br>
<br>
</div></div>Well, it addresses them as long as users are aware of the risk, and<br>
properly document weather their various library functions will release<br>
the lock or not. However I agree that it's unlikely that they will do<br>
so correctly.</blockquote><div><br></div><div><div>I thought the point of not having lock APIs was that users shouldn't have</div><div>to understand locks ;-) The issue I've raised here is super subtle. We</div>
<div>have not succeeded in avoiding subtlety!</div></div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div class="im">
>> > Moreover, there are other examples which have been discussed on the<br>
>> > list. There are some DOM operations that can result in a frame<br>
>> > receiving<br>
>> > a DOM event synchronously. That can result in a nesting of storage<br>
>> > locks,<br>
>> > which can force us to have to implicitly unlock the outermost lock to<br>
>> > avoid<br>
>> > deadlocks. Again, the programmer will have very poor visibility into<br>
>> > when<br>
>> > these things can happen.<br>
>><br>
>> So far I don't think it has been shown that these events need to be<br>
>> synchronous. They all appear to be asynchronous in gecko, and in the<br>
>> case of different-origin frames, I'm not even sure there's a way for<br>
>> pages to detect if the event was fired asynchronously or not.<br>
><br>
> IE and WebKit dispatch some of them synchronously. It's hard to say which<br>
> is correct or if it causes any web compat isues. I'm also not sure that we<br>
> have covered all of the cases.<br>
<br>
</div>It still seems to me that it's extremely unlikely that pages depend on<br>
cross origin events to fire synchronously. I can't even think of a way<br>
to test if a browser dispatches these events synchronously or not. Can<br>
you?</blockquote><div><br></div><div><div><br></div><div>i agree that it seems uncommon. maybe there could be some odd app that</div><div>does something after resizing an iframe that could be dependent on the</div><div>
event handler setting some data field. this kind of thing is probably even</div><div>less common in the cross-origin case.</div></div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class="im">
> Our approach to implementing implicit locking (if we must) will be to detect<br>
> nested locking, and simply unlock the first held lock to basically prevent<br>
> nested locking. This way we don't have to know all of the cases where this<br>
> can happen.<br>
<br>
</div>So is this the plan even for the WebDatabase API? I.e. a page starts a<br>
transaction, and thereby grab a lock on the database, and then call<br>
some function that requires the UA to grab another lock, will you at<br>
that point release the lock? And if so, will you commit or release the<br>
transaction?<br>
<div><div></div><div class="h5"><br></div></div></blockquote><div><br></div><div>no. for database, we can just defer the second transaction. this is what</div><div><div>i would like to be able to do with local storage, but i cannot since local</div>
<div>storage is synchronous.</div><div><br></div><div><br></div></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div><div class="h5">
>> >> >> But, as imperfect as the current API is, I think the following is a<br>
>> >> >> decent way forward:<br>
>> >> >><br>
>> >> >> * Allow pages that want the convenience of localStorage to use it.<br>
>> >> >> For<br>
>> >> >> multi-process browsers this will mean poor UI *for pages that use<br>
>> >> >> localStorage*. Especially when said pages hold on to localStorage<br>
>> >> >> for<br>
>> >> >> a long time.<br>
>> >> >> * Add alternative APIs that don't suffer from the same problems.<br>
>> >> >> More<br>
>> >> >> below.<br>
>> >> >><br>
>> >> >> >> > In addition, this argument assumes that Microsoft (and other<br>
>> >> >> >> > UAs)<br>
>> >> >> >> > will<br>
>> >> >> >> > implement the structured clone version of LocalStorage. Has<br>
>> >> >> >> > anyone<br>
>> >> >> >> > (or<br>
>> >> >> >> > can<br>
>> >> >> >> > anyone) from Microsoft comment on this?<br>
>> >> >> >><br>
>> >> >> >> Given that I've never heard microsoft commit to a webstandard,<br>
>> >> >> >> ever,<br>
>> >> >> >> I<br>
>> >> >> >> doubt that we'll hear anything here. Or that the lack of hearing<br>
>> >> >> >> anything means we can draw any conclusions.<br>
>> >> >> >><br>
>> >> >> >> > This is not a small feature to add. Yes, it's smaller than<br>
>> >> >> >> > creating<br>
>> >> >> >> > a<br>
>> >> >> >> > new<br>
>> >> >> >> > storage mechanism (that everyone is willing to adopt), but I<br>
>> >> >> >> > still<br>
>> >> >> >> > think<br>
>> >> >> >> > that's what we should be looking at. Rather than polishing a<br>
>> >> >> >> > turd.<br>
>> >> >> >><br>
>> >> >> >> I do think that localStorage is a decent API that developers will<br>
>> >> >> >> want<br>
>> >> >> >> to, and should, use. I think looking into adding a async accessor<br>
>> >> >> >> to<br>
>> >> >> >> get a storage object so that people can use an localStorage-like<br>
>> >> >> >> API<br>
>> >> >> >> while avoiding risks of blocking. This would also allow sharing<br>
>> >> >> >> data<br>
>> >> >> >> between worker threads and the main window.<br>
>> >> >> ><br>
>> >> >> > i think the async callback to get a storage object is an<br>
>> >> >> > improvement,<br>
>> >> >> > but<br>
>> >> >> > i'm not sure that it addresses all of the problems. for example,<br>
>> >> >> > if<br>
>> >> >> > a<br>
>> >> >> > worker<br>
>> >> >> > wants to read values from storage, compute, and then put a value<br>
>> >> >> > into<br>
>> >> >> > storage, it would probably do all of this from the storage<br>
>> >> >> > callback.<br>
>> >> >> > that<br>
>> >> >> > would result in holding the lock for a long time, which would lock<br>
>> >> >> > out<br>
>> >> >> > any<br>
>> >> >> > other threads, including non-worker threads.<br>
>> >> >> > the problem here is that localStorage is a pile of global<br>
>> >> >> > variables.<br>
>> >> >> > we<br>
>> >> >> > are<br>
>> >> >> > trying to give people global variables without giving them tools<br>
>> >> >> > to<br>
>> >> >> > synchronize<br>
>> >> >> > access to them. the claim i've heard is that developers are not<br>
>> >> >> > savy<br>
>> >> >> > enough<br>
>> >> >> > to use those tools properly. i agree that developers tend to use<br>
>> >> >> > tools<br>
>> >> >> > without<br>
>> >> >> > fully understanding them. ok, but then why are we giving them<br>
>> >> >> > global<br>
>> >> >> > variables?<br>
>> >> >> > there has to be a better answer.<br>
>> >> >><br>
>> >> >> I actually described an potential solution in the thread on worker<br>
>> >> >> storage.<br>
>> >> >><br>
>> >> >> The problem you describe is a worker holding on the the storage for<br>
>> >> >> an<br>
>> >> >> very long (indefinite) time, thereby locking out other<br>
>> >> >> threads/windows<br>
>> >> >> from accessing the same storage area. This seems inevitable if we<br>
>> >> >> want<br>
>> >> >> to prevent race conditions while at the same time not forcing the<br>
>> >> >> complexities of locks onto web developers. The WebDatabase API<br>
>> >> >> suffers<br>
>> >> >> from exactly the same problem.<br>
>> >> ><br>
>> >> > Hmm... are you saying that from the SQLStatementCallback used to read<br>
>> >> > some data out of the database, you might compute on that data, and<br>
>> >> > then<br>
>> >> > issue an executeSql call to write a computed result, and that in this<br>
>> >> > scenario,<br>
>> >> > the fact that it is the same transaction means that other threads are<br>
>> >> > locked<br>
>> >> > out of accessing the same database? I hadn't considered chaining<br>
>> >> > executeSql<br>
>> >> > calls like this to keep the transaction alive. Hmm...<br>
>> >><br>
>> >> Indeed.<br>
>> >><br>
>> >> >> However, we can lessen the problem. By adding multiple storage<br>
>> >> >> areas,<br>
>> >> >> we can allow a worker to use one storage area, while allowing other<br>
>> >> >> parties to simultaneously use other storage areas. This way, if a<br>
>> >> >> worker and a window aren't sharing data at all, they never get in<br>
>> >> >> the<br>
>> >> >> way of each other.<br>
>> >> >><br>
>> >> >> So a very simplistic design would be something like the following:<br>
>> >> >><br>
>> >> >> getStorageArea(name, callback)<br>
>> >> >><br>
>> >> >> when called will asynchronously call the callback parameter once the<br>
>> >> >> storage area named by the first parameter becomes available. The<br>
>> >> >> callback receives the storage area as an argument. We would also<br>
>> >> >> have<br>
>> >> >> the function<br>
>> >> >><br>
>> >> >> getMultipleStorageAreas(names, callback)<br>
>> >> >><br>
>> >> >> Same as above, but names is an array of strings indicating multiple<br>
>> >> >> storage areas that need to be acquired before the callback is<br>
>> >> >> called.<br>
>> >> >> The callback receives all the areas in an array as an argument. This<br>
>> >> >> function allows transferring data between multiple storage areas<br>
>> >> >> without risking racing.<br>
>> >> >><br>
>> >> >> There's several problems with this, such as the names are sort of<br>
>> >> >> crappy, and that getting storage areas an array isn't very friendly.<br>
>> >> >> However you get the basic idea.<br>
>> >> >><br>
>> >> >> We don't even need to use Storage objects for this. In fact, I hope<br>
>> >> >> mozilla will in a not too distant future come up with an alternative<br>
>> >> >> proposal to the WebDatabase SQL API. Something like this might fit<br>
>> >> >> into such a proposal as I think that'll have multiple separate<br>
>> >> >> storage<br>
>> >> >> areas anyway.<br>
>> >> >><br>
>> >> >> / Jonas<br>
>> >> ><br>
>> >> ><br>
>> >> > Maybe we should just invent a similar transaction method for<br>
>> >> > name/value<br>
>> >> > storage? Wouldn't that be better than inventing a new idiom?<br>
>> >> > Ideally,<br>
>> >> > we'd also make reads and writes on storage be asynchronous. The<br>
>> >> > transaction would then be usable to hold the lock across multiple<br>
>> >> > asynchronous reads and writes. Since local storage is backed by<br>
>> >> > disk,<br>
>> >> > it seems like a more ideal local storage API would not<br>
>> >> > require synchronous<br>
>> >> > filesystem access.<br>
>> >><br>
>> >> Not quite following what you're suggesting, but there's lots of ways<br>
>> >> to design this. The critical part is to allow grabbing (with<br>
>> >> associated locking) of just a subset of the available storage space.<br>
>> >><br>
>> >> / Jonas<br>
>> ><br>
>> ><br>
>> > I was suggesting that we only provide asynchronous getItem / setItem<br>
>> > calls,<br>
>> > where<br>
>> > each call is parameterized by a transaction. This is how database<br>
>> > works.<br>
>><br>
>> Not quite sure I follow your proposal. How would you for example<br>
>> increase the value of a property by one without risking race<br>
>> conditions? Or keep two values in different properties in sync? I.e.<br>
>> so that if you update one always update the other, so that they never<br>
>> have different values.<br>
>><br>
>> / Jonas<br>
><br>
><br>
> Easy. Just like with database, the transaction is the storage lock. Any<br>
> storage<br>
> operation performed on that transaction are done atomically. However, all<br>
> storage<br>
> operations are asynchronous. You basically string together asynchronous<br>
> storage<br>
> operations by using the same transaction for each.<br>
> We could add methods to get/set multiple items at once to simplify life for<br>
> the coder.<br>
<br>
</div></div>I think I still don't understand your proposal, could you give some<br>
code examples?<br><font class="Apple-style-span" color="#888888"><br></font></blockquote><div><br></div><div><br></div><div><div>ripping off database:</div><div><br></div><div>interface ValueStorage {</div><div> void transaction(in DOMString namespace, in ValueStorageTransactionCallback callback);</div>
<div>};</div><div>interface ValueStorageTransactionCallback {</div><div> void handleEvent(in ValueStorageTransaction transaction);</div><div>};</div><div>interface ValueStorageTransaction {</div><div> void readValue(in DOMString name, in ValueStorageReadCallback callback);</div>
<div> void writeValue(in DOMString name, in DOMString value);</div><div>};</div><div>interface ValueStorageReadCallback {</div><div> void handleEvent(in ValueStorageTransaction transaction, in DOMString value);</div><div>
};</div><div><br></div><div>then, to use these interfaces, you could implement thread-safe increment:</div><div><br></div><div>window.localStorage.transaction("slice", function(transaction) {</div><div> transaction.readValue("foo", function(transaction, fooValue) {</div>
<div> transaction.writeValue("foo", ++fooValue);</div><div> })</div><div>})</div><div><br></div><div>to fetch multiple values, you could do this:</div><div><br></div><div>var values = [];</div><div>var numValues = 10;</div>
<div>function readNextValue(transaction) {</div><div> if (values.length == numValues)</div><div> return; // done!</div><div> var index = values.length;</div><div> transaction.readValue("value" + index, function(transaction, value) {</div>
<div> values.push(value);</div><div> readNextValue(transaction);</div><div> })</div><div>}</div><div>window.localStorage.transaction("slice", readNextValue);</div><div><br></div><div>This has the property that all IO is non-blocking and the "lock" is held only</div>
<div>for a very limited scope. The programmer is however free to extend the</div><div>life of the lock as needed.</div><div><br></div><div>-Darin</div></div><div> </div></div>