[whatwg] [canvas] request for {create, get, put}ImageDataHD and ctx.backingStorePixelRatio

[Darin Fisher]

On Sun, Apr 22, 2012 at 6:03 PM, Maciej Stachowiak <mjs at apple.com> wrote:

>
> On Apr 20, 2012, at 6:53 AM, Glenn Maynard wrote:
>
> On Thu, Apr 19, 2012 at 11:28 PM, Maciej Stachowiak <mjs at apple.com> wrote:
>
>> You could also address this by adding a way to be notified when the
>> contents of an ImageData are available without blocking. That would work
>> with both vanilla getImageData and the proposed getImageDataHD. It would
>> also give the author the alternative of just blocking (e.g. if they know
>> the buffer is small) or of sending the data off to a worker for processing.
>>
>
> This would result in people writing poor code, based on incorrect
> assumptions.  It doesn't matter how big the buffer is; all that matters is
> how long the drawing calls before the getImageData take.  For example, if
> multiple canvases are being drawn to (eg. on other pages running in the
> same thread), they may share a single drawing queue.
>
> Any time you retrieve image data synchronously, and it happens to require
> a draw flush, you freeze the UI for all pages sharing that thread.  Why is
> that okay for people to do?  We should know better by now than to expose
> APIs that encourage people to block the UI thread, after spending so much
> time trying to fix that mistake in early APIs.
>
> (This should expose a synchronous API in workers if and when Canvas makes
> it there, of course, just like all other APIs.)
>
>
> All JavaScript that runs on the main thread has the potential to "freeze
> the UI for all pages sharing that thread". One can imagine models that
> avoid this by design - for example, running all JavaScript on one or more
> threads separate from the UI thread. But from where we are today, it's not
> practical to apply such a solution. It's also not practical to make every
> API asynchronous - it's just too hard to code that way.
>
> In light of this, we need some sort of rule for what types of APIs should
> only be offered in asynchronous form on the main thread. Among the major
> browser vendors, there seems to be a consensus that this should at least
> include APIs that do any network or disk I/O. Network and disk are slow
> enough and unpredictable enough that an author could never correctly judge
> that it's safe to do synchronous I/O.
>
> Some feel that a call that reads from the GPU may also be in this category
> of "intrinsically too slow/unpredictable". However, we are talking about
> operations with a much lower upper bound on their execution time. We're
> also talking about an operation that has existed in its synchronous form
> (getImageData) for several years, and we don't have evidence of the types
> of severe problems that, for instance, synchronous XHR has been known to
> cause. Indeed, the amount of trouble caused is low enough that no one has
> yet proposed or implemented an async version of this API.
>

The point is not about whether the jank introduced by GPU readbacks is
emergency level.  The point is that it can be costly, and it can interfere
greatly with having an interactive main thread.  If you assume a goal of 60
FPS, then smallish jank can be killer.  It is common for new GL programmers
to call glGetError too often for example, and that can kill the performance
of the app.  Of course this is no where near as bad as synchronous XHR.  It
doesn't have to be at that level to be a problem.  I think it is fair to
focus on 60 FPS as a goal in other words.

That said, I've come around to being OK with getImageDataHD.  As I wrote
recently, this is because it is possible to implement that in a
non-blocking fashion.  It can just queue up a readback.  It only becomes
necessary to block the calling thread when a pixel is dereferenced.  This
affords developers with an opportunity to instead pass the ImageData off to
a web worker before dereferencing.  Hence, the main thread should not jank
up.  This of course requires developers to be very smart about what they
are doing, and for browsers to be smart too.

I'm still sad that getImageData{HD} makes it easy for bad code in one web
page to screw over other web pages.  The argument that this is easy to do
anyways with long running script is a cop out.  We should guide developers
to do the right thing in this cooperatively multi-tasking system.

-Darin



>
> If adding an async version has not been an emergency so far, then I don't
> think it is critical enough to block adding scaled backing store support.
> Nor am I convinced that we need to deprecate or phase out the synchronous
> version. Perhaps future evidence will change the picture, but that's how it
> looks to me so far.
>
> Regards,
> Maciej
>
>