[whatwg] [canvas] request for {create, get, put}ImageDataHD and ctx.backingStorePixelRatio

[Darin Fisher]

On Mon, Apr 16, 2012 at 4:05 PM, Darin Fisher <darin at chromium.org> wrote:

>
>
> On Mon, Apr 16, 2012 at 2:57 PM, Oliver Hunt <oliver at apple.com> wrote:
>
>>
>> On Apr 16, 2012, at 2:34 PM, Darin Fisher <darin at chromium.org> wrote:
>>
>> > On Mon, Apr 16, 2012 at 1:39 PM, Oliver Hunt <oliver at apple.com> wrote:
>> >
>> >>
>> >> On Apr 16, 2012, at 1:12 PM, Darin Fisher <darin at chromium.org> wrote:
>> >>
>> >> Glenn summarizes my concerns exactly.  Deferred rendering is indeed the
>> >> more precise issue.
>> >>
>> >> On Mon, Apr 16, 2012 at 12:18 PM, Oliver Hunt <oliver at apple.com>
>> wrote:
>> >>
>> >>> Could someone construct a demonstration of where the read back of the
>> >>> imagedata takes longer than a runloop cycle?
>> >>>
>> >>
>> >> I bet this would be fairly easy to demonstrate.
>> >>
>> >>
>> >> Then by all means do :D
>> >>
>> >
>> >
>> > Here's an example.
>> >
>> > Take http://ie.microsoft.com/testdrive/Performance/FishIETank/, and
>> apply
>> > the following diff (changing the draw function):
>> >
>> > BEGIN DIFF
>> > --- fishie.htm.orig     2012-04-16 14:23:29.224864338 -0700
>> > +++ fishie.htm  2012-04-16 14:21:38.115489276 -0700
>> > @@ -177,10 +177,17 @@
>> >             // Draw each fish
>> >             for (var fishie in fish) {
>> >                 fish[fishie].swim();
>> >             }
>> >
>> > +
>> > +            if (window.read_back) {
>> > +                var data = ctx.getImageData(0, 0, WIDTH, HEIGHT).data;
>> > +                var x = data[0];  // force readback
>> > +            }
>> > +
>> > +
>> >                        //draw fpsometer with the current number of fish
>> >             fpsMeter.Draw(fish.length);
>> >         }
>> >
>> >         function Fish() {
>> > END DIFF
>> >
>> > Running on a Mac Pro, with Chrome 19 (WebKit @r111385), with 1000 fish,
>> I
>> > get 60 FPS.  Setting read_back to true (using dev tools), drops it down
>> to
>> > 30 FPS.
>> >
>> > Using about:tracing (a tool built into Chrome), I can see that the read
>> > pixels call is taking ~15 milliseconds to complete.  The implied GL
>> flush
>> > takes ~11 milliseconds.
>> >
>> > The page was sized to 1400 x 1000 pixels.
>>
>> How does that compare to going through the runloop -- how long does it
>> take to get from that point to a timeout being called if you do var start =
>> new Date; setTimeout(function() {console.log(new Date - start);}, 0);
>> ?
>>
>
> The answer is ~0 milliseconds.  I know this because without the
> getImageData call, the frame rate is 60 FPS.  The page calls the draw()
> function from an interval timer that has a period of 16.7 milliseconds.
>  The trace indicates that nearly all of that budget is used up prior to the
> getImageData() call that I inserted.
>
>
>
>>
>> This also ignores that possibility that in requesting the data, i
>> probably also want to do some processing on the data, so for the sake of
>> simplicity how long does it take to subsequently iterate through every
>> pixel and set it to 0?
>>
>
> That adds about 44 milliseconds.  I would hope that developers would
> either perform this work in chunks or pass ImageData.data off to a web
> worker for processing.
>

^^^ This got me thinking...

In Chrome at least, getImageData() doesn't actually block to fetch pixels.
 The thread is only blocked when the first dereference of the pixel buffer
occurs.  I believe this is done so that a getImageData() followed by
putImageData() call will not need to block the calling thread.

The above suggests that making getImageData() asynchronous would not
actually provide any benefit for cases where the page does not dereference
the pixel buffer.  Another use case where this comes up is passing the
ImageData to a web worker.  If the web worker is the first to dereference
the ImageData, then only the web worker thread should block.

I think this becomes an argument for keeping getImageData() as is.  It
assumes that ImageData is just a handle, and we could find another way to
discourage dereferencing the pixel buffer on the UI thread.

Hmm...

-Darin




>
>
>>
>> Remember the goal of making this asynchronous is to improve performance,
>> so the 11ms of drawing does have to occur at some point, you're just hoping
>> that by making things asynchronous you can mask that.  But I doubt you
>> would see an actual improvement in wall clock performance.
>>
>
> The 11 ms of drawing occurs on a background thread.  Yes, that latency
> exists, but it doesn't have to block the main thread.
>
> Let me reiterate the point I made before.  There can be multiple web pages
> sharing the same main thread.  (Even in Chrome this can be true!)  Blocking
> one web page has the effect of blocking all web pages that share the same
> main thread.
>
> It is not nice for one web page to jank up the browser's main thread and
> as a result make other web pages unresponsive.
>
>
>
>>
>> I also realised something else that I had not previously considered -- if
>> you're doing bitblit based sprite movement the complexity goes way up if
>> this is asynchronous.
>
>
> I don't follow.  Can you clarify?
>
> Thanks,
> -Darin
>