[whatwg] [canvas] request for {create, get, put}ImageDataHD and ctx.backingStorePixelRatio

[Oliver Hunt]

On Apr 17, 2012, at 3:32 PM, Darin Fisher <darin at chromium.org> wrote:

> 
> 
> On Mon, Apr 16, 2012 at 4:05 PM, Darin Fisher <darin at chromium.org> wrote:
> 
> 
> On Mon, Apr 16, 2012 at 2:57 PM, Oliver Hunt <oliver at apple.com> wrote:
> 
> On Apr 16, 2012, at 2:34 PM, Darin Fisher <darin at chromium.org> wrote:
> 
> > On Mon, Apr 16, 2012 at 1:39 PM, Oliver Hunt <oliver at apple.com> wrote:
> >
> >>
> >> On Apr 16, 2012, at 1:12 PM, Darin Fisher <darin at chromium.org> wrote:
> >>
> >> Glenn summarizes my concerns exactly.  Deferred rendering is indeed the
> >> more precise issue.
> >>
> >> On Mon, Apr 16, 2012 at 12:18 PM, Oliver Hunt <oliver at apple.com> wrote:
> >>
> >>> Could someone construct a demonstration of where the read back of the
> >>> imagedata takes longer than a runloop cycle?
> >>>
> >>
> >> I bet this would be fairly easy to demonstrate.
> >>
> >>
> >> Then by all means do :D
> >>
> >
> >
> > Here's an example.
> >
> > Take http://ie.microsoft.com/testdrive/Performance/FishIETank/, and apply
> > the following diff (changing the draw function):
> >
> > BEGIN DIFF
> > --- fishie.htm.orig     2012-04-16 14:23:29.224864338 -0700
> > +++ fishie.htm  2012-04-16 14:21:38.115489276 -0700
> > @@ -177,10 +177,17 @@
> >             // Draw each fish
> >             for (var fishie in fish) {
> >                 fish[fishie].swim();
> >             }
> >
> > +
> > +            if (window.read_back) {
> > +                var data = ctx.getImageData(0, 0, WIDTH, HEIGHT).data;
> > +                var x = data[0];  // force readback
> > +            }
> > +
> > +
> >                        //draw fpsometer with the current number of fish
> >             fpsMeter.Draw(fish.length);
> >         }
> >
> >         function Fish() {
> > END DIFF
> >
> > Running on a Mac Pro, with Chrome 19 (WebKit @r111385), with 1000 fish, I
> > get 60 FPS.  Setting read_back to true (using dev tools), drops it down to
> > 30 FPS.
> >
> > Using about:tracing (a tool built into Chrome), I can see that the read
> > pixels call is taking ~15 milliseconds to complete.  The implied GL flush
> > takes ~11 milliseconds.
> >
> > The page was sized to 1400 x 1000 pixels.
> 
> How does that compare to going through the runloop -- how long does it take to get from that point to a timeout being called if you do var start = new Date; setTimeout(function() {console.log(new Date - start);}, 0);
> ?
> 
> The answer is ~0 milliseconds.  I know this because without the getImageData call, the frame rate is 60 FPS.  The page calls the draw() function from an interval timer that has a period of 16.7 milliseconds.  The trace indicates that nearly all of that budget is used up prior to the getImageData() call that I inserted.
> 
>  
> 
> This also ignores that possibility that in requesting the data, i probably also want to do some processing on the data, so for the sake of simplicity how long does it take to subsequently iterate through every pixel and set it to 0?
> 
> That adds about 44 milliseconds.  I would hope that developers would either perform this work in chunks or pass ImageData.data off to a web worker for processing.
> 
> ^^^ This got me thinking...
> 
> In Chrome at least, getImageData() doesn't actually block to fetch pixels.  The thread is only blocked when the first dereference of the pixel buffer occurs.  I believe this is done so that a getImageData() followed by putImageData() call will not need to block the calling thread.
> 
> The above suggests that making getImageData() asynchronous would not actually provide any benefit for cases where the page does not dereference the pixel buffer.  Another use case where this comes up is passing the ImageData to a web worker.  If the web worker is the first to dereference the ImageData, then only the web worker thread should block.
> 
> I think this becomes an argument for keeping getImageData() as is.  It assumes that ImageData is just a handle, and we could find another way to discourage dereferencing the pixel buffer on the UI thread.
> 
> Hmm...

A long time ago I and Dmitry tried to get canvas to be available on a worker thread, and then through some bizarre set of events that desire morphed into the image scaling API, which was then discarded due to being too weird.

It does occur to me though that it could be interesting to allow a canvas context to be transferred to a worker.  Think about this for a moment:  It would allow arbitrarily expensive rendering to occur in the worker, and then you just need to have some flush style API that would allow the worker to indicate that the content of the canvas was ready to render -- essentially this would be a join() on the UI thread, but the rendering would never blocked the UI.

Alas when I think about it, i think it may require double buffering the canvas, but it could provide a substantial performance boost, with minimal developer-side complexity.

--Oliver

> 
> -Darin
> 
> 
>  
>  
> 
> Remember the goal of making this asynchronous is to improve performance, so the 11ms of drawing does have to occur at some point, you're just hoping that by making things asynchronous you can mask that.  But I doubt you would see an actual improvement in wall clock performance.
> 
> The 11 ms of drawing occurs on a background thread.  Yes, that latency exists, but it doesn't have to block the main thread.
> 
> Let me reiterate the point I made before.  There can be multiple web pages sharing the same main thread.  (Even in Chrome this can be true!)  Blocking one web page has the effect of blocking all web pages that share the same main thread.
> 
> It is not nice for one web page to jank up the browser's main thread and as a result make other web pages unresponsive.
> 
>  
> 
> I also realised something else that I had not previously considered -- if you're doing bitblit based sprite movement the complexity goes way up if this is asynchronous.
> 
> I don't follow.  Can you clarify?
> 
> Thanks,
> -Darin 
>