[whatwg] [canvas] request for {create, get, put}ImageDataHD and ctx.backingStorePixelRatio

[Oliver Hunt]

On Apr 16, 2012, at 11:38 AM, Darin Fisher <darin at chromium.org> wrote:

> On Mon, Apr 16, 2012 at 11:17 AM, Oliver Hunt <oliver at apple.com> wrote:
> 
> On Apr 16, 2012, at 11:07 AM, Darin Fisher <darin at chromium.org> wrote:
> 
> >
> > Carrots and Sticks.
> >
> > Aren't we missing an opportunity here?  By giving web developers this easy
> > migration path, you're also giving up the opportunity to encourage them to
> > use a better API.  Asynchronous APIs are harder to use, and that's why we
> > need to encourage their adoption.  If you just give people a synchronous
> > version that accomplishes the same thing, then they will just use that,
> > even if doing so causes their app to perform poorly.
> >
> > See synchronous XMLHttpRequest.  I'm sure every browser vendor wishes that
> > didn't exist.  Note how we recently withdrew support for synchronous
> > ArrayBuffer access on XHR?  We did this precisely to discourage use of
> > synchronous mode XHR. Doing so actually broke some existing web pages.  The
> > pain was deemed worth it.
> >
> > GPU readback of a HD buffer is going to suck.  Any use of this new API is
> > going to suck.
> >
> > -Darin
> >
> 
> Any use of imagedata i've seen assumes that they can avoid intermediate states in the canvas ever being visible, if you make reading and writing the data asynchronous you break that invariant and suddenly makes things much harder for the user.
> 
> I agree with Charles Pritchard that it is only the reading of pixel data that should be asynchronous.
> 
> I think developers could learn to cope with this new design just as they do with other asynchronous facets of the platform.
> 
>  
> 
> The reason we don't want IO synchronous is because IO can take a potentially unbound amount of time, if you're on a platform that makes a memcpy take similarly unbound time, i recommend that you work around it.
> 
> Of course, GPU readbacks do not compare to network IO.  However, if the goal is to achieve smooth animations, then it is important that the main thread not hitch for multiple animation frames.  GPU readbacks are irregular in duration and can sometimes be quite expensive if the GPU pipeline is heavily burdened.
> 
>  
> 
> Anyway, the sensible approach to imagedata + hardware backed canvas is to revert to a software backed canvas, as once someone has used imagedata once, they're likely to do it again (and again, and again) so it is probably a win to just do everything in software at that point.  Presumably you could through in heuristics to determine whether or not it's worth going back to the GPU at some point, but many of the common image data use cases will have awful perf if you try to keep them on the GPU 100% of the time.
> 
> I don't think it is OK if at application startup (or animation startup) there is a big UI glitch as the system determines that it should not GPU-back a canvas.  We have the opportunity now to design an API that does not have that bug.
> 
> Why don't you want to take advantage of this opportunity?

We can already do imagedata based access on a gpu backed canvas in webkit without ill effects simply by pulling the canvas off GPU memory.   I don't understand why adding a runloop cycle to any read seems like something that would introduce a much more noticable delay than a memcopy.  I also don't understand what makes reading from the GPU so expensive that adding a runloop cycle is necessary for good perf, but it's unnecessary for a write.  This feels like an argument along the lines of "we hate synchronous APIs, but they make sense for graphics.  Let's try and make at least part of this asynchronous to satisfy that particular desire."

Moving data to and from the GPU may be expensive, but i doubt it holds a candle to the cost of waiting for a full runloop cycle, unless you're doing something really inefficient in your backing store management.  The fact is that the ImageData is a pixel manipulation API, and any such API is not conducive to good performance on the GPU.

--Oliver

> 
> -Darin
> 
> 
> 
>  
> >
> >
> >>
> >>
> >>>
> >>> - James
> >>> On Mar 20, 2012 10:29 AM, "Edward O'Connor" <eoconnor at apple.com>
> >> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> Unfortunately, lots of <canvas> content (especially content which calls
> >>>> {create,get,put}ImageData methods) assumes that the <canvas>'s backing
> >>>> store pixels correspond 1:1 to CSS pixels, even though the spec has been
> >>>> written to allow for the backing store to be at a different scale
> >>>> factor.
> >>>>
> >>>> Especially problematic is that developers have to round trip image data
> >>>> through a <canvas> in order to detect that a different scale factor is
> >>>> being used.
> >>>>
> >>>> I'd like to propose the addition of a backingStorePixelRatio property to
> >>>> the 2D context object. Just as window.devicePixelRatio expresses the
> >>>> ratio of device pixels to CSS pixels, ctx.backingStorePixelRatio would
> >>>> express the ratio of backing store pixels to CSS pixels. This allows
> >>>> developers to easily branch to handle different backing store scale
> >>>> factors.
> >>>>
> >>>> Additionally, I think the existing {create,get,put}ImageData API needs
> >>>> to be defined to be in terms of CSS pixels, since that's what existing
> >>>> content assumes. I propose the addition of a new set of methods for
> >>>> working directly with backing store image data. (New methods are easier
> >>>> to feature detect than adding optional arguments to the existing
> >>>> methods.) At the moment I'm calling these {create,get,put}ImageDataHD,
> >>>> but I'm not wedded to the names. (Nor do I want to bikeshed them.)
> >>>>
> >>>>
> >>>> Thanks for your consideration,
> >>>> Ted
> >>>>
> >>
> >>
> 
>