[whatwg] Canvas in workers

[Kenneth Russell]

On Tue, Oct 15, 2013 at 5:30 PM, Kenneth Russell <kbr at google.com> wrote:
> On Tue, Oct 15, 2013 at 4:41 PM, Robert O'Callahan <robert at ocallahan.org> wrote:
>> On Wed, Oct 16, 2013 at 11:55 AM, Kenneth Russell <kbr at google.com> wrote:
>>>
>>> On Mon, Oct 14, 2013 at 1:34 PM, Robert O'Callahan <robert at ocallahan.org>
>>> wrote:
>>> > On Mon, Oct 14, 2013 at 2:20 PM, Kenneth Russell <kbr at google.com> wrote:
>>> >>
>>> >> Would you mind looking at the proposal
>>> >> http://wiki.whatwg.org/wiki/CanvasInWorkers and commenting on it?
>>> >
>>> >
>>> > Sure. Kyle and I looked at it while we were working on our proposal. The
>>> > main issues I have with it are that rearchitecting <canvas> to introduce
>>> > the
>>> > DrawingBuffer layer of abstraction seems unnecessarily complex, and it
>>> > doesn't handle direct presentation of frames from the worker, bypassing
>>> > the
>>> > main thread.
>>>
>>> Note that the CanvasInWorkers draft solves some other longstanding
>>> issues not addressed by the WorkerCanvas proposal. It provides the
>>> ability to render to multiple canvases from a single context, whether
>>> workers are involved or not.
>>
>>
>> That may be a useful feature, but I'd like to see it justified in its own
>> right.
>
> There has been a lot of developer feedback on the WebGL mailing lists
> over the past couple of years about exactly this feature. Web sites
> like Turbosquid want to present lots of little thumbnails of models --
> see for example http://www.turbosquid.com/Search/3D-Models/Vehicle/Car
> -- and have them be interactive. It's too resource-intensive to create
> a separate WebGL context for each. The most direct solution is to
> allow one context to render to multiple canvases.
>
>
>>> It achieves ideal memory utilization by
>>> being very explicit in the API, without the need for extensive and
>>> subtle optimizations behind the scenes.
>>
>>
>> We can be more explicit with ImageBitmaps. We could provide
>> WorkerCanvas.transferToImageBitmap which transfers the current canvas
>> contents to an ImageBitmap and clears the canvas. (Canvas implementations
>> are already optimized to support a zero-cost "cleared" state, because
>> existing benchmarks require it.) Sharing ImageBitmap contents across threads
>> during structured clone is not subtle. We can add an
>> HTMLImageElement.srcObject attribute which could take a Blob or an
>> ImageBitmap to enable explicit zero-copy rendering of ImageBitmaps. Would
>> that be explicit enough for you?
>
> Yes, that generally sounds good.
>
>
>> Personally I think high-performance manipulation of ImageBitmaps would be
>> more generally useful than detachable DrawingBuffers, and would be easier
>> for authors to understand.
>>
>> If you squint, WorkerCanvas.transferToImageBitmap is similar to detaching a
>> DrawingBuffer. But I don't see a need to reattach a buffer to a canvas for
>> further drawing. Do you?
>
> Not immediately. The ability to transfer out the canvas's contents,
> and render them in an HTMLImageElement without incurring an extra
> blit, should address the Maps team's requirements.
>
> Actually, adding transferToImageBitmap to HTMLCanvasElement as well
> would address the use case of rendering to multiple targets using one
> context. Instead of using multiple canvases as the targets, one would
> simply use multiple images. That sounds appealing.
>
> If WorkerCanvas is changed so that its width and height are mutable
> within the worker as you mentioned above, it sounds like it's
> addressing the known use cases.

Capturing Glenn Maynard's feedback from the other thread started by
Rik Cabanier, Glenn made a good point that there needs to be a way to
explicitly deallocate the ImageBitmap. Otherwise, the JavaScript
objects will have to be garbage collected before the GPU resource
(texture) it references can be freed, and that will not work -- GPU
resources will quickly pile up.

Would it be acceptable to say that setting the 'src' property of an
HTMLImageElement to an ImageBitmap neuters the incoming ImageBitmap?
If not, would it be feasible to add another method to HTMLImageElement
like setToImageBitmap which does this?

-Ken


>>> It's worth considering whether a change to the CanvasInWorkers
>>> proposal could support presenting frames directly from the worker.
>>
>>
>> Sure, by adding a commit() method to DrawingBuffer. Right?
>
> I'm not exactly sure how it would be done. In the proposal as written,
> the DrawingBuffer's not shared between threads, only transferred.
>
> -Ken
>
>
>> Rob
>> --
>> Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni le
>> atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
>> stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
>> 'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
>> waanndt  wyeonut  thoo mken.o w