[whatwg] Canvas in workers

[Justin Novosad]

Robert, I think your argument makes sense. The DrawingBuffer mechanism
described in  http://wiki.whatwg.org/wiki/CanvasInWorkers can be made
implicit and hidden under the hood in a way that is just as memory
efficient by using the right combination of read locks and copy-on-write.
However, I have concerns about how resizing would be handled by the
proposed commit mechanism. I think there is no perfect solution, but
whatever we do, I think animation smoothness should be a primary concern,
so we should avoid blocking any threads and we should avoid dropping frames
as much as possible.  I like the concept of handling canvas size mismatches
that is brought forward in the CanvasInWorkers proposal.

If we put that idea into the WorkerCanvas proposal, here is an idea of how
commits could handle resizes asynchronously:

1) Main thread executes some JS that changes the size of the canvas, and
posts an event to the Worker in order to propagate the change to the
WorkerCanvas.
2) On the worker thread, the WorkerCanvas stores the new size, but does not
apply it right away to avoid resetting canvas contents mid frame.
3) commit() is called on the WorkerCanvas, causing the current canvas
contents to be sent to the main thread for display, and the new canvas size
comes into effect (lazily?)
4) The main thread receives the committed pixels. N.B.: The size of the
received buffer does not match canvas element's intrinsic size
5) At paint time, the canvas contents get displayed respecting the canvas
element's current size. The mismatched pixel buffer is either
streched/squashed or clipped/padded to fit.

An alternative that was proposed earlier in this (e-mail) thread was for
the main thread to reject commits when there is a size mismatch. That does
respect the desire to be asynchronous, and it avoid the inconvenience of
resizing contents, but I don't like the idea because it means there can be
cases where we may have a large number of consecutive dropped frames. Think
of drag-resizing for example. We should avoid making that common use case
behave poorly.





On Mon, Oct 14, 2013 at 4:34 PM, Robert O'Callahan <robert at ocallahan.org>wrote:

> On Mon, Oct 14, 2013 at 2:20 PM, Kenneth Russell <kbr at google.com> wrote:
>
> > Would you mind looking at the proposal
> > http://wiki.whatwg.org/wiki/CanvasInWorkers and commenting on it?
>
>
> Sure. Kyle and I looked at it while we were working on our proposal. The
> main issues I have with it are that rearchitecting <canvas> to introduce
> the DrawingBuffer layer of abstraction seems unnecessarily complex, and it
> doesn't handle direct presentation of frames from the worker, bypassing the
> main thread.
>
> There's been some recent discussion in the WebGL WG on this topic and
> > concerns were raised from other parties at Mozilla about the
> > DrawingBuffer proposal above, including that it isn't possible to
> > render from a worker without synchronizing with the main thread.
> >
>
> This is a huge limitation. For games and other animated content, achieving
> a stable frame rate is super important and a key motivation for adding
> canvas support to workers.
>
> My vision for handling the Maps use-cases based on our proposal is this:
> the worker renders to one or more WorkerCanvases, then does
> createImageBitmap(canvasContext), then ships the ImageBitmap(s) to the main
> thread using postMessage, and then renders those ImageBitmaps either by
> drawing them to a canvas or in some more direct way.
>
> This can all be implemented in a zero-copy way with the APIs currently in
> the spec, though it's not trivial. You'd need to do a few optimizations:
> -- createImageBitmap(canvasContext) would take a lazy snapshot of the
> canvas contents; i.e., further modifications to the canvas would trigger a
> copy (on the worker), but if the canvas is untouched no copy is required.
> -- structured clone of the ImageBitmap would not copy. This may actually
> require a small spec change.
> -- drawing an ImageBitmap to a 2D <canvas>, if it covers the whole canvas,
> would simply replace the canvas buffer with the ImageBitmap contents (and
> perform copy-on-write if the script later makes changes to that canvas).
> These optimizations would all be useful in other contexts too. Whatever
> happens with canvas-in-a-worker, I bet we'll end up doing these
> optimizations for other reasons.
>
> It might make sense to create an API that renders an ImageBitmap more
> directly than drawing it to a canvas. For example we could create an API
> that allows an <img> element to render an ImageBitmap. It would be a bit
> simpler for authors and perhaps a bit simpler to implement than the final
> optimization in my list above.
>
> Rob
> --
> Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni
> le atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
> stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
> 'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
> waanndt  wyeonut  thoo mken.o w  *
> *
>