[whatwg] Counterproposal for canvas in workers

[Rik Cabanier]

Hi Justin,

I will do reach out to some developers to see if they think it's useful.

As an experiment, I wrote a javascript polyfill that implements a rough
version of my proposal.
I took a box 2d example and ported it over. Since it draws to canvas under
the hood, I didn't have to change any code to make it work.
Original example: http://cabanier.github.io/CanvasWorker/test2.html
Example that uses a worker: http://cabanier.github.io/CanvasWorker/test.html
I added an js-driven animation to show how the original example is hogging
the main thread.

Unfortunately, on Firefox there's a lot of GC overhead so it's better to
use Safari or Chrome.


On Fri, Oct 18, 2013 at 11:30 AM, Justin Novosad <junov at google.com> wrote:

>
>
>
> On Fri, Oct 18, 2013 at 12:36 PM, Rik Cabanier <cabanier at gmail.com> wrote:
>
>> Hi Justin,
>>
>> no, everything is running synchronized and there is no added latency.
>>
>> API calls on canvas will be recorded if there are outstanding tasks.
>> So, for this API call:
>>
>> ctx.drawImage(MinionCanvas, ...)
>>
>> Since it is happening in a task, drawImage will be recorded. It will only
>> execute once the task and its subtasks (ie MinionCanvas.executeTask("drawMinion",
>> {}) ) are done.
>>
>> It sounds complicated but I think it's much easier for an author than
>> having to send bitmaps and message back and forth.
>>
>
> Ok, got it.  I think this is compelling for many use cases, but I am not
> sure whether it is as generally useful/usable as WorkerCanvas.  It would be
> great to get more web developer feedback.  In particular from graphics
> intensive game and interactive app people.
>
>
>> On Fri, Oct 18, 2013 at 6:48 AM, Justin Novosad <junov at google.com> wrote:
>>
>>> Rik, I don't think the nested tasks in your example are a good use case.
>>>  That workflow adds a frame of latency to the sub tasks. This is a problem
>>> because the drawImage call would be drawing from a source canvas that it
>>> out of phase with the mainscene context.  To synchronize the content, I
>>> think the drawImage calls would have to be placed in a Promise resolution
>>> handler that gets invoked once all the executeTask Promises for the
>>> subtasks are resolved.  That means that the drawImage calls necessary for
>>> drawing "mainscene" would end-up executing asynchronously, therefore
>>> outside the scope of the "mainscene" task function, which is a problem.  So
>>> I don't think the executeTask proposal is well suited for sharding
>>> rendering jobs, at least not the way you illustrated it with that example.
>>>
>>>
>>>
>>> On Fri, Oct 18, 2013 at 12:50 AM, Rik Cabanier <cabanier at gmail.com>wrote:
>>>
>>>> Extra methods on the canvas API:
>>>>
>>>> Promise setTaskScript(DOMString script); // can error be in promise?
>>>> Promise executeTask(DOMString id, dictionary json, boolean synchronized
>>>> = true); // Transferable elements allowed in dictionary
>>>>
>>>> Object that is active in the task:
>>>>
>>>> interface CanvasTask {
>>>>
>>>> HTMLCanvasElement createCanvas(unsigned long width, unsigned long
>>>> height);
>>>> attribute Function onTask;
>>>>
>>>> }
>>>>
>>>> CanvasTask implements HTMLCanvasElement;
>>>>
>>>> Example code:
>>>>
>>>> var c = document.getElementById("gameCanvas");
>>>>
>>>> var gameState = {};
>>>>
>>>> window.addEventListener("load", function(){
>>>>
>>>> c.setTaskScript("gameLogic.js").then(function(){
>>>>
>>>> c.executeTask("mainscene", gameState);
>>>>
>>>> });
>>>>
>>>> });
>>>>
>>>>
>>>> window.requestAnimationFrame(function(){
>>>>
>>>> c.executeTask("mainscene", gameState);
>>>>
>>>> }
>>>>
>>>>
>>>> Example code for gameLogic.js:
>>>>
>>>> var ctx = getContext("2d");
>>>>
>>>> onTask = function(DOMString id, dictionary json) {
>>>>
>>>> if(id == "mainscene") {
>>>>
>>>> if(typeof(MinionCanvas)=="Undefined") {
>>>>
>>>> MinionCanvas = createCanvas(200, 300);
>>>>
>>>>  MinionCanvas.executeTask("drawMinion", {}) // creates promise under
>>>> the hood
>>>>
>>>> }
>>>>
>>>> if(typeof(SpaceShipCanvas)=="Undefined")
>>>>
>>>> SpaceShipCanvas = createCanvas(300, 300);
>>>>
>>>>
>>>> SpaceShipCanvas.executeTask("drawSpaceShip", gameState); // redraw
>>>> spaceship
>>>>
>>>>
>>>> executeTask("drawBackDrop",  gameState); // in other task
>>>>
>>>> executeTask("drawBoss", gameState); // lots of js to draw the boss so
>>>> better done in task
>>>>
>>>>
>>>> for(...) //for each minion {
>>>>
>>>>  ... // set the matrix
>>>> ctx.drawImage(MinionCanvas, ...); // draw the minion <- note that the
>>>> minion might still be drawing in the other thread
>>>>
>>>>   }
>>>> for(...) //for each spaceship {
>>>>
>>>> ..// set the matrix
>>>>
>>>> ctx.drawImage(SpaceShipCanvas); // draw the spaceship <- it might still
>>>> be drawing in the other task
>>>>
>>>> }
>>>>
>>>> .. // other drawing commands for score, controls, etc
>>>>
>>>> } else if(id == "drawMinion") {
>>>>
>>>> ...
>>>>
>>>> }  else if(id == "drawSpaceShip") {
>>>>
>>>> ... // set up tasks to draw parts of the ship?
>>>>
>>>> } ...
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> On Thu, Oct 17, 2013 at 8:10 PM, Rik Cabanier <cabanier at gmail.com>wrote:
>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Oct 17, 2013 at 4:01 PM, Robert O'Callahan <
>>>>> robert at ocallahan.org> wrote:
>>>>>
>>>>>> On Fri, Oct 18, 2013 at 10:56 AM, Justin Novosad <junov at google.com>wrote:
>>>>>>
>>>>>>> On Thu, Oct 17, 2013 at 5:50 PM, Rik Cabanier <cabanier at gmail.com>wrote:
>>>>>>>
>>>>>>>> Creating temporary canvases is still possible. I'm unsure how it
>>>>>>>> would be different from a worker.
>>>>>>>> An advantage would be that you can draw to the temporary canvases
>>>>>>>> in parallel to using them. Only PIXEL access is disallowed, you can still
>>>>>>>> call drawImage using a canvas that has outstanding tasks.
>>>>>>>>
>>>>>>>
>>>>>>> Right. The write-only restriction would only apply to canvas
>>>>>>> contexts that commit (push their tasks) directly down to the compositor.
>>>>>>> You could still create a canvas that is local to the worker, rasterize it
>>>>>>> in the worker and do readbacks in the worker, create ImageBitmaps from it,
>>>>>>> etc.
>>>>>>>
>>>>>>
>>>>>> I'm not sure that you and Rik are talking about the same thing, since
>>>>>> he's still talking about "outstanding tasks". If you are talking about the
>>>>>> same thing, I don't know what it is. I'd like to see some concrete details
>>>>>> for what you'd change in the current WorkerCanvas proposal. For the sake of
>>>>>> clarity I've put (my understand of) it here:
>>>>>> https://wiki.mozilla.org/User:Roc/WorkerCanvasProposal
>>>>>>
>>>>>
>>>>> I'll work on drawing up an example of my proposal.
>>>>>
>>>>> With WorkerCanvas and transferToImageBitmap, you can draw multiple
>>>>>> layers in parallel (and actually draw, not just queue drawing commands) by
>>>>>> creating multiple workers, having them each produce an ImageBitmap, and
>>>>>> compositing those ImageBitmaps together by stacking <img> elements or
>>>>>> drawing them all to a single canvas. It uses more memory but you get more
>>>>>> parallelism.
>>>>>>
>>>>>
>>>>> They would still have to wait for each other so the images are
>>>>> composited in-order. If you don't care about that, the 'synchronized'
>>>>> option would let you draw as soon as you exit the task (which is how Chrome
>>>>> always draws since it's faster)
>>>>>
>>>>> In fact, an implementation could choose to take the deferred-drawing
>>>>>> approach instead. You would queue up drawing commands in the WorkerCanvas
>>>>>> (or the drawing context), and then transferToImageBitmap would not
>>>>>> immediately render but produce an ImageBitmap implementation encapsulating
>>>>>> the list of drawing commands to be drawn later, wherever/whenever that
>>>>>> ImageBitmap ended up being used. I think for commit() the implementation
>>>>>> would always want to force rasterization on the worker (or possibly some
>>>>>> dedicated canvas-rendering thread); you could forward a list of drawing
>>>>>> commands to the compositor thread for rasterization but I don't think
>>>>>> there's any reason to do that (and some good reasons not to).
>>>>>>
>>>>>
>>>>> Can you tell me how you can ensure that you don't do too much work?
>>>>> Drawing in a continuous loop using 'Commit' would waste a lot of resources.
>>>>>
>>>>
>>>>
>>>
>>
>