[whatwg] Counterproposal for canvas in workers

[Justin Novosad]

On Fri, Oct 18, 2013 at 12:36 PM, Rik Cabanier <cabanier at gmail.com> wrote:

> Hi Justin,
>
> no, everything is running synchronized and there is no added latency.
>
> API calls on canvas will be recorded if there are outstanding tasks.
> So, for this API call:
>
> ctx.drawImage(MinionCanvas, ...)
>
> Since it is happening in a task, drawImage will be recorded. It will only
> execute once the task and its subtasks (ie MinionCanvas.executeTask("drawMinion",
> {}) ) are done.
>
> It sounds complicated but I think it's much easier for an author than
> having to send bitmaps and message back and forth.
>

Ok, got it.  I think this is compelling for many use cases, but I am not
sure whether it is as generally useful/usable as WorkerCanvas.  It would be
great to get more web developer feedback.  In particular from graphics
intensive game and interactive app people.


> On Fri, Oct 18, 2013 at 6:48 AM, Justin Novosad <junov at google.com> wrote:
>
>> Rik, I don't think the nested tasks in your example are a good use case.
>>  That workflow adds a frame of latency to the sub tasks. This is a problem
>> because the drawImage call would be drawing from a source canvas that it
>> out of phase with the mainscene context.  To synchronize the content, I
>> think the drawImage calls would have to be placed in a Promise resolution
>> handler that gets invoked once all the executeTask Promises for the
>> subtasks are resolved.  That means that the drawImage calls necessary for
>> drawing "mainscene" would end-up executing asynchronously, therefore
>> outside the scope of the "mainscene" task function, which is a problem.  So
>> I don't think the executeTask proposal is well suited for sharding
>> rendering jobs, at least not the way you illustrated it with that example.
>>
>>
>>
>> On Fri, Oct 18, 2013 at 12:50 AM, Rik Cabanier <cabanier at gmail.com>wrote:
>>
>>> Extra methods on the canvas API:
>>>
>>> Promise setTaskScript(DOMString script); // can error be in promise?
>>> Promise executeTask(DOMString id, dictionary json, boolean synchronized
>>> = true); // Transferable elements allowed in dictionary
>>>
>>> Object that is active in the task:
>>>
>>> interface CanvasTask {
>>>
>>> HTMLCanvasElement createCanvas(unsigned long width, unsigned long
>>> height);
>>> attribute Function onTask;
>>>
>>> }
>>>
>>> CanvasTask implements HTMLCanvasElement;
>>>
>>> Example code:
>>>
>>> var c = document.getElementById("gameCanvas");
>>>
>>> var gameState = {};
>>>
>>> window.addEventListener("load", function(){
>>>
>>> c.setTaskScript("gameLogic.js").then(function(){
>>>
>>> c.executeTask("mainscene", gameState);
>>>
>>> });
>>>
>>> });
>>>
>>>
>>> window.requestAnimationFrame(function(){
>>>
>>> c.executeTask("mainscene", gameState);
>>>
>>> }
>>>
>>>
>>> Example code for gameLogic.js:
>>>
>>> var ctx = getContext("2d");
>>>
>>> onTask = function(DOMString id, dictionary json) {
>>>
>>> if(id == "mainscene") {
>>>
>>> if(typeof(MinionCanvas)=="Undefined") {
>>>
>>> MinionCanvas = createCanvas(200, 300);
>>>
>>>  MinionCanvas.executeTask("drawMinion", {}) // creates promise under
>>> the hood
>>>
>>> }
>>>
>>> if(typeof(SpaceShipCanvas)=="Undefined")
>>>
>>> SpaceShipCanvas = createCanvas(300, 300);
>>>
>>>
>>> SpaceShipCanvas.executeTask("drawSpaceShip", gameState); // redraw
>>> spaceship
>>>
>>>
>>> executeTask("drawBackDrop",  gameState); // in other task
>>>
>>> executeTask("drawBoss", gameState); // lots of js to draw the boss so
>>> better done in task
>>>
>>>
>>> for(...) //for each minion {
>>>
>>>  ... // set the matrix
>>> ctx.drawImage(MinionCanvas, ...); // draw the minion <- note that the
>>> minion might still be drawing in the other thread
>>>
>>>   }
>>> for(...) //for each spaceship {
>>>
>>> ..// set the matrix
>>>
>>> ctx.drawImage(SpaceShipCanvas); // draw the spaceship <- it might still
>>> be drawing in the other task
>>>
>>> }
>>>
>>> .. // other drawing commands for score, controls, etc
>>>
>>> } else if(id == "drawMinion") {
>>>
>>> ...
>>>
>>> }  else if(id == "drawSpaceShip") {
>>>
>>> ... // set up tasks to draw parts of the ship?
>>>
>>> } ...
>>>
>>> }
>>>
>>>
>>>
>>> On Thu, Oct 17, 2013 at 8:10 PM, Rik Cabanier <cabanier at gmail.com>wrote:
>>>
>>>>
>>>>
>>>>
>>>> On Thu, Oct 17, 2013 at 4:01 PM, Robert O'Callahan <
>>>> robert at ocallahan.org> wrote:
>>>>
>>>>> On Fri, Oct 18, 2013 at 10:56 AM, Justin Novosad <junov at google.com>wrote:
>>>>>
>>>>>> On Thu, Oct 17, 2013 at 5:50 PM, Rik Cabanier <cabanier at gmail.com>wrote:
>>>>>>
>>>>>>> Creating temporary canvases is still possible. I'm unsure how it
>>>>>>> would be different from a worker.
>>>>>>> An advantage would be that you can draw to the temporary canvases in
>>>>>>> parallel to using them. Only PIXEL access is disallowed, you can still call
>>>>>>> drawImage using a canvas that has outstanding tasks.
>>>>>>>
>>>>>>
>>>>>> Right. The write-only restriction would only apply to canvas contexts
>>>>>> that commit (push their tasks) directly down to the compositor. You could
>>>>>> still create a canvas that is local to the worker, rasterize it in the
>>>>>> worker and do readbacks in the worker, create ImageBitmaps from it, etc.
>>>>>>
>>>>>
>>>>> I'm not sure that you and Rik are talking about the same thing, since
>>>>> he's still talking about "outstanding tasks". If you are talking about the
>>>>> same thing, I don't know what it is. I'd like to see some concrete details
>>>>> for what you'd change in the current WorkerCanvas proposal. For the sake of
>>>>> clarity I've put (my understand of) it here:
>>>>> https://wiki.mozilla.org/User:Roc/WorkerCanvasProposal
>>>>>
>>>>
>>>> I'll work on drawing up an example of my proposal.
>>>>
>>>> With WorkerCanvas and transferToImageBitmap, you can draw multiple
>>>>> layers in parallel (and actually draw, not just queue drawing commands) by
>>>>> creating multiple workers, having them each produce an ImageBitmap, and
>>>>> compositing those ImageBitmaps together by stacking <img> elements or
>>>>> drawing them all to a single canvas. It uses more memory but you get more
>>>>> parallelism.
>>>>>
>>>>
>>>> They would still have to wait for each other so the images are
>>>> composited in-order. If you don't care about that, the 'synchronized'
>>>> option would let you draw as soon as you exit the task (which is how Chrome
>>>> always draws since it's faster)
>>>>
>>>> In fact, an implementation could choose to take the deferred-drawing
>>>>> approach instead. You would queue up drawing commands in the WorkerCanvas
>>>>> (or the drawing context), and then transferToImageBitmap would not
>>>>> immediately render but produce an ImageBitmap implementation encapsulating
>>>>> the list of drawing commands to be drawn later, wherever/whenever that
>>>>> ImageBitmap ended up being used. I think for commit() the implementation
>>>>> would always want to force rasterization on the worker (or possibly some
>>>>> dedicated canvas-rendering thread); you could forward a list of drawing
>>>>> commands to the compositor thread for rasterization but I don't think
>>>>> there's any reason to do that (and some good reasons not to).
>>>>>
>>>>
>>>> Can you tell me how you can ensure that you don't do too much work?
>>>> Drawing in a continuous loop using 'Commit' would waste a lot of resources.
>>>>
>>>
>>>
>>
>