[whatwg] Canvas pixel manipulation and performance

[Oliver Hunt]

On Nov 26, 2009, at 11:45 AM, Jason Oster wrote:

> Hello Group,
> 
> I've been using canvas to draw pixel art (NES/SNES game screens and sprites) similar to what an emulator would do.  Doing this kind of drawing requires direct access to the pixel buffer.
> 
> My problem with the canvas spec (as it is now) is that it tends to artificially bounds pixel drawing performance to JavaScript when doing any sort of pixel access.  Setting four unsigned 8-bit array elements (R, G, B, and A) is a slower operation that setting just one unsigned 32-bit array element (RGBA or ABGR).  Sadly, we don't have this latter option for canvas.
> 
> My comment is a request for a new set of pixel access methods on the CanvasRenderingContext2D object.  Specifically, alternatives to createImageData(), getImageData(), and putImageData() methods for providing an array of unsigned 32-bit elements for pixel manipulation.
> 
> One proposal is the reuse of the CanvasArrayBuffer introduced by WebGL[1].  The reference explains the use of CanvasArrayBuffer in the context of RGBA color space: "... RGBA colors, with each component represented as an unsigned byte."  This appears to be a useful solution, with an existing implementation to build from (at least in Mozilla).  The single concern here is that it neglects any mention of support for hardware utilizing native-ABGR (eg. "little endian") byte order, or more "obscure" formats.  I assume the idea was to handle any necessary conversions in the back-end.  Including 32-bit color depth->16-bit color depth, for example.
WebGL has completely different constraints to that of the 2d canvas -- when the developer provides resources to GL the developer has to provide a myriad of type details, this means that the developer needs to be able to request storage of a specific type.  The WebGL array types are specifically targeting this use case -- they don't make sense for canvas2d where the only storage is not a developer specified format.

> A second option is allowing the web developer to handle byte order issues, similar in concept to SDL[2].  In addition to general endian handling, SDL also supports "mapping" color components to an unsigned 32-bit integer[3].  It seems to me this is the best way to cover hardware byte order/color depth independence while achieving the best "user land" performance possible.

History has shown that any time a developer won't handle both byte orders -- developers tend to work on the assumption that if something works for them it must be correct, this is why we end up with sites that claim "This site needs IE/Safari/Firefox to run" type messages.  Even conscientious developers who test multiple browsers, and validate their content, etc will be able to produce accidentally broken sites because this would add a hardware dependency on spec behaviour.

Realistically simply making an separate object that has indexes 32bit rgba pixels would resolve the problem you're trying to describe -- the implementation would need to do byte order correct, but given that 2/3 canvas implementations already do unpre->premultiplied data conversion on putImageData this is unlikely to add any cost at all (in fact in the webkit implementation i don't believe there would be any difference in the logic in get/putImageData).

> 
> Take for instance, the following pseudo code:
> 
>  var canvas = document.getElementById("canvas");
>  var ctx = canvas.getContext("2d");
>  var pixels = ctx.createUnsignedByteArray(8, 8);
>  // Fill with medium gray
>  for (var i = 0; i < 8 * 8; i++) {
>    pixels.data[i] = ctx.mapRGBA(128, 128, 128, 255);
>  }
>  ctx.putUnsignedByteArray(pixels, 0, 0);

Adding a function call would make your code much slower.

> 
> That appears more sane than the current method:
> 
>  var canvas = document.getElementById("canvas");
>  var ctx = canvas.getContext("2d");
>  var pixels = ctx.createImageData(8, 8);
>  // Fill with medium gray
>  for (var i = 0; i < 8 * 8; i++) {
>    pixels.data[i * 4 + 0] = 128;
>    pixels.data[i * 4 + 1] = 128;
>    pixels.data[i * 4 + 2] = 128;
>    pixels.data[i * 4 + 3] = 255;
>  }
>  ctx.putImageData(pixels, 0, 0);
> 
> I understand this a bad way to fill a portion of a canvas with a solid color; this is for illustration purposes only.  The overall idea is that setting fewer array elements per pixel will perform better.

Have you actually measured this?  How long is spent in each part?  I suspect if you're not using the dirty region arguments you're pushing back (and doing premult conversion) on a lot more pixels than necessary.  Yes setting 4 properties is slower than setting 1, but where is your time actually being spent.

> 
> We've already seen the emergence of emulators written in JavaScript/Canvas.  In fact, there are loads of them[4], and they would all benefit from having a better way to interact directly with canvas pixels.  Of course, the use cases are not limited to emulation; my NES/SNES level editor projects would enjoy faster pixel manipulation as well.  These kinds of projects can use arbitrarily sized canvases (up to 4864px × 3072px in one case[5]) and can take a good deal of time to fully render, even with several off-ImageData optimization tricks.

Without seeing the code for your demo i'd have no idea whether what you're doing is actually efficient -- have you profiled?  Both Safari and Firefox have built in profilers.

--Oliver

> 
> Looking to discuss more options!
> Jason Oster
> 
> 
> [1] http://blog.vlad1.com/2009/11/06/canvasarraybuffer-and-canvasarray/
> [2] http://www.libsdl.org/intro.en/usingendian.html
> [3] http://www.libsdl.org/cgi/docwiki.cgi/SDL_MapRGBA
> [4] http://www.google.com/search?q=javascript+emulator
> [5] http://parasyte.kodewerx.org/projects/syndrome/stages/2009-07-05/12_wily3.png