[whatwg] Canvas pixel manipulation and performance

[Jason Oster]

Hello Group,

I've been using canvas to draw pixel art (NES/SNES game screens and sprites) similar to what an emulator would do.  Doing this kind of drawing requires direct access to the pixel buffer.

My problem with the canvas spec (as it is now) is that it tends to artificially bounds pixel drawing performance to JavaScript when doing any sort of pixel access.  Setting four unsigned 8-bit array elements (R, G, B, and A) is a slower operation that setting just one unsigned 32-bit array element (RGBA or ABGR).  Sadly, we don't have this latter option for canvas.

My comment is a request for a new set of pixel access methods on the CanvasRenderingContext2D object.  Specifically, alternatives to createImageData(), getImageData(), and putImageData() methods for providing an array of unsigned 32-bit elements for pixel manipulation.

One proposal is the reuse of the CanvasArrayBuffer introduced by WebGL[1].  The reference explains the use of CanvasArrayBuffer in the context of RGBA color space: "... RGBA colors, with each component represented as an unsigned byte."  This appears to be a useful solution, with an existing implementation to build from (at least in Mozilla).  The single concern here is that it neglects any mention of support for hardware utilizing native-ABGR (eg. "little endian") byte order, or more "obscure" formats.  I assume the idea was to handle any necessary conversions in the back-end.  Including 32-bit color depth->16-bit color depth, for example.

A second option is allowing the web developer to handle byte order issues, similar in concept to SDL[2].  In addition to general endian handling, SDL also supports "mapping" color components to an unsigned 32-bit integer[3].  It seems to me this is the best way to cover hardware byte order/color depth independence while achieving the best "user land" performance possible.

Take for instance, the following pseudo code:

  var canvas = document.getElementById("canvas");
  var ctx = canvas.getContext("2d");
  var pixels = ctx.createUnsignedByteArray(8, 8);
  // Fill with medium gray
  for (var i = 0; i < 8 * 8; i++) {
    pixels.data[i] = ctx.mapRGBA(128, 128, 128, 255);
  }
  ctx.putUnsignedByteArray(pixels, 0, 0);

That appears more sane than the current method:

  var canvas = document.getElementById("canvas");
  var ctx = canvas.getContext("2d");
  var pixels = ctx.createImageData(8, 8);
  // Fill with medium gray
  for (var i = 0; i < 8 * 8; i++) {
    pixels.data[i * 4 + 0] = 128;
    pixels.data[i * 4 + 1] = 128;
    pixels.data[i * 4 + 2] = 128;
    pixels.data[i * 4 + 3] = 255;
  }
  ctx.putImageData(pixels, 0, 0);

I understand this a bad way to fill a portion of a canvas with a solid color; this is for illustration purposes only.  The overall idea is that setting fewer array elements per pixel will perform better.

We've already seen the emergence of emulators written in JavaScript/Canvas.  In fact, there are loads of them[4], and they would all benefit from having a better way to interact directly with canvas pixels.  Of course, the use cases are not limited to emulation; my NES/SNES level editor projects would enjoy faster pixel manipulation as well.  These kinds of projects can use arbitrarily sized canvases (up to 4864px × 3072px in one case[5]) and can take a good deal of time to fully render, even with several off-ImageData optimization tricks.

Looking to discuss more options!
Jason Oster


[1] http://blog.vlad1.com/2009/11/06/canvasarraybuffer-and-canvasarray/
[2] http://www.libsdl.org/intro.en/usingendian.html
[3] http://www.libsdl.org/cgi/docwiki.cgi/SDL_MapRGBA
[4] http://www.google.com/search?q=javascript+emulator
[5] http://parasyte.kodewerx.org/projects/syndrome/stages/2009-07-05/12_wily3.png