[whatwg] Canvas pixel manipulation and performance

[Jason Oster]

My apologies for the direct reply, Oliver.  This was meant to go back to the list:


On Nov 26, 2009, at 3:35 PM, Oliver Hunt wrote:
> WebGL has completely different constraints to that of the 2d canvas -- when the developer provides resources to GL the developer has to provide a myriad of type details, this means that the developer needs to be able to request storage of a specific type.  The WebGL array types are specifically targeting this use case -- they don't make sense for canvas2d where the only storage is not a developer specified format.
That is understandable.

> History has shown that any time a developer won't handle both byte orders -- developers tend to work on the assumption that if something works for them it must be correct, this is why we end up with sites that claim "This site needs IE/Safari/Firefox to run" type messages.  Even conscientious developers who test multiple browsers, and validate their content, etc will be able to produce accidentally broken sites because this would add a hardware dependency on spec behaviour.
We certainly don't want any more of that.

> Realistically simply making an separate object that has indexes 32bit rgba pixels would resolve the problem you're trying to describe -- the implementation would need to do byte order correct, but given that 2/3 canvas implementations already do unpre->premultiplied data conversion on putImageData this is unlikely to add any cost at all (in fact in the webkit implementation i don't believe there would be any difference in the logic in get/putImageData).
Once again, I agree.  My confusion on the type-specific arrays for WebGL is that they were specific and general enough to use in other cases.  If they should not be used in 2D canvas implementations (or elsewhere) then a 2D-canvas-specific array or object would be the way forward.

>> Take for instance, the following pseudo code:
>> 
>> var canvas = document.getElementById("canvas");
>> var ctx = canvas.getContext("2d");
>> var pixels = ctx.createUnsignedByteArray(8, 8);
>> // Fill with medium gray
>> for (var i = 0; i < 8 * 8; i++) {
>>  pixels.data[i] = ctx.mapRGBA(128, 128, 128, 255);
>> }
>> ctx.putUnsignedByteArray(pixels, 0, 0);
> 
> Adding a function call would make your code much slower.
Yes, it would.  Once again, this was only for illustrative purposes.  More commonly, a Look-Up Table would be created, containing all of the colors used in the scene before any pixels are touched.  For any kind of low-resolution pixel art (as found in classic gaming consoles), the palette is typically indexed and consisting of 256 colors or fewer.  In extreme cases, an LUT with thousands of colors would be far faster than using such a function call.

I neglected to mention any optimal way of using a "mapRGBA" function; that's not what I was trying to illustrate.

>> I understand this a bad way to fill a portion of a canvas with a solid color; this is for illustration purposes only.  The overall idea is that setting fewer array elements per pixel will perform better.
> 
> Have you actually measured this?  How long is spent in each part?  I suspect if you're not using the dirty region arguments you're pushing back (and doing premult conversion) on a lot more pixels than necessary.  Yes setting 4 properties is slower than setting 1, but where is your time actually being spent.
I have not directly done any measurements, sorry.  What I do have is a mecurial repository for a level editor project (which draws independent pixels directly to very large canvas elements) showing the progression of optimizations I've introduced.  Many of the modifications intended to make the drawing faster have done so by avoiding pixel access wherever possible.  Certainly it is not the most efficient code, but I've optimized enough to make the time spent setting pixel arrays worth investigating.  I still do not have any actual numbers to throw around, however.

>> We've already seen the emergence of emulators written in JavaScript/Canvas.  In fact, there are loads of them[4], and they would all benefit from having a better way to interact directly with canvas pixels.  Of course, the use cases are not limited to emulation; my NES/SNES level editor projects would enjoy faster pixel manipulation as well.  These kinds of projects can use arbitrarily sized canvases (up to 4864px × 3072px in one case[5]) and can take a good deal of time to fully render, even with several off-ImageData optimization tricks.
> 
> Without seeing the code for your demo i'd have no idea whether what you're doing is actually efficient -- have you profiled?  Both Safari and Firefox have built in profilers.

The trouble with profiling my project is that it is a XULRunner application, and does not run directly in web browsers as-is.  The code can largely be hacked to work as a web application.  If you are interested in a "demo" of sorts, the code is all available here:http://hg.kodewerx.org/index.cgi/syndrome/  and documentation here: http://www.kodewerx.org/wiki/index.php/Syndrome

Changeset 2b56c4771d5c reduced the number of pixel array elements accessed by caching the 256px x 256px "rooms" within the stage map, and passing the cached rooms to putImageData().  Once again, it is not the most efficient code.  There are several areas I could continue to improve, but that is not my point.

My point, as you concur, is that setting four array elements (properties) is slower than setting just one.  I am looking for a viable solution to this specific issue, and not merely working around it.  If a workaround was sufficient, I would not have come to the WHATWG for a discussion.

Thanks for listening!
Jason Oster
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.whatwg.org/pipermail/whatwg-whatwg.org/attachments/20091128/a1c8d57b/attachment-0002.htm>