[whatwg] custom ImageData objects

[Philip Taylor]

On 14/05/07, Maciej Stachowiak <mjs at apple.com> wrote:
> If we disallow scaling the backing store for a possible UI scale
> factor, then <canvas> content will look significantly worse than
> equivalent SVG content at the same resolution. <canvas> already has
> the problem of worse printed output, as screens become higher and
> higher DPI the same problem will start happening on-screen. It will
> also lead to wasted memory if your UI is scaled down.
>
> I think the only things that expose this scale factor are ImageData
> and toDataURL(). It would be a shame to limit the output quality of
> <canvas> on high DPI screens solely for these two features.
>
> [...]
>
> I'm not sure of the right fix for ImageData. One possibility is that
> ImageData is in <canvas> coordinates and the pixel values are
> averaged if the backing store is scaled, but then putImageData
> (getImageData(...)) could not be idempotent. Another possibility is
> to have ImageData contain representations at both <canvas> resolution
> and true backing store resolution, so authors have the possibility of
> ignoring scale factor or not. But then you couldn't just use an
> arbitrary JS object as an ImageData, since the two representations
> would have to be kept in sync.

Looking at all the places I can see getImageData/putImageData being
used, there appears to be four main categories:

Reading a single pixel:
  http://canvaspaint.org/paint.js
  http://www.sweetvision.com/2007/03/25/enhanceing-the-javascript-color-picker-with-the-canvas-element/
  http://canvex.lazyilluminati.com/tests/tests/

Drawing pixels:
  http://svn.sourceforge.net/viewvc/jsmsx/trunk/msx.js?view=markup
  http://www.thescripts.com/forum/thread521112.html

Image filtering:
  http://oxine.opera.com/gallery/canvas/2dgame/sepia.html [not
actually ImageData, but close]
  http://tech.groups.yahoo.com/group/canvas-developers/files/buttons.html

Copying images:
  http://groups.google.com/group/mozilla.dev.apps.firefox/browse_thread/thread/742111eaf2e3e0da/b8777a855fcfa32b
  http://forums.mozillazine.org/viewtopic.php?=&p=2759433


For reading a single pixel, any form of getImageData should be fine -
perfect accuracy isn't expected, and it just needs to correspond to
nearly what the user thinks they're pointing at.

For drawing pixels, you could use fillRect(x, y, 1, 1) but it's
impractically slow - http://canvex.lazyilluminati.com/misc/filter.html
in Firefox goes over 50 times slower with fillRect vs putImageData, so
something like putImageData is very useful. For cases like jsMSX,
which wants to draw to individual <canvas> pixels using
get/putImageData, the ImageData needs to contain canvas pixels so the
JS code doesn't have to do complex ImageData<->canvas pixel mapping
itself.

For copying images, drawImage should be used unless you need to copy
outside the browser (e.g. to a server or local storage), and toDataURL
should be used if you do want to (because ImageData is horribly
space-inefficient), so I don't think these cases are relevant to
ImageData at all. (I'm not thinking about toDataURL for now.)

For image filtering, you probably do want perfect accuracy/resolution
to get the best possible output. In some cases (like blurring) you
also want to know the mapping between ImageData pixels and <canvas>
pixels, so that the effect of the filter can be independent of the
resolution. Particularly in those cases, you have to be quite careful
to get it right if you're only ever testing in browsers with a 1:1
pixel mapping, though it's not that hard once you understand the
issues and if you can find someone to test and report bugs.



So, I was considering the function
    getImageData(sx, sy, sw, sh, hires)
where hires=true makes it return an ImageData with however many pixels
the browser is using to represent the image. (I think it's reasonable
to assume they're always representing it internally as a 32-bit bitmap
and not as a vector format or something crazy...). The default
hires=false/undefined (or unspecified) makes it return an ImageData
with exactly sw*sh pixels, doing some kind of filtering/averaging if
it's got a higher resolution internally - when people don't care about
high-res browsers, or are intentionally trying to work with <canvas>
pixels, and use the normal basic form of getImageData, then it will
just do the right thing.

hires=false will lose some data and reduce the quality of the canvas
output if you're drawing the ImageData back again, but this shouldn't
impact high-res browser users much since most canvas code won't use
ImageData at all. When authors do use ImageData but don't know or care
about high-res users, at least their code will still work correctly,
and just be a bit blurrier.

When authors do care, then they can set hires=true and it's up to them
to do it right, and nothing will save us if they set that flag and
still get it wrong.

(The addition of a hires argument is also compatible with existing
implementations (or at least Firefox) and existing content (which
assumes the hires=false behaviour), and means browsers which don't
care about high-res displays won't have to do any new implementation
work at all. Incidentally, this could be an argument for allowing (and
ignoring) unexpected arguments in all function calls, to permit easier
extension.)

For authors who use hires=true and need to determine the device:canvas
ratio, they could calculate ImageData.width/sw, or ImageData could
provide an extra field specifying the ratio. (The latter would give a
more precise value, but would lose compatibility with browsers that
ignore hires=true and treat it identically to hires=false.)


putImageData needs an equivalent flag - presumably either
putImageData(data, sx, sy, hires), or ImageData could have a 'hires'
attribute added when it came from a call to getImageData(..., true) so
there's less chance of forgetting to specify the flag when doing a
get->filter->put sequence.


A similar alternative would be to have entirely separate functions
(perhaps getImageData and getRawPixelData or something) instead of a
flag, with the same effect.


>
> Regards,
> Maciej
>

-- 
Philip Taylor
excors at gmail.com