[whatwg] ImageData feedback

[Ian Hickson]

On Tue, 19 Feb 2008, Oliver Hunt wrote:
> 
> The first is relatively simple, 3.14.11.1.10 states
> "If any of the arguments to createImageData() or getImageData() are infinite
> or NaN, or if either the sw or sh arguments are zero, the method must instead
> raise an INDEX_SIZE_ERR exception."
> 
> I feel this should probably be extended to throw on negative sw and sh.

This is intended to support negative arguments, hence the mention of 
"absolute magnitude" in the definition of what the method does. This is 
for consistency with the other methods, which work fine with negative 
dimensions.


> Additionally sw and sh are both floats and can therefore have a fractional
> component, so this needs to be handled.  I think we should specify the finite
> value clamps in device pixels, eg.
>
> "If any of the arguments to createImageData() or getImageData() are infinite
> or NaN, or if either the sw or sh arguments are less than one when scaled to
> the device pixel space, the method must instead raise an INDEX_SIZE_ERR
> exception."

I don't think we want to make exception raising be dependent on the 
resolution of the backing store, since that's a UA decision. I think it 
makes more sense to make the height and width be always at least 1. Right 
now UAs are required to do this, though it's not explicitly stated in 
terms of rounding. I can add a note if you like.

(I've added createImageData() to the paragraph that talks about device 
rounding, by the way. This was an unintentional omission.)


> The other issues relate to the data member of ImageData.  Currently this 
> is referred to as an int array, however the DOM does not define such an 
> array, and certainly doesn't specify any kind of clamping (let alone 
> clamping to a smaller range than that supported by the type).  I think 
> the behaviour of this "array" needs to be defined, and givevn that only 
> Opera currently implements an actual ImageData type i've been looking at 
> their implementation.
>
> Assuming we have an ImageData object with width, w and height, h then Opera
> provides a data member of "CanvasPixelArray", with the following behaviour:
> * A readonly length property is provided return the value w*h*4
> * Assignment to elements [0..w*h*4) (approximately) follows the behaviour
> defined in section 3.14.11.1.10 (assignment never throws NaN is stored a zero
> which i believe is preferable)
> * Assignment to elements outside [0..w*h*4) results in no clamping and does
> not alter the length property, this includes non-numeric properties
> * Iteration returns only the length property and those properties that have
> been added programmatically -- iteration does *not* include [0..w*h*4)
> 
> By and large I am fine with this behaviour, although i would prefer that 
> it were not possible to add arbitrary properties to the data array -- 
> but i can't come up with a real justification for such a restriction :D

I've added a CanvasPixelArray interface and used that. (The setter and 
getter are temporarily named XXX5 and XXX6 while we wait for WebIDL to 
have a way to hide members from objects.)


On Thu, 21 Feb 2008, Oliver Hunt wrote:
>
> At the moment the spec merely states that 
> putImageData(getImageData(sx,sy,..),sx,sy) should not result in any 
> visible change to the canvas, however for those implementations that use 
> a premultiplied buffer there is a necessary premultiplication stage 
> during blitting that results in a loss of precision in some 
> circumstances -- the most obvious being the case of alpha == 0, but many 
> other cases exist, eg. (254, 254, 254, alpha < 255).  This loss of 
> precision has no actual effect on the visible output, but does mean that 
> in the following case:
>
> imageData = context.getImageData(0,0,...);
> imageData.data[0]=254;
> imageData.data[1]=254;
> imageData.data[2]=254;
> imageData.data[3]=1;
> context.putImageData(imageData,0,0);
> imageData2.data = context.getImageData(0,0,...);
> 
> At this point implementations that use premultiplied buffers can't 
> guarantee imageData.data[0] == imageData2.data[0]

They don't have to. As Philip says:

On Wed, 30 Apr 2008, Philip Taylor wrote:
> 
> The spec does not state that getImageData(putImageData(data)) == data,
> which is where the problem would occur. It only states that
> putImageData(getImageData) == identity function, which is not a
> problem for premultiplied implementations (since the conversion from
> premultiplied to non-premultiplied is lossless and reversible). So I
> don't think the spec needs to change at all (except that it could have
> a note mentioning the issue).
> 
> (getImageData can convert internal premultiplied (pr,pg,pb,a) into
> ImageData's (r,g,b,a):
> 
>     if (a == 0) {
>         r = g = b = 0;
>     } else {
>         r = (pr * 255) / a;
>         g = (pg * 255) / a;
>         b = (pb * 255) / a;
>     }
> 
> (using round-to-zero integer division). putImageData can convert the other way:
> 
>     pr = (r*a + 254) / 255;
>     pg = (g*a + 254) / 255;
>     pb = (b*a + 254) / 255;
> 
> Then put(get()) has no effect on the values in the premultiplied buffer.)


On Thu, 21 Feb 2008, Oliver Hunt wrote:
>
> Currently no UA can guarantee a roundtrip so i would suggest the spec be
> updated to state that implementations do not have to guarantee a roundtrip for
> any pixel where alpha < 255.

The spec doesn't require roundtrip, so I'm not sure what the note would 
say.


> The other issue is the behaviour of NaN in ImageData.  Currently the 
> spec states that attempting to assign NaN (or any value for which 
> toNumber produces NaN) into the ImageData object should throw.  I feel 
> that this is another place where NaN should be simply ignored, which is 
> the behaviour given by Opera, which is the only UA that implements the 
> ImageData API (that i'm aware of).
> 
> In Opera's implementation all non-finite numbers are stored as zero, which is
> (to my mind) much better than throwing an exception, however i would expect
> NaN => 0
> -Infinity => 0
> Infinity => 255
> 
> So +/-Infinity would treated in line with standard clamping rules.

I believe the spec now requires this.



On Sat, 10 May 2008, Vladimir Vukicevic wrote:
> > 
> > This restriction was made because it allows for dramatic (many orders 
> > of magnitude) optimisations. With createImageData(), the use cases for 
> > custom imageData objects should be catered for -- what are the cases 
> > where you would need another solution? (Note that hand-rolling 
> > imageData objects is dangerous since you don't know what resolution 
> > the backing store is using, necessarily, which is something else that 
> > createImageData() solves.)
> 
> Well, I don't agree that it's "dangerous"; canvas resolution 
> independence has always been hard to pin down, and I still maintain that 
> it shouldn't be treated any differently than an image is treated.  

"dangerous" may be the wrong term. The real problem is that there's no 
real way to find out what the dimensions should be without calling 
createID() or getID(), and once you've done that you might as well just 
reuse the object you were handed.


> However, regardless of that, I don't think there's a reason to disallow 
> custom-created data objects, perhaps with a caveat that there may be 
> issues.. get/putImageData in Firefox 2, so adding that restriction may 
> unnecessarily break existing code that uses putImageData with a hand- 
> constructed ImageData object.

The restriction was put in because of a request from the WebKit guys to 
enable them to do order-of-magnitude performance improvements. On the long 
term, I think we're better of enabling those performance improvements than 
supporting the earliest users of the API, especially given that those uses 
only ever worked in one UA.


> I would amend the spec to state that if an object is passed to 
> putImageData with the necessary properties, but without having been 
> created by create/getImageData beforehand, that its dimensions are aways 
> in device pixels.

We could support a conversion from custom (JS) objects to native ImageData 
objects when the method is passed a non-native object, but that seems like 
a lot of complications for little gain. After all, calling 
createImageData() isn't an onerous requirement.


> One problem with the desired goal of resolution independence is that it 
> only really makes sense if the target resolution is an integer multiple 
> of a CSS pixel.  For example, with a 144dpi output device, that's 
> exactly 1.5 times CSS resolution.  If I call createImageData with 
> dimensions 10,10, I would get an ImageData object with width 15 and 
> height 15.  What do I get if I call it with 11,11 though?  That's 16.5 
> device pixels, and you're going to lose data either way you go, because 
> at putImageData time you're going to get seams no matter what direction 
> you round.  This can maybe be solved with language in the spec that 
> specifies that a canvas should use a different ratio of CSS pixels to 
> device pixels only if one is an integer multiple of the other, but that 
> seems like an odd limitation (and it still requires the implementation 
> to decide what to do if a clean ratio isn't possible).

I don't understand what use cases you are expecting where this would be a 
problem. Typically the idea here is doing filters or generating patterns 
like clouds or fractals, in which case, so long as the UA follows the 
spec's rules on rounding behaviour, the seams shouldn't be a big problem. 
Could you illustrate the case you are concerned about with an example?


> Another approach would be to not try to solve this in canvas at all, and
> instead specify that by default, all canvas elements are 96dpi, and provide
> authors a way to explicitly override this -- then using a combination of CSS
> Media Queries and other CSS, the exact dpi desired could be specified.  (You
> can sort of do this today, given that the canvas width/height attributes are
> in CSS pixels, and that if CSS dimensions are present a canvas is scaled like
> an image... so canvas { width: 100px; height: 100px; } ... <canvas width="200"
> height="200"/> would give a 192dpi canvas today, no?)

A UA would be quite within its rights today to make that a canvas with a 
native backing store dimension of 100x100. It could also make it 
4000x4000. There's really no reason for the user to know.


On Sat, 10 May 2008, Vladimir Vukicevic wrote:
> 
> Eh?  The resolution used should be whatever the appropriate resolution 
> should be;  I'm certainly not suggesting that everyone unilaterally 
> create canvases with 2x pixel resolution, I'm saying that the features 
> exist to allow authors to (dynamically) create a canvas at whatever the 
> appropriate resolution is relative to CSS resolution.  Canvas was 
> designed to allow for programmatic 2D rendering for web content; 
> resolution independence would certainly be nice, but it was never a goal 
> of the canvas spec.  In fact, the spec explicitly states that the 
> "canvas element represents a resolution-dependent bitmap canvas".

Right, the resolution in "resolution-dependent" being the display's native 
resolution. It was always intended (at least from my point of view, and as 
I understand it from the original Apple proposal) that <canvas> provide a 
way to render bitmap graphics that will always be the "best" resolution 
for the display. Resolution-independent from the author's point of view.


[snipped a discussion about whether to change some of the current 
validation rules, since it concluded that the current spec was fine]

On Tue, 13 May 2008, Vladimir Vukicevic wrote:
> 
> Some suggested language in section 3.12.11.1.11(!):
> 
> Instead of:
> 
> > If the first argment to the method is null or not an ImageData object 
> > that was returned by createImageData() or getImageData() then the 
> > putImageData() method must raise a TYPE_MISMATCH_ERR exception.
> 
> I would suggest:
> 
> The first argument to the method must be an ImageData object returned by 
> createImageData(), getImageData(), or an object constructed with the 
> necessary properties by the user.  If the object was constructed by the 
> user, its width and height dimensions are specified in device pixels 
> (which may not map directly to CSS pixels).  If null or any other object 
> is given that does not present the ImageData interface, then the 
> putImageData() method must raise a TYPE_MISMATCH_ERR exception.

On Tue, 13 May 2008, Oliver Hunt wrote:
> 
> If we were to add that we should include a note to indicate that using a 
> custom object is not recommended -- Any code that uses a custom created 
> object will never benefit from improvements in ImageData performance 
> made by the UA. That said I still don't believe custom objects should be 
> allowed, aside from the resolution (which may or may not be relevant) 
> and performance issues, a custom object with a generic JS array, rather 
> than an ImageData object will have different behaviour -- a proper 
> ImageData will clamp on assignment, and throw in cases that a custom 
> object won't.

On Tue, 13 May 2008, Vladimir Vukicevic wrote:
> 
> I'm fine with adding that language (the first part, anyway); something 
> like "Using a custom object is not recommended as the UA may be able to 
> optimize operations using ImageData if they were created via 
> createImageData() or getImageData()."

On Mon, 2 Jun 2008, Vladimir Vukicevic wrote:
> 
> [...] based on the discussion, I'd suggest:
> 
> - user-created ImageData-like objects should be supported, e.g. with 
> language such as:
> 
> The first argument to the method must be an ImageData object returned by 
> createImageData(), getImageData(), or an object constructed with the 
> necessary properties by the user.  If the object was constructed by the 
> user, its width and height dimensions are specified in device pixels 
> (which may not map directly to CSS pixels).  If null or any other object 
> is given that does not present the ImageData interface, then the 
> putImageData() method must raise a TYPE_MISMATCH_ERR exception.

On Mon, 9 Jun 2008, Jonas Sicking wrote:
> 
> Do note that dealing with user-created objects isn't trivial. You have 
> to be prepared for dealing with the user-created object changing the 
> whole world under you during a callback, this includes things like doing 
> any modification to the canvas object itself, but also things like 
> script navigating away and then causing a GC to tear down the world 
> around you.
> 
> This is usually not very hard to deal with, as long as you are not deep 
> inside a long callstack when calling out to content. This is because you 
> have to ensure that the whole callstack is ok with the world changing 
> around it.

The request seems to be aimed at addressing the small number of existing 
sites already deployed that rely on the Firefox 2 implementation. There 
doesn't seem to be an actual use case for the feature other than that.

Given this, and given that other UAs have already made changes that are 
incompatible with these very, very early pages in order to line up with 
the needs of implementations such as Mozilla, I don't think it really 
makes sense to add this to the spec. It isn't at all clear that other UAs 
actually want to support this anyway; indeed even Mozilla implementors 
seem reluctant to continue to support it (q.v. Jonas' comment above).

Vlad: I realise that this basically consists of rejecting your feedback on 
this, but I'm not sure how else to proceed given the clear statement from 
at least one other vendor that they don't want this feature. Sorry.

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'