[whatwg] Adding features needed for WebGL to ImageBitmap

[Rik Cabanier]

On Wed, Jun 19, 2013 at 4:47 PM, Gregg Tavares <gman at google.com> wrote:

>
>
>
> On Wed, Jun 19, 2013 at 4:03 PM, Rik Cabanier <cabanier at gmail.com> wrote:
>
>>
>>
>> On Wed, Jun 19, 2013 at 3:24 PM, Gregg Tavares <gman at google.com> wrote:
>>
>>>
>>>
>>>
>>> On Wed, Jun 19, 2013 at 3:13 PM, Rik Cabanier <cabanier at gmail.com>wrote:
>>>
>>>>
>>>> On Wed, Jun 19, 2013 at 2:47 PM, Gregg Tavares <gman at google.com> wrote:
>>>>
>>>>> In order for ImageBitmap to be useful for WebGL we need more options
>>>>>
>>>>> Specifically
>>>>>
>>>>> premultipliedAlpha: true/false (default true)
>>>>> Nearly all GL games use non-premultipiled alpha textures. So all those
>>>>> games people want to port to WebGL will require non-premultipied
>>>>> textures.
>>>>> Often in games the alpha might not even be used for alpha but rather
>>>>> for
>>>>> glow maps or specular maps or the other kinds of data.
>>>>>
>>>>
>>>> When would premultipliedAlpha ever be true?
>>>> 2D Canvas always works with non-premultiplied alpha in its APIs.
>>>>
>>>
>>> AFAIK the canvas API expects all images to be premultiplied. Certainly
>>> in WebKit and Blink images used in the canvas and displayed in the img tag
>>> are loaded premultiplied which is why we had to add the option on WebGL
>>> since we needed those images before they had lost data.
>>>
>>>
>>>>
>>>>
>>>>>
>>>>> flipY: true/false (default false)
>>>>> Nearly all 3D modeling apps expect the bottom left pixel to be the
>>>>> first
>>>>> pixel in a texture so many 3D engines flip the textures on load. WebGL
>>>>> provides this option but it takes time and memory to flip a large image
>>>>> therefore it would be nice if that flip happened before the callback
>>>>> from
>>>>> ImageBitmap
>>>>>
>>>>
>>>> Couldn't you just draw upside down?
>>>>
>>>
>>> No, games often animate texture coordinates and other things making it
>>> far more complicated. There are ways to work around this issue in code yes,
>>> they often require a ton of work.
>>>
>>> Most professional game engines pre-process the textures and flip them
>>> offline but that doesn't help when you're downloading models off say
>>> http://sketchup.google.com/3dwarehouse/
>>>
>>>
>>>>
>>>>
>>>>>
>>>>> colorspaceConversion: true/false (default true)
>>>>> Some browsers apply color space conversion to match monitor settings.
>>>>> That's fine for images with color but WebGL apps often load heightmaps,
>>>>> normalmaps, lightmaps, global illumination maps and many other kinds of
>>>>> data through images. If the browser applies a colorspace conversion the
>>>>> data is not longer suitable for it's intended purpose therefore many
>>>>> WebGL
>>>>> apps turn off color conversions. As it is now, when an image is
>>>>> uploaded to
>>>>> WebGL, if colorspace conversion is
>>>>> off<
>>>>> http://www.khronos.org/registry/webgl/specs/latest/#PIXEL_STORAGE_PARAMETERS
>>>>> >,
>>>>
>>>>
>> OK, I see what you're trying to accomplish. You want to pass
>> non-premultiplied data and color converted (from sRGB to monitor) pixels to
>> WebGL
>> I think your API looks fine, except that the defaults should all be
>> false...
>>
>
> Yes, that's what I meant. I think I choose bad labels. the intent of the
> colorspaceConversion flag is
>
> colorspaceConversion: true   = browser does whatever it thinks is best for
> color images.
> colorspaceConversion: false  = give me the bits in the image file. Don't
> manipulate them with either embedded color data or local machine gamma
> corrections or anything else.
>
> So maybe a better name?
>

yeah. I'm still fuzzy about what exactly you're trying to do. Is this
specified anywhere?


>
> for premultipiedAlpha again, maybe there are 3 options needed
>
> 1) do whatever is best for drawing with drawImage for perf
> 2) give me the data with premutlipied alpha
> 3) give me the data with non-premultied alpha.
>

I think that sounds good. That way implementations aren't forced to
premultiply the alpha.
Maybe not passing premultipiedAlpha in the dictionary (= leave it
undefined) will signal this.


>
> It's possible that #1 is not needed as maybe GPU code can use different
> blend modes for drawImage with non-premultpiled alpha. It's just my
> understanding that at least in Chrome all images are loaded premultiplied.
> In fact I don't think you can get non-premultipied data from canvas. At
> least this does not make it appear that way
>
>     c = document.createElement("canvas");
>     ctx = c.getContext("2d");
>     i = ctx.getImageData(0, 0, 1, 1);
>     i.data[0] = 255;
>     ctx.putImageData(i, 0, 0);
>     i2 = ctx.getImageData(0, 0, 1, 1);
>     console.log(i2.data[0])  // prints 0 on both FF and Chrome
>
> I mean I know you get unpremultiplied data from getIamgeData but the data
> in the canvas is premultipied which means if alpha is zero you lose the RGB
> data. In other words a round trip of putImageData, getImageData is lossy
> for any non 255 alpha.  Change the above to
>
>     c = document.createElement("canvas");
>     ctx = c.getContext("2d");
>     i = ctx.getImageData(0, 0, 1, 1);
>     i.data[0] = 128;
>     i.data[3] = 1;
>     ctx.putImageData(i, 0, 0);
>     i2 = ctx.getImageData(0, 0, 1, 1);
>     console.log(i2.data[0])  // prints 255.
>
> This is because the data in the canvas is premutliplied and it's being
> un-premultiplied when calling getImageData
>

Yes, the canvas spec calls this out:

Due to the lossy nature of converting to and from premultiplied alpha color
values, pixels that have just been set using
putImageDataHD()<http://www.whatwg.org/specs/web-apps/current-work/multipage/the-canvas-element.html#dom-context-2d-putimagedatahd>
might
be returned to an equivalent
getImageDataHD()<http://www.whatwg.org/specs/web-apps/current-work/multipage/the-canvas-element.html#dom-context-2d-getimagedatahd>
as
different values.

I think this is really unfortunate...


>
>

>>
>>>
>>>>> WebGL has to synchronously re-decode the image. It would be nice if
>>>>> ImageBitmap could handle this case so it can decode the image without
>>>>> applying any colorspace manipulations.
>>>>>
>>>>
>>>> Shouldn't the color space conversion happen when the final canvas bit
>>>> are blitted to the screen?
>>>> It seems like you should never do it during compositing since you could
>>>> get double conversions.
>>>>
>>>
>>> Maybe but that's not relevant to ImageBitmap is it? The point here is we
>>> want the ImageBitmap to give us the data in the format we need. It's
>>> designed to be async so it can do this but it we can't prevent it from
>>> applying colorspace conversions.
>>> Some browsers did that for regular img tags which pointed out the
>>> original problem. The browser can't guess how the image is going to be used
>>> and since it's a lot of work to decode an image you'd like to be able to
>>> tell it what you really need before it guesses wrong.
>>>
>>>
>>>>
>>>>
>>>>>
>>>>> If it was up to me I'd make createImageBitmap take on object with
>>>>> properties so that new options can be added later as in
>>>>>
>>>>>     createImageBitmap(src, callback, {
>>>>>        premultipliedAlpha: false,
>>>>>        colorspaceConversion: false,
>>>>>        x: 123,
>>>>>     });
>>>>>
>>>>> But I'm not familiar if there is a common way to make APIs take a
>>>>> options
>>>>> like this except for the XHR way which is to create a request, set
>>>>> properties on the request, and finally execute the request.
>>>>
>>>>
>>>>  Like Tab said, it's fine to implement it that way.
>>>> Be aware that you might have to do some work in your idl compiler since
>>>> I *think* there are no other APIs (in Blink) that take a dictionary.
>>>>
>>>>
>>>
>>
>