[whatwg] Adding features needed for WebGL to ImageBitmap

[Gregg Tavares]

On Wed, Jun 19, 2013 at 4:03 PM, Rik Cabanier <cabanier at gmail.com> wrote:

>
>
> On Wed, Jun 19, 2013 at 3:24 PM, Gregg Tavares <gman at google.com> wrote:
>
>>
>>
>>
>> On Wed, Jun 19, 2013 at 3:13 PM, Rik Cabanier <cabanier at gmail.com> wrote:
>>
>>>
>>> On Wed, Jun 19, 2013 at 2:47 PM, Gregg Tavares <gman at google.com> wrote:
>>>
>>>> In order for ImageBitmap to be useful for WebGL we need more options
>>>>
>>>> Specifically
>>>>
>>>> premultipliedAlpha: true/false (default true)
>>>> Nearly all GL games use non-premultipiled alpha textures. So all those
>>>> games people want to port to WebGL will require non-premultipied
>>>> textures.
>>>> Often in games the alpha might not even be used for alpha but rather for
>>>> glow maps or specular maps or the other kinds of data.
>>>>
>>>
>>> When would premultipliedAlpha ever be true?
>>> 2D Canvas always works with non-premultiplied alpha in its APIs.
>>>
>>
>> AFAIK the canvas API expects all images to be premultiplied. Certainly in
>> WebKit and Blink images used in the canvas and displayed in the img tag are
>> loaded premultiplied which is why we had to add the option on WebGL since
>> we needed those images before they had lost data.
>>
>>
>>>
>>>
>>>>
>>>> flipY: true/false (default false)
>>>> Nearly all 3D modeling apps expect the bottom left pixel to be the first
>>>> pixel in a texture so many 3D engines flip the textures on load. WebGL
>>>> provides this option but it takes time and memory to flip a large image
>>>> therefore it would be nice if that flip happened before the callback
>>>> from
>>>> ImageBitmap
>>>>
>>>
>>> Couldn't you just draw upside down?
>>>
>>
>> No, games often animate texture coordinates and other things making it
>> far more complicated. There are ways to work around this issue in code yes,
>> they often require a ton of work.
>>
>> Most professional game engines pre-process the textures and flip them
>> offline but that doesn't help when you're downloading models off say
>> http://sketchup.google.com/3dwarehouse/
>>
>>
>>>
>>>
>>>>
>>>> colorspaceConversion: true/false (default true)
>>>> Some browsers apply color space conversion to match monitor settings.
>>>> That's fine for images with color but WebGL apps often load heightmaps,
>>>> normalmaps, lightmaps, global illumination maps and many other kinds of
>>>> data through images. If the browser applies a colorspace conversion the
>>>> data is not longer suitable for it's intended purpose therefore many
>>>> WebGL
>>>> apps turn off color conversions. As it is now, when an image is
>>>> uploaded to
>>>> WebGL, if colorspace conversion is
>>>> off<
>>>> http://www.khronos.org/registry/webgl/specs/latest/#PIXEL_STORAGE_PARAMETERS
>>>> >,
>>>
>>>
> OK, I see what you're trying to accomplish. You want to pass
> non-premultiplied data and color converted (from sRGB to monitor) pixels to
> WebGL
> I think your API looks fine, except that the defaults should all be
> false...
>

Yes, that's what I meant. I think I choose bad labels. the intent of the
colorspaceConversion flag is

colorspaceConversion: true   = browser does whatever it thinks is best for
color images.
colorspaceConversion: false  = give me the bits in the image file. Don't
manipulate them with either embedded color data or local machine gamma
corrections or anything else.

So maybe a better name?

for premultipiedAlpha again, maybe there are 3 options needed

1) do whatever is best for drawing with drawImage for perf
2) give me the data with premutlipied alpha
3) give me the data with non-premultied alpha.

It's possible that #1 is not needed as maybe GPU code can use different
blend modes for drawImage with non-premultpiled alpha. It's just my
understanding that at least in Chrome all images are loaded premultiplied.
In fact I don't think you can get non-premultipied data from canvas. At
least this does not make it appear that way

    c = document.createElement("canvas");
    ctx = c.getContext("2d");
    i = ctx.getImageData(0, 0, 1, 1);
    i.data[0] = 255;
    ctx.putImageData(i, 0, 0);
    i2 = ctx.getImageData(0, 0, 1, 1);
    console.log(i2.data[0])  // prints 0 on both FF and Chrome

I mean I know you get unpremultiplied data from getIamgeData but the data
in the canvas is premultipied which means if alpha is zero you lose the RGB
data. In other words a round trip of putImageData, getImageData is lossy
for any non 255 alpha.  Change the above to

    c = document.createElement("canvas");
    ctx = c.getContext("2d");
    i = ctx.getImageData(0, 0, 1, 1);
    i.data[0] = 128;
    i.data[3] = 1;
    ctx.putImageData(i, 0, 0);
    i2 = ctx.getImageData(0, 0, 1, 1);
    console.log(i2.data[0])  // prints 255.

This is because the data in the canvas is premutliplied and it's being
un-premultiplied when calling getImageData











>
>
>>
>>>> WebGL has to synchronously re-decode the image. It would be nice if
>>>> ImageBitmap could handle this case so it can decode the image without
>>>> applying any colorspace manipulations.
>>>>
>>>
>>> Shouldn't the color space conversion happen when the final canvas bit
>>> are blitted to the screen?
>>> It seems like you should never do it during compositing since you could
>>> get double conversions.
>>>
>>
>> Maybe but that's not relevant to ImageBitmap is it? The point here is we
>> want the ImageBitmap to give us the data in the format we need. It's
>> designed to be async so it can do this but it we can't prevent it from
>> applying colorspace conversions.
>> Some browsers did that for regular img tags which pointed out the
>> original problem. The browser can't guess how the image is going to be used
>> and since it's a lot of work to decode an image you'd like to be able to
>> tell it what you really need before it guesses wrong.
>>
>>
>>>
>>>
>>>>
>>>> If it was up to me I'd make createImageBitmap take on object with
>>>> properties so that new options can be added later as in
>>>>
>>>>     createImageBitmap(src, callback, {
>>>>        premultipliedAlpha: false,
>>>>        colorspaceConversion: false,
>>>>        x: 123,
>>>>     });
>>>>
>>>> But I'm not familiar if there is a common way to make APIs take a
>>>> options
>>>> like this except for the XHR way which is to create a request, set
>>>> properties on the request, and finally execute the request.
>>>
>>>
>>>  Like Tab said, it's fine to implement it that way.
>>> Be aware that you might have to do some work in your idl compiler since
>>> I *think* there are no other APIs (in Blink) that take a dictionary.
>>>
>>>
>>
>