[whatwg] Blending, filtering

[Ian Hickson]

On Thu, 16 Aug 2012, David Geary wrote:
>
> It looks like there is general agreement that CSS filters should be 
> added to Canvas. Now how do we make it happen?

On Thu, 16 Aug 2012, Rik Cabanier wrote:
>
> File a bug on it: [...]

Actually no need to file a bug, just mentioning it in this mailing list is 
enough for it to get on the radar.

If you do want to file a bug, though, you can. I recently created this 
short URL:

   http://whatwg.org/newbug 

You can also use the little box in the spec at the bottom right, it files 
a bug also.

(I only guarantee responses to new feedback on the spec sent to this 
mailing list; bugs may not get detailed responses, though they will all 
get looked at.)


On Tue, 10 Apr 2012, Charles Pritchard wrote:
> 
> About three or four years ago we talked about adding addition blend 
> modes and/or filters to the Canvas API. Is there still room for 
> discussion on that?

There's always room for discussion. Discussions that don't introduce new 
content may get very short answers though. :-)


On Tue, 24 Jan 2012, Ronald Jett wrote:
> 
> I think that bringing the new CSS filters 
> (http://html5-demos.appspot.com/static/css/filters/index.html) to canvas 
> might be a good idea. Some of the new filters, specifically blur, would 
> definitely speed up some applications. I saw that there was a previous 
> discussion on this list about bringing SVG filters to canvas, but it was 
> a few years back and it doesn't seem like the discussion yielded much.

On Mon, 16 Jul 2012, Ashley Gullen wrote:
> 
> IMO the big use case here is games - the CSS filters are great for
> interesting visual effects.

On Tue, 10 Apr 2012, Jonas Sicking wrote:
> 
> I recently was shown an canvas based game which had been forced to use 
> WebGL purely for the reason of needing blending modes. I believe the 
> use-case in that example was when the game character was shot at, the 
> background of the game should be rendered with a varying red tint. I 
> unfortunately forget the specifics of why rendering a partially 
> transparent red square over the background produced the wrong colors, 
> but if needed I can try to find the specifics.

These use cases seem solid.


On Tue, 24 Jan 2012, Ronald Jett wrote:
> 
> It would be great if you could turn the filters on and off while 
> drawing. Something like:
> 
> ctx.blur(20); // turns on a 20px blur
> ctx.drawRect(0, 0, 50, 50); // this will be blurred
> ctx.blur(0); // turns off blur
> ctx.drawRect(100, 100, 50, 50); // this will not be blurred
> 
> You could even do multiples:
> 
> ctx.blur(2);
> ctx.sepia(1);
> ctx.drawImage(img, 0, 0);
> ctx.endFilters(); // turn all filters off
> 
> Another benefit of having these effects in canvas is that we could 
> utilize toDataURL to save out an image that a user/application has 
> filtered.

On Mon, 16 Jul 2012, Ashley Gullen wrote:
> 
> One way to define this is to specify that drawImage(), when passed another
> canvas as a parameter, must take in to account the canvas' 'filter' CSS
> property.  So to draw an image with a blur you'd render using an
> intermediate canvas, e.g.
> 
> tempcanvascontext.drawImage(myimage, 0, 0);
> tempcanvas.style.filter = "blur(10px)";
> gamecanvascontext.drawImage(tempcanvas, 0, 0); // draws with blur
> 
> Another way would be just to add a 'filter' property to the 2D context,
> e.g.:
> 
> gamecanvascontext.filter = "blur(10px)";
> gamecanvascontext.drawImage(myimage, 0, 0);
> 
> This would also be extremely powerful if custom CSS shaders are also
> supported, allowing for user-written effects in the canvas 2D context.
> 
> Effects should should apply to all drawing operations for consistency,
> including lines, paths, rectangles and patterns.
> 
> Another argument is that you should just use WebGL and write shaders for 
> advanced effects.  This is an option, but given that a filter effect can 
> be applied to an entire canvas, it seems a waste not to enable it for 
> individual draw calls, especially given the 2D context is considerably 
> easier and quicker to code for.

Adding such filters would be pretty useful...


On Tue, 10 Apr 2012, Charles Pritchard wrote:
> 
> https://dvcs.w3.org/hg/FXTF/rawfile/tip/compositing/index.html

On Fri, 21 Sep 2012, Tyler Larson wrote:
> 
> If we had access to matrix transformation on the pixels we wouldn't need 
> a sepia or contrast methods, all of this could be created as a simple 
> transform.

On Fri, 21 Sep 2012, Tab Atkins Jr. wrote:
> 
> Not true - having useful predefined things *in addition to* lower-level 
> primitives is good for authors.
> 
> [...] once canvas supports CSS Filters, you can do per-pixel matrix 
> transformations via an SVG <feComponentMatrix> filter.  However, you 
> can't yet use SVG filters to do matrix transforms over multiple pixels, 
> like you need to do emboss or blur filters yourself.

On Wed, 11 Apr 2012, Rik Cabanier wrote:
> 
> I'm working on a spec to add blending and compositing through simple CSS 
> keywords. It is trying to define a generic model that is not specific to 
> Canvas, HTML or SVG and then lists how the model could be implemented. 
> We've gotten some comments that this feature would be useful in Canvas 
> as well so I was wondering if it made sense to add it to the canvas API.

Is there any chance of adding filters to this also?


> I can see 2 ways of adding this:
> 1. extend the list of compositing operators (
> http://www.whatwg.org/specs/web-apps/current-work/multipage/the-canvas-element.html#compositing)
> with blending. This is what is currently in the draft spec (
> https://dvcs.w3.org/hg/FXTF/rawfile/tip/compositing/index.html chapter 7)
> 
> 2. create a new attribute on the context called 'globalBlendOperation' that
> takes the same list of blend operations as css (
> https://dvcs.w3.org/hg/FXTF/rawfile/tip/compositing/index.html#blend-mode)

Either (or both) of these seems reasonable to me. Would it make sense to 
have the canvas section defer to this spec for all blending, filtering, 
and compositing?


On Fri, 27 Jul 2012, Rik Cabanier wrote:
> 
> On another note, wouldn't it be nice if you could add a grouping 
> operator such as this:
> 
> gamecanvascontext.filter = '...';
> gamecanvascontext.beginGroup();
> ... // lots of drawing operators
> gamecanvascontext.endGroup();
> 
> and have everything in that group at endGroup time?

On Sat, 28 Jul 2012, Ashley Gullen wrote:
>
> Do you mean applying an effect to multiple draw operations?  Usually 
> that is achieved with rendering to an offscreen canvas, then rendering 
> that with the effect.

On Sat, 28 Jul 2012, Rik Cabanier wrote:
> 
> True, but you would have to know the size of the offscreen canvas which 
> is sometimes hard. I'm unsure what happens if you scale or rotate that 
> offscreen canvas. Will the artwork and text antialias correctly? How 
> does the up/downsampling happen?

On Sun, 29 Jul 2012, Ashley Gullen wrote:
>
> Re: beginGroup()/endGroup(): I assume browsers would implement it as an 
> offscreen canvas anyway, so it would be better to write a JS library to 
> take care of it for you rather than requiring a browser feature for 
> that.
> 
> You would not need to rotate or scale the off-screen canvas.  You'd make 
> it the size of the main canvas, draw everything with all the 
> rotation/scaling you want, then just draw it over the main canvas at (0, 
> 0) with 100% scale. This will not affect antialiasing or artwork 
> compared to just drawing it directly to the main canvas.  Fancier 
> implementations can work out the changed bounding box and only draw that 
> with the effect for efficiency.

I haven't added grouping, for the reason described above. But if it's 
something that could benefit from native support from browsers and browser 
vendors want to implement it, we can definitely add something like this.


On Fri, 21 Sep 2012, Tyler Larson wrote:
> 
> I like the idea of abstracting this into an ArrayBuffer, I like the 
> sound of this but ArrayBuffers seem to promote nested arrays, whereas 
> the Canvas spec uses a simpler structure with r,g,b,a,… pixel values. It 
> would be awesome to have these types of inconsistencies worked out so we 
> didn't need to transform our data into different structures before 
> applying it.
> 
> An abstraction between all of these contexts (Canvas,Element 
> Transforms,Audio,WebGL,...) seems doable but if it means sacrificing 
> performance later because what is output by the generic structure can't 
> be used in each of these APIs as is, I would rather have more targeted 
> transformation methods on each of these APIs.

On Sat, 22 Sep 2012, Rob Manson wrote:
> On Fri, 21 Sep 2012, Tyler Larson wrote:
> > On Sep 21, 2012, at 3:34 AM, Jussi Kalliokoski 
> > <jussi.kalliokoski at gmail.com> wrote:
> > > Please see the DSP API [1]. It's currently developed unofficially 
> > > under the W3C Audio WG [2], so if you have input, please post it to 
> > > the audiowg public mailing list. This should scratch your itch and 
> > > more. ;)
> > > 
> > > [1] http://people.opera.com/mage/dspapi/
> > > [2] http://www.w3.org/2011/audio/  
> > 
> > This stuff looks really interesting, and actually lower level than I 
> > was asking for which is cool but I'm not sure how this could be 
> > abstracted out so that it is useful for things outside of Audio? I'm 
> > likely just not putting it all together.
> 
> I agree that this looks really useful and I don't think it's intimately 
> tied to Audio as it simply manipulates Typed Arrays.  In fact that the 
> page at link [1] doesn't seem to even contain the term "audio" at all. 
> So it's definitely just a generic DSP model built upon Type Arrays.  So 
> it seems like there's no need to raise this as feedback on the 
> ArrayBuffer spec at all since Type Arrays are built on top of 
> ArrayBuffers.

I agree that it would be nice to have a generic way to mutate 
ArrayBuffers. I recommend bringing this up with the editor of the Typed 
Array specification.


On Fri, 21 Sep 2012, Tab Atkins Jr. wrote:
>
> Usually, these kinds of things are written as a flat array of 
> components, with each item cycling between being an r, g, b, or a. This 
> is what is already used by ImageData, for example.  It seems reasonable 
> in the future to let the functions that take an ImageData also take an 
> ArrayBuffer directly.

ImageData actually holds an actual ArrayBuffer, so we're pretty much 
there.

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'