[whatwg] Blending, filtering

[Rik Cabanier]

On Tue, Sep 25, 2012 at 12:55 PM, Ian Hickson <ian at hixie.ch> wrote:

>
> On Thu, 16 Aug 2012, David Geary wrote:
> >
> > It looks like there is general agreement that CSS filters should be
> > added to Canvas. Now how do we make it happen?
>
> On Thu, 16 Aug 2012, Rik Cabanier wrote:
> >
> > File a bug on it: [...]
>
> Actually no need to file a bug, just mentioning it in this mailing list is
> enough for it to get on the radar.
>
> If you do want to file a bug, though, you can. I recently created this
> short URL:
>
>    http://whatwg.org/newbug
>
> You can also use the little box in the spec at the bottom right, it files
> a bug also.
>
> (I only guarantee responses to new feedback on the spec sent to this
> mailing list; bugs may not get detailed responses, though they will all
> get looked at.)
>
>
> On Tue, 10 Apr 2012, Charles Pritchard wrote:
> >
> > About three or four years ago we talked about adding addition blend
> > modes and/or filters to the Canvas API. Is there still room for
> > discussion on that?
>
> There's always room for discussion. Discussions that don't introduce new
> content may get very short answers though. :-)
>
>
> On Tue, 24 Jan 2012, Ronald Jett wrote:
> >
> > I think that bringing the new CSS filters
> > (http://html5-demos.appspot.com/static/css/filters/index.html) to canvas
> > might be a good idea. Some of the new filters, specifically blur, would
> > definitely speed up some applications. I saw that there was a previous
> > discussion on this list about bringing SVG filters to canvas, but it was
> > a few years back and it doesn't seem like the discussion yielded much.
>
> On Mon, 16 Jul 2012, Ashley Gullen wrote:
> >
> > IMO the big use case here is games - the CSS filters are great for
> > interesting visual effects.
>
> On Tue, 10 Apr 2012, Jonas Sicking wrote:
> >
> > I recently was shown an canvas based game which had been forced to use
> > WebGL purely for the reason of needing blending modes. I believe the
> > use-case in that example was when the game character was shot at, the
> > background of the game should be rendered with a varying red tint. I
> > unfortunately forget the specifics of why rendering a partially
> > transparent red square over the background produced the wrong colors,
> > but if needed I can try to find the specifics.
>
> These use cases seem solid.
>
>
> On Tue, 24 Jan 2012, Ronald Jett wrote:
> >
> > It would be great if you could turn the filters on and off while
> > drawing. Something like:
> >
> > ctx.blur(20); // turns on a 20px blur
> > ctx.drawRect(0, 0, 50, 50); // this will be blurred
> > ctx.blur(0); // turns off blur
> > ctx.drawRect(100, 100, 50, 50); // this will not be blurred
> >
> > You could even do multiples:
> >
> > ctx.blur(2);
> > ctx.sepia(1);
> > ctx.drawImage(img, 0, 0);
> > ctx.endFilters(); // turn all filters off
> >
> > Another benefit of having these effects in canvas is that we could
> > utilize toDataURL to save out an image that a user/application has
> > filtered.
>
> On Mon, 16 Jul 2012, Ashley Gullen wrote:
> >
> > One way to define this is to specify that drawImage(), when passed
> another
> > canvas as a parameter, must take in to account the canvas' 'filter' CSS
> > property.  So to draw an image with a blur you'd render using an
> > intermediate canvas, e.g.
> >
> > tempcanvascontext.drawImage(myimage, 0, 0);
> > tempcanvas.style.filter = "blur(10px)";
> > gamecanvascontext.drawImage(tempcanvas, 0, 0); // draws with blur
> >
> > Another way would be just to add a 'filter' property to the 2D context,
> > e.g.:
> >
> > gamecanvascontext.filter = "blur(10px)";
> > gamecanvascontext.drawImage(myimage, 0, 0);
> >
> > This would also be extremely powerful if custom CSS shaders are also
> > supported, allowing for user-written effects in the canvas 2D context.
> >
> > Effects should should apply to all drawing operations for consistency,
> > including lines, paths, rectangles and patterns.
> >
> > Another argument is that you should just use WebGL and write shaders for
> > advanced effects.  This is an option, but given that a filter effect can
> > be applied to an entire canvas, it seems a waste not to enable it for
> > individual draw calls, especially given the 2D context is considerably
> > easier and quicker to code for.
>
> Adding such filters would be pretty useful...
>
>
> On Tue, 10 Apr 2012, Charles Pritchard wrote:
> >
> > https://dvcs.w3.org/hg/FXTF/rawfile/tip/compositing/index.html
>
> On Fri, 21 Sep 2012, Tyler Larson wrote:
> >
> > If we had access to matrix transformation on the pixels we wouldn't need
> > a sepia or contrast methods, all of this could be created as a simple
> > transform.
>
> On Fri, 21 Sep 2012, Tab Atkins Jr. wrote:
> >
> > Not true - having useful predefined things *in addition to* lower-level
> > primitives is good for authors.
> >
> > [...] once canvas supports CSS Filters, you can do per-pixel matrix
> > transformations via an SVG <feComponentMatrix> filter.  However, you
> > can't yet use SVG filters to do matrix transforms over multiple pixels,
> > like you need to do emboss or blur filters yourself.
>
> On Wed, 11 Apr 2012, Rik Cabanier wrote:
> >
> > I'm working on a spec to add blending and compositing through simple CSS
> > keywords. It is trying to define a generic model that is not specific to
> > Canvas, HTML or SVG and then lists how the model could be implemented.
> > We've gotten some comments that this feature would be useful in Canvas
> > as well so I was wondering if it made sense to add it to the canvas API.
>
> Is there any chance of adding filters to this also?
>

CSS Filters are already defined here:
https://dvcs.w3.org/hg/FXTF/raw-file/tip/filters/index.html.
The spec refer to each other for some concepts but are pretty separate.
(Filtering generally just works on 1 image while blending and compositing
describes how 2 images are combined)

One possibility would be to add a globalFilterOperation property that takes
the arguments of the 'filter' property [1]


>
> > I can see 2 ways of adding this:
> > 1. extend the list of compositing operators (
> >
> http://www.whatwg.org/specs/web-apps/current-work/multipage/the-canvas-element.html#compositing
> )
> > with blending. This is what is currently in the draft spec (
> > https://dvcs.w3.org/hg/FXTF/rawfile/tip/compositing/index.html chapter
> 7)
> >
> > 2. create a new attribute on the context called 'globalBlendOperation'
> that
> > takes the same list of blend operations as css (
> >
> https://dvcs.w3.org/hg/FXTF/rawfile/tip/compositing/index.html#blend-mode)
>
> Either (or both) of these seems reasonable to me. Would it make sense to
> have the canvas section defer to this spec for all blending, filtering,
> and compositing?
>

I think it does since then everyting will be in 1 location.
I would need to update the blending spec for canvas because it behaves
differently than HTML and SVG.


>
>
> On Fri, 27 Jul 2012, Rik Cabanier wrote:
> >
> > On another note, wouldn't it be nice if you could add a grouping
> > operator such as this:
> >
> > gamecanvascontext.filter = '...';
> > gamecanvascontext.beginGroup();
> > ... // lots of drawing operators
> > gamecanvascontext.endGroup();
> >
> > and have everything in that group at endGroup time?
>
> On Sat, 28 Jul 2012, Ashley Gullen wrote:
> >
> > Do you mean applying an effect to multiple draw operations?  Usually
> > that is achieved with rendering to an offscreen canvas, then rendering
> > that with the effect.
>
> On Sat, 28 Jul 2012, Rik Cabanier wrote:
> >
> > True, but you would have to know the size of the offscreen canvas which
> > is sometimes hard. I'm unsure what happens if you scale or rotate that
> > offscreen canvas. Will the artwork and text antialias correctly? How
> > does the up/downsampling happen?
>
> On Sun, 29 Jul 2012, Ashley Gullen wrote:
> >
> > Re: beginGroup()/endGroup(): I assume browsers would implement it as an
> > offscreen canvas anyway, so it would be better to write a JS library to
> > take care of it for you rather than requiring a browser feature for
> > that.
> >
> > You would not need to rotate or scale the off-screen canvas.  You'd make
> > it the size of the main canvas, draw everything with all the
> > rotation/scaling you want, then just draw it over the main canvas at (0,
> > 0) with 100% scale. This will not affect antialiasing or artwork
> > compared to just drawing it directly to the main canvas.  Fancier
> > implementations can work out the changed bounding box and only draw that
> > with the effect for efficiency.
>
> I haven't added grouping, for the reason described above. But if it's
> something that could benefit from native support from browsers and browser
> vendors want to implement it, we can definitely add something like this.
>

Grouping is definitely a heavy-weight process which introduces hidden
internal bitmaps.
It is a very handy feature but will impact performance if not handled with
care.


>
>
> On Fri, 21 Sep 2012, Tyler Larson wrote:
> >
> > I like the idea of abstracting this into an ArrayBuffer, I like the
> > sound of this but ArrayBuffers seem to promote nested arrays, whereas
> > the Canvas spec uses a simpler structure with r,g,b,a,… pixel values. It
> > would be awesome to have these types of inconsistencies worked out so we
> > didn't need to transform our data into different structures before
> > applying it.
> >
> > An abstraction between all of these contexts (Canvas,Element
> > Transforms,Audio,WebGL,...) seems doable but if it means sacrificing
> > performance later because what is output by the generic structure can't
> > be used in each of these APIs as is, I would rather have more targeted
> > transformation methods on each of these APIs.
>
> On Sat, 22 Sep 2012, Rob Manson wrote:
> > On Fri, 21 Sep 2012, Tyler Larson wrote:
> > > On Sep 21, 2012, at 3:34 AM, Jussi Kalliokoski
> > > <jussi.kalliokoski at gmail.com> wrote:
> > > > Please see the DSP API [1]. It's currently developed unofficially
> > > > under the W3C Audio WG [2], so if you have input, please post it to
> > > > the audiowg public mailing list. This should scratch your itch and
> > > > more. ;)
> > > >
> > > > [1] http://people.opera.com/mage/dspapi/
> > > > [2] http://www.w3.org/2011/audio/
> > >
> > > This stuff looks really interesting, and actually lower level than I
> > > was asking for which is cool but I'm not sure how this could be
> > > abstracted out so that it is useful for things outside of Audio? I'm
> > > likely just not putting it all together.
> >
> > I agree that this looks really useful and I don't think it's intimately
> > tied to Audio as it simply manipulates Typed Arrays.  In fact that the
> > page at link [1] doesn't seem to even contain the term "audio" at all.
> > So it's definitely just a generic DSP model built upon Type Arrays.  So
> > it seems like there's no need to raise this as feedback on the
> > ArrayBuffer spec at all since Type Arrays are built on top of
> > ArrayBuffers.
>
> I agree that it would be nice to have a generic way to mutate
> ArrayBuffers. I recommend bringing this up with the editor of the Typed
> Array specification.
>
>
> On Fri, 21 Sep 2012, Tab Atkins Jr. wrote:
> >
> > Usually, these kinds of things are written as a flat array of
> > components, with each item cycling between being an r, g, b, or a. This
> > is what is already used by ImageData, for example.  It seems reasonable
> > in the future to let the functions that take an ImageData also take an
> > ArrayBuffer directly.
>
> ImageData actually holds an actual ArrayBuffer, so we're pretty much
> there.
>
> --
> Ian Hickson               U+1047E                )\._.,--....,'``.    fL
> http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
> Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'


1:
https://dvcs.w3.org/hg/FXTF/raw-file/tip/filters/index.html#FilterProperty