[whatwg] Canvas hit regions

[Rik Cabanier]

On Fri, Mar 14, 2014 at 4:56 PM, Ian Hickson <ian at hixie.ch> wrote:

>
> I've done some more work on the spec for event rerouting for hit regions,
> based on the feedback sent to this list.
>
> On Wed, 5 Mar 2014, Robert O'Callahan wrote:
> > On Wed, Mar 5, 2014 at 12:53 PM, Ian Hickson <ian at hixie.ch> wrote:
> > > On Fri, 28 Feb 2014, Rik Cabanier wrote:
> > > > For instance, if the fallback is an edit control and the user
> > > > drag-selects some text on the canvas, is it expected that this text
> > > > is also selected in the edit control?
> > >
> > > You can't validly include a text field in canvas fallback precisely
> > > because of this kind of thing. See:
> > >
> > >    http://whatwg.org/html#best-practices
> >
> > The question remains: what should happen in Rik's example?
>
> If the control is a text edit control, the event isn't rerouted. This was
> always the intention (hit regions couldn't be set for text edit controls),
> but there was a loophole before, where you could register a hit region for
> one kind of control and then change that control to be something else.
> I've adjusted the spec to close that loophole.
>
> Event retargetting now explicitly applies to "the control represented by
> the region", which is always null if the given control is now a text
> field.
>

Does this change the eventTarget attribute on the event object [1]. It
doesn't seem like it does but should it?
I'm not an expert but it seems strange to send an event to an element with
a different eventTarget.


> > More generally, is this event rerouting supposed to be able to trigger
> > browser default event handling behavior, or only DOM event dispatch?
>
> As it was specified, I don't see how it could trigger default actions of
> anything other than the canvas and its ancestors. The canvas hook ran in
> the middle of the "When a pointing device is clicked, the user agent must
> run these steps" algorithm, which refers to the origin target, not the
> rerouted target.
>
> I've now changed this so that it does in fact trigger the default action
> if applicable.
>

This will still just reroute events, right?
For instance, if the fallback element is a <a href="...">, will clicking on
the region cause the browser to follow the hyperlink?


> On Wed, 5 Mar 2014, Robert O'Callahan wrote:
> >
> > The problem is that if these retargeted events can trigger default
> > browser behavior, the browser has to be able to compute the position of
> > the event relative to the new target DOM node, and it's not clear how to
> > do that.
>
> I've made it explicit that the elements that can get clicks targetted to
> them only include elements that don't have subregions. In particular,
> image maps and image buttons are excluded.
>

Thanks for updating the spec. It's getting quite complex though :-(
Maybe it's simpler to just add the id to the event and leave the canvas
element as the target? Since this is not a major feature, the complexity
might stop implementors.


> On Tue, 4 Mar 2014, Rik Cabanier wrote:
> > On Tue, Mar 4, 2014 at 8:30 PM, Ian Hickson <ian at hixie.ch> wrote:
> > > On Tue, 4 Mar 2014, Rik Cabanier wrote:
> > > > >
> > > > > So what would you do in the case where you start two touches on
> > > > > different regions, then move them at the same time to two other
> > > > > different regions at the same time? What would you put in the
> > > > > touchmove event's object?
> >
> > The touches attribute [1] of the touch event would contain 2 touch
> > elements.
> >
> > Each touch element would have as target the canvas element and contain
> > the id of the hit region.
>
> Oh so it's not the TouchEvent object you think should be adjusted, but the
> Touch object itself? (I'm assuming that's what you are referring to when
> you say "touch element".)
>

yes. Thanks for changing this.


>
> Presumably we would just set the region at creation time, like the
> "target" attribute, right?
>

yes

I've specced this.
>
>
> On Mon, 10 Mar 2014, Rik Cabanier wrote:
> >
> > Currently, the specification states that if you create a region and then
> > create another region that completely covers region, the first region is
> > removed from the hit region list [1]
> >
> > This is a complex operation that involves either drawing the regions to a
> > bitmaps and counting pixels, or path intersection.
>
> There's two trivial ways to implement this, depending on whether the hit
> regions are backed by a bitmap (the simplest and fastest solution but uses
> a lot of memory) or a region list (slower, but much more memory
> efficient). In the case of a bitmap, you just draw on the new region, and
> the old region is no longer in the bitmap, so it's trivially gone.

In the
> case of a list, you put the new region ahead of the old region so that you
> never actually get around to checking the old region.
>

The following step still needs to run though:
http://www.whatwg.org/specs/web-apps/current-work/multipage/the-canvas-element.html#garbage-collect-the-regions

Let victim be the first hit region in list to have an empty set of pixels
and a zero child count, if any.


If this was implemented with a bitmap, the only way to figure this out is
to walk the individual pixels (= expensive).

We should also not forget that a11y needs the bounding box of the hit
region which also means constantly walking of the pixels.


> In the latter case you would want to run "compression" algorithms every
> now and then to filter out old regions that no longer matter (though in
> most cases that would probably not be an issue since I expect most people
> will clear the whole canvas every frame, which can be trivially detected
> and can just cause the whole list to clear).
>
>
> > It is also unintuitive because an author might expect that he could
> > remove the second region and still have the first region active.
>
> That would be inconsistent with how canvas works. Canvas is an immediate-
> mode API. If you draw on the canvas, and then clear it, you don't get back
> what was there before.
>

What if an author doesn't clear it but just calls fillRect or is smart and
just invalidates/redraws portions of the canvas?
Hit regions should keep state, regardless of the canvas pixels.


> On Tue, 11 Mar 2014, Rik Cabanier wrote:
> > >
> > >
> http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-July/thread.html#36556
> >
> > It looks like that thread never came to a conclusion.
>
> Is there anything specifically that was raised in that thread that you
> think hasn't been responded to?
>

Well, you had the last response but I don't think it came to a conclusion
:-)
At the time, I was under the impression that we could mimic it with paths
but when I read the spec closer, the step that removes the region pixels is
too complex to implement and unintuitive for authors [2]


> > The arguments against using a bitmap presentation still stand:
> > - it will be too expensive to keep an actual copy of the canvas pixels
> > in memory to do hit testing
>
> It's actually pretty common to do exactly this. Note that you don't
> necessarily need a bitmap that has the same bit depth or pixel density as
> the visible bitmap.
>

Where else does this happen?
As Dirk mentioned, SVG does hit testing with paths as well.

Creating a canvas bitmap for constant reading will also be extremely costly
since so many implementations run canvas operations on the GPU.
I'm unsure if anyone supports a 8 bit backbuffer so at best, the hit region
bitmap is half the size. This is too expensive.


> > - if you have to mimic the behavior with paths, you need access to
> > expensive/difficult path manipulation algorithms
>
> The maths for determining if a region is contained in another region is
> pretty well understood at this point, as far as I can tell.
>

It's still a hard problem. Looking at Firefox' and Apple's implementation,
I don't know how they could determine if a path is contained within another
path.
Google has a library in development. We looked at it 6 months ago and it
had many issues.


> > Hit regions should be redesigned so they work on the path geometry as
> > opposed to pixels. We already have the necessary code to do hit testing
> > on paths (see isPointInPath)
>
> isPointInPath() works on pixels just like hit regions works on pixels.
>

No, this is not how it's implemented.
WebKit, Blink and Firefox use the geometry of the path. They don't use
pixels.


> On Wed, 12 Mar 2014, Dirk Schulze wrote:
> >
> > In SVG we tried to avoid having hit testing based on pixel values
> > obviously for performance reasons.
>
> SVG is a retained-mode API, so naturally it has a different model.
>

I don't see why. A browser draws the SVG DOM to a screen bitmap and then
does hit testing with fine paths.
Canvas could do the exact same thing.


> The whole point of <canvas> is to be an immediate-mode API that
> complements SVG.
>

People think of this API as populating a hit region OM. Why not go this
route?


> On Mon, 10 Mar 2014, Rik Cabanier wrote:
> >
> > clearRect is currently defined as a subtraction mechanism for hit
> > regions [1].
> >
> > It's unlikely that a UA will implement hit regions using pixels so this
> > would have to be done using path subtraction which is expensive.
>
> I'm not sure why you think it's expensive. It's trivial to just add a
> rectangle to the front of the list of regions.
>

That is true. This does mean that if there are a lot of small clearRect
calls, the list of regions could become very long.


> > Why was this special behavior added to clearRect?
>
> Because it's the most obvious mechanism for authors. You clear a part of
> the canvas, naturally that part of the canvas no longer has regions.
>

Why is that naturally? So, if an author clears an area there are no more
regions in it, but if he draws over it, they are still there?
Clipping also doesn't affect regions.


> On Tue, 4 Mar 2014, Rik Cabanier wrote:
> >
> > The spec implies--
>
> The spec doesn't imply anything. It either requires something, or doesn't.
> If you ever find yourself reading between the lines, then there is either
> a spec bug, or you are reading something that the spec doesn't require.
>

I know that. So, if I write "the spec implies", you can assume that I
believe that the spec is incomplete.

1: http://dom.spec.whatwg.org/#event
2:
http://www.whatwg.org/specs/web-apps/current-work/multipage/the-canvas-element.html#clear-regions-that-cover-the-pixels