[whatwg] Canvas hit regions

[Ian Hickson]

I've done some more work on the spec for event rerouting for hit regions, 
based on the feedback sent to this list.

On Wed, 5 Mar 2014, Robert O'Callahan wrote:
> On Wed, Mar 5, 2014 at 12:53 PM, Ian Hickson <ian at hixie.ch> wrote:
> > On Fri, 28 Feb 2014, Rik Cabanier wrote:
> > > For instance, if the fallback is an edit control and the user 
> > > drag-selects some text on the canvas, is it expected that this text 
> > > is also selected in the edit control?
> >
> > You can't validly include a text field in canvas fallback precisely 
> > because of this kind of thing. See:
> >
> >    http://whatwg.org/html#best-practices
> 
> The question remains: what should happen in Rik's example?

If the control is a text edit control, the event isn't rerouted. This was 
always the intention (hit regions couldn't be set for text edit controls), 
but there was a loophole before, where you could register a hit region for 
one kind of control and then change that control to be something else. 
I've adjusted the spec to close that loophole.

Event retargetting now explicitly applies to "the control represented by 
the region", which is always null if the given control is now a text 
field.


> More generally, is this event rerouting supposed to be able to trigger 
> browser default event handling behavior, or only DOM event dispatch?

As it was specified, I don't see how it could trigger default actions of 
anything other than the canvas and its ancestors. The canvas hook ran in 
the middle of the "When a pointing device is clicked, the user agent must 
run these steps" algorithm, which refers to the origin target, not the 
rerouted target.

I've now changed this so that it does in fact trigger the default action 
if applicable.


On Wed, 5 Mar 2014, Robert O'Callahan wrote:
> 
> The problem is that if these retargeted events can trigger default 
> browser behavior, the browser has to be able to compute the position of 
> the event relative to the new target DOM node, and it's not clear how to 
> do that.

I've made it explicit that the elements that can get clicks targetted to 
them only include elements that don't have subregions. In particular, 
image maps and image buttons are excluded.


On Tue, 4 Mar 2014, Rik Cabanier wrote:
> On Tue, Mar 4, 2014 at 8:30 PM, Ian Hickson <ian at hixie.ch> wrote:
> > On Tue, 4 Mar 2014, Rik Cabanier wrote:
> > > >
> > > > So what would you do in the case where you start two touches on 
> > > > different regions, then move them at the same time to two other 
> > > > different regions at the same time? What would you put in the 
> > > > touchmove event's object?
>
> The touches attribute [1] of the touch event would contain 2 touch 
> elements.
>
> Each touch element would have as target the canvas element and contain 
> the id of the hit region.

Oh so it's not the TouchEvent object you think should be adjusted, but the 
Touch object itself? (I'm assuming that's what you are referring to when 
you say "touch element".)

Presumably we would just set the region at creation time, like the 
"target" attribute, right?

I've specced this.


On Mon, 10 Mar 2014, Rik Cabanier wrote:
>
> Currently, the specification states that if you create a region and then 
> create another region that completely covers region, the first region is 
> removed from the hit region list [1]
> 
> This is a complex operation that involves either drawing the regions to a
> bitmaps and counting pixels, or path intersection.

There's two trivial ways to implement this, depending on whether the hit 
regions are backed by a bitmap (the simplest and fastest solution but uses 
a lot of memory) or a region list (slower, but much more memory 
efficient). In the case of a bitmap, you just draw on the new region, and 
the old region is no longer in the bitmap, so it's trivially gone. In the 
case of a list, you put the new region ahead of the old region so that you 
never actually get around to checking the old region.

In the latter case you would want to run "compression" algorithms every 
now and then to filter out old regions that no longer matter (though in 
most cases that would probably not be an issue since I expect most people 
will clear the whole canvas every frame, which can be trivially detected 
and can just cause the whole list to clear).


> It is also unintuitive because an author might expect that he could 
> remove the second region and still have the first region active.

That would be inconsistent with how canvas works. Canvas is an immediate- 
mode API. If you draw on the canvas, and then clear it, you don't get back 
what was there before.


On Tue, 11 Mar 2014, Rik Cabanier wrote:
> >
> > http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-July/thread.html#36556
>
> It looks like that thread never came to a conclusion.

Is there anything specifically that was raised in that thread that you 
think hasn't been responded to?


> The arguments against using a bitmap presentation still stand:
> - it will be too expensive to keep an actual copy of the canvas pixels 
> in memory to do hit testing

It's actually pretty common to do exactly this. Note that you don't 
necessarily need a bitmap that has the same bit depth or pixel density as 
the visible bitmap.


> - if you have to mimic the behavior with paths, you need access to
> expensive/difficult path manipulation algorithms

The maths for determining if a region is contained in another region is 
pretty well understood at this point, as far as I can tell.


> Hit regions should be redesigned so they work on the path geometry as 
> opposed to pixels. We already have the necessary code to do hit testing 
> on paths (see isPointInPath)

isPointInPath() works on pixels just like hit regions works on pixels.


On Wed, 12 Mar 2014, Dirk Schulze wrote:
> 
> In SVG we tried to avoid having hit testing based on pixel values 
> obviously for performance reasons.

SVG is a retained-mode API, so naturally it has a different model.

The whole point of <canvas> is to be an immediate-mode API that 
complements SVG.


On Mon, 10 Mar 2014, Rik Cabanier wrote:
>
> clearRect is currently defined as a subtraction mechanism for hit 
> regions [1].
>
> It's unlikely that a UA will implement hit regions using pixels so this 
> would have to be done using path subtraction which is expensive.

I'm not sure why you think it's expensive. It's trivial to just add a 
rectangle to the front of the list of regions.


> Why was this special behavior added to clearRect?

Because it's the most obvious mechanism for authors. You clear a part of 
the canvas, naturally that part of the canvas no longer has regions.


On Tue, 4 Mar 2014, Rik Cabanier wrote:
> 
> The spec implies--

The spec doesn't imply anything. It either requires something, or doesn't. 
If you ever find yourself reading between the lines, then there is either 
a spec bug, or you are reading something that the spec doesn't require.


On Fri, 21 Feb 2014, Charles McCathie Nevile wrote:
> On Thu, 20 Feb 2014 22:38:18 +0100, Nils Dagsson Moskopp wrote:
> > Dominic Mazzoni writes:
> > > 
> > > I'm happy to keep chasing after "legitimate" use-cases instead of 
> > > contrived ones, but just because we can't think of one, doesn't mean 
> > > it doesn't exist... Maybe the vast majority of web apps that use 
> > > canvas for a grid, or a slider, or a list box would be better off 
> > > using standard html5 objects. But what if there's one app that 
> > > can't, for some reason we haven't anticipated? If we wait until that 
> > > app appears to allow that control to have a hit region, then it will 
> > > be months or years before that app can be accessible.
> 
> Which is why I think the "but it shouldn't be done that way" argument 
> needs to be viewed through the prism of the question "but will it"?
> 
> History strongly suggests that people will use canvas because it suits 
> them, whatever "experts" like us say they *should* do.

People often misuse tools, but the correct reaction is to try to reduce 
the affordances that encourage such misuse, and emphasise the affordances 
that encourage correct use. This is exactly what we're doing here.

We certainly shouldn't take APIs and make misuse easier. That makes no 
design sense.


> > > > Well for example, is the use case one of the controls on 
> > > > Bugzilla's advanced search page?:
> > > > 
> > > >   https://www.w3.org/Bugs/Public/query.cgi?format=advanced
> > > > 
> > > > If so, I feel comfortable saying that we don't need to make 
> > > > <canvas> support that, since that use case is already handled very 
> > > > well by the <select> element.
> 
> It probably is for bugzilla.

Right. That's why it's important to know what the concrete use cases are.


> somebody wants a dropdown menu for selecting things, that matches the 
> particularities of their site design.

That use case sounds like something for CSS and Web Components to handle.


> the site is being built by someone who wants to ensure that their CSS 
> can't be overridden; I don't know why people think this, but many still 
> do.

Are you arguing that we should support such authors?

I am very comfortable saying we have no need to support authors who are 
trying to trump users' abilities to control their browsers.


> It apparently isn't for search suggestions. Whatever the reason, Yandex 
> search doesn't use it for their suggestions. While they don't use canvas 
> either, it seems that select is not sufficient for Yandex. Nor is it 
> used by Google, Yahoo, Bing or Baidu for the task. And none of them have 
> a complicated layout to work with. But they all use styled text 
> elements.

I'm not sure what you're arguing here. (What is "it"?)


> > > I'll try, though: what if I had a list of choices displayed as a pie 
> > > chart? Each slice of the pie is a focusable object that, when you 
> > > click on it, allows you to take an action on that pie slice.
> > 
> > What if I look at that page in a textual user agent?
> 
> Where the hit regions are based on a list as 'fallback content'? You get 
> a list. With links that match the actions. If the developer got 
> accessibility right.
>
> But if they would only get it a bit right, it seems stupid to suggest 
> that we should gratuitously deny them the option - letting perfect be 
> the enemy of good here is an extremely effective way to reduce 
> accessibility, which is very rarely done perfectly.

This particular use case is handled by the spec perfectly well already, as 
far as I can tell.


> http://yandex.com/yandsearch?text=pie%20chart%20canvas suggests that a 
> lot of people appear to think it is a natural thing to do. I got ~310k 
> answers, and while not all of them will be about using canvas to make 
> pie charts the overwhelming majority of the first few pages are.

This line of argument doesn't seem useful. "tables for layout", which I 
hope we agree is not something we should be encouraging, gives 12 million 
hits. "text editor canvas", the poster-child for misguided use of canvas, 
gives 2 million.

"http server canvas" gives 1 million.

"going to the gym canvas" also gives 1 million.

"fixing golden gate bridge canvas" gives half a million.

"pie chart canvas with clickable slices" gives 5 thousand.

Should we be adding APIs to make it possible to use <canvas> as an HTTP 
server? It certainly seems to have more hits than making a pie chart have 
clickable pie slices...

(That these numbers differ wildly from those on Google or Bing also 
suggests that this line of reasoning is flawed.)


> > May I ask if you have designed human-computer interfaces. If so, may 
> > you provide examples?
> 
> I'm not sure how that is relevant.

It seems at least as relevant as the Yandex numbers...


> > > > > What if I do want a <select>, but I just want a canvas to render 
> > > > > it visually?
> > > > 
> > > > Are Web Components and CSS unable to get the effects you need? 
> > > > Maybe we should be improving those rather than <canvas>. It's hard 
> > > > to tell without knowing precisely what you want to do.
> > 
> > This requests reminds me of customers wanting “page intro in flash.”.
> 
> And the response reminds me of http://xkcd.com/1319/

I don't understand the relevance of automation to this discussion.


> > > > If a region has a car theft problem, you don't solve it by giving 
> > > > all the thieves the car keys. You solve it by improving the 
> > > > economy so that thieves have better things to do (like get an 
> > > > interesting job), and you solve the remainder by improving law 
> > > > enforcement. The same applies here. We solve it by providing 
> > > > better tools for custom text editing controls (e.g. better 
> > > > contenteditable APIs), and by making it non-conforming to abuse 
> > > > <canvas> for this purpose.
> 
> Only if that actually stops people using canvas for this purpose.

What would you propose would stop canvas misuse?

Canvas misuse is not a minor issue. It can have a serious impact on 
accessibility (both for AT users and other users). We should take our 
responsibility for discouraging such misuse quite seriously, IMHO.


> And while we might disagree about what they *should* do, in practice it 
> seems that some developers will use canvas for almost any kind of 
> rendering, probably including things we can't even imagine.

Certainly. But that seems orthogonal to the issue at hand.


On Fri, 21 Feb 2014, Nils Dagsson Moskopp wrote:
> 
> Maybe you misunderstood me. I consider advocating a specific solution 
> for a poorly-defined problem a sure way to accumulate technical debt.

That's a very good way of summarising why I try to avoid adding features 
without a good understanding of the use cases.



-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'