[whatwg] <canvas> feedback

[Ian Hickson]

On Mon, 7 Dec 2009, Gregg Tavares wrote:
>
> Has there been a proposal for allowing mouse events to go through a 
> canvas element where it is transparent to the element below?

On Mon, 7 Dec 2009, Jason Oster wrote:
>
> The pointer-events CSS property was recently added to Firefox for web 
> (HTML) content:  https://developer.mozilla.org/en/CSS/pointer-events
> 
> Although the current implementation only supports the values 'auto' and 
> 'none' for web content, it seems like some of the SVG-only values could 
> possibly be used for your example case.  Or at the very least, I expect 
> the property could be extended with additional values specifically for 
> similar use cases with web content.

On Mon, 7 Dec 2009, Jonas Sicking wrote:
> 
> Indeed. We've talked about using this CSS property as a way to let only 
> the non-transparent parts of an image receive events. Unfortunately if I 
> recall correctly none of the currently defined values of pointer-events 
> fit that behavior, but the intent would be to add one. In any case I 
> think this CSS property is the way to add the feature you want, and so 
> the discussion is probably best had on the www-style mailing list.

On Tue, 8 Dec 2009, Robert O'Callahan wrote:
> 
> www-svg would be better, since it's an SVG property.

I agree that pointer-events should be the way to fix this.


On Sat, 5 Dec 2009, Franz Buchinger wrote:
>
> Gears introduced the concept of an "offscreen canvas" that doesn't draw 
> anything in the browser window, but can be used to manipulate images in 
> a web worker.
> 
> I used this functionality to implement a "resize-before-upload" feature 
> in my photo gallery uploader. Now I'm trying to port my uploader to 
> HTML5 but there seems no way to delegate the scaling work to a HTML5 web 
> worker. Surely I could use the DOM canvas to scale down the photos in 
> the main "browser thread", but this means that the UI gets blocked 
> during this process.

The specific use case of resizing images before uploading them would also 
require some kind of image object in the worker, unless you had the main 
thread paint each image to a canvas and then grabbed the pixels and sent 
them over, which would be almost as expensive as resizing them in the main 
thread in the first place. To have an image in the worker, we'd need the 
DOM in the worker, and for now, that's not on the cards, as several 
implementations have non-thread-safe DOM implementations.

I imagine we'll handle this use case in due course, though.


On Sun, 6 Dec 2009, Oliver Hunt wrote:
> 
> That said I'm not sure whether we should really be trying to spec such a 
> feature until we've had more time to see how workers are used in the 
> wild.

Indeed.


On Fri, 11 Dec 2009, Jeremy Orlow wrote:
> 
> Resizing images was just one use.  I could easily imagine apps wanting 
> to generate more complex images on background threads without needing to 
> implement things like spline drawing, pattern filling, and text 
> themselves.

If there are specific use cases other than batch image resizing for 
upload, it would be good to have specific use cases put forward. We can 
only evaluate proposals based on the use cases they're intended to solve 
if we know the use cases.


On Mon, 21 Dec 2009, Gregg Tavares wrote:
>
> What is the intent of the getContext function on the <canvas> tag?
> 
> Should it be possible to get multiple simultaneous different contexts as 
> in?
> 
> var ctx2d = canvas.getContext("2d");
> var ctxText = canvas.getContext("fancy-text-api");
> var ctxFilter = canvas.getContext("image-filter-api");
> 
> ctx2d.drawImage(someImage, 0, 0);
> ctxText.drawText(0, 0, "hello world");
> ctxFilter.radialBlur(0.1);
> 
> ?
> 
> OR
> 
> is canvas only allowed 1 context at a time?

That's basically up to the contexts to define.


On Mon, 21 Dec 2009, Gregg Tavares wrote:
>
> Is disallowing other contexts when certain contexts, eg "webgl", okay or 
> is that really an incompatible extension of the canvas tag?
> 
> Can portable code be written if some browsers let me get both a "2d" 
> context and a "3d" context and others don't?

All the browsers should do the same thing, but what that thing is depends 
on the context's spec.

In the case of the 2d context, whenever you invoke methods on the context, 
the bitmap is changed. If another context clears the context, or does 
something more complicated, it has to define how it works with the 2d 
context.


On Thu, 7 Jan 2010, Jonas Sicking wrote:
>
> So at mozilla we've been implementing and playing around with various 
> File APIs lately. One of the most common use cases today for file 
> manipulation is photo uploaders. For example to sites like flickr, 
> facebook, twitpic and icanhascheezburger. Some of these sites allow 
> additional modifications of the uploaded image, most commonly cropping 
> and rotating the image. But even things like manipulating colors, adding 
> text, and red eye reduction are things that sites do, or would like to 
> do.
> 
> We do already have a great tool for image manipulation in HTML, the 
> canvas element. However if a site wants to upload the resulting image, 
> after the modifications, things become more painful. Currently you have 
> to call toDataURL(), send that URL using XMLHttpRequest, and then 
> serverside convert to binary data. Things will be a little better once 
> XHR supports sending manually constructed binary data, but you still 
> have to manually convert the data url to binary data in javascript.
> 
> I suggest we add a function like:
> 
> File toFile(in DOMString name, in optional DOMString type, in any...  args);
> 
> This function takes the same arguments as toDataURL(), plus an 
> additional 'name' argument. It produces the same result as toDataURL(), 
> except that it returns the result as a File object rather than as a 
> data-url encoded string. This can then be directly sent using 
> XMLHttpRequest.

On Thu, 7 Jan 2010, Maciej Stachowiak wrote:
> 
> I think it would make more sense to have an actual type for binary data 
> (for example along the lines of my proposal on public-script-coord and 
> es-discuss) and enable getting one from <canvas> and sending via XHR. I 
> don't think using File for temporary in-memory binary data, as opposed 
> to a file on disk, makes sense. The File should stick to representing 
> files. After all, you would not make a File object to convert something 
> to text for uploading. Nor would it make sense to do an asynchronous 
> read from this. And under the covers, you would want to pass the actual 
> binary data to the upload code, not a file with a file name.

On Thu, 7 Jan 2010, João Eiras wrote:
> 
> How about just overloading xhr.send() to handle a <canvas> element ?

On Thu, 7 Jan 2010, Jonas Sicking wrote:
> 
> I'm reluctant to overload the meaning of sending an Element object. When 
> a Document is passed to xhr.send() we already serialize that document 
> into markup, it seems likely to me that in the future we'll want to do 
> the same thing for Elements.

On Thu, 7 Jan 2010, Jonas Sicking wrote:
> 
> Additionally, this doesn't allow specifying the encoding type, such as 
> JPEG or PNG, or encoding parameters, such as JPEG quality.

On Fri, 8 Jan 2010, Shumpei Shiraishi wrote:
> 
> I think the method name "toFile()" is missleading, because I agree to 
> the Maciej's openion.
> 
> > I don't think using File for temporary in-memory binary data, as 
> > opposed to a file on disk, makes sense.
> 
> So, how about changing the method name to "toBlob()" and signature as 
> follows?
> 
> Blob toBlob(in optional DOMString type, in any... args);

I'm with Maciej on this in that the right solution here seems to be to 
actually support raw binary data. I'll wait a bit longer on this and see 
what happens with the binary support in JS. (WebSocket also needs 
something like this.)


On Wed, 3 Feb 2010, Tim Hutt wrote:
> 
> 1. You can only set the size exactly in pixels. It is very hard to get a 
> resizable canvas that fills the page. You *can* set the size in CSS, but 
> it doesn't work very well (e.g. using left,right-margin: auto; to centre 
> the canvas doesn't work. Also it scales the canvas rather than resizes 
> it.

The centering issue is a CSS issue (you probably have to fiddle with 
'display' or 'text-align'). The sizing issue is somewhat by design, 
though. The solution is to set the size manually when the page changes 
size, at the same time as doing the repaint.


> 2. There doesn't appear to be any kind of double buffering capability. 
> This is pretty critical for animations where you might need to clear the 
> canvas and then draw stuff. You currently get flickering if you try.

Just do it all in one go (not in several timeouts) and you should be fine.


On Wed, 3 Feb 2010, Simon Fraser wrote:
>
> Canvas drawing should have been callback-based from the start. Maybe we 
> can retrofit this by having a way the author can specify that the canvas 
> backing store resizes with the element, and firing an event when the 
> backing store size changes.

We might want to consider something like that in a future version, 
especially to help with getting animation done smoothly, similar to the 
video timeupdate event. It depends on how people end up using it.


On Wed, 3 Feb 2010, Tim Hutt wrote:
>
> So my revised suggestion is:
> 
> 1. Support more length specifiers for the width and height of a
> canvas(%, em, etc.).
> 2. Add an onresize event to the canvas tag. When the canvas is resized
> it is reset and this event is fired.
> 3. CSS size specifiers resize rather than scale the canvas. I.e. it
> should always be that 1 unit = 1 pixel initially.

You can do all this today, except that at the start of the onresize event 
(which fires on the Window, not the element) you reset the size of the 
coordinate space manually by setting the height/width attributes. I don't 
really see much of a gain to be had by changing this.


On Sat, 13 Feb 2010, Evgeny Burzak wrote:
> 
> as Ilmari pointed out in his letter some.. hmm.. time ago Canvas spec
> is missed method isPointInStroke(). It is _important_ thing as this
> will be used for generating dynamic charts and graphs. For the moment,
> selecting some point on a curve is not trivial problem...
> (Although my real intention was to make simple graphics editor for
> the web, but without isPointInStroke() it is impossible so far.)

I think when we add path primitives we should add a way of converting a 
path stroke to a path, as well as ways to grow a path or shrink a path, 
and other such features. Then the ability to test if a point is in a path 
would solve your problem as well as many others.


> In addition, i allow to use flash blend modes [2] in Canvas implementation 
> for IE [3], so could it be added in specs too?
> [2] http://livedocs.adobe.com/flash/9.0/ActionScriptLangRefV3/flash/display/BlendMode.html
> [3] http://burzak.com/pro/fxcanvas/tests/test_flash_blendmodes.html

What are the use cases?


> Also renamed globalCompositeOperation:
> * lighter -> add
> * darker -> subtract
> * source-over -> normal

I'd rather not rename these since they are now widely implemented.


On Mon, 15 Feb 2010, Stef Epardaud wrote:
> 
> In a related question, is there any guarantee that when we draw an image 
> with EXIF information into a canvas for resizing, we get (or not) the 
> EXIF back in the resized image (currently via toDataURL())?

You are guaranteed not to.


On Mon, 15 Feb 2010, Anne van Kesteren wrote:
>
> Seems easier to take care of this by allowing extraction of metadata 
> (some W3C WG is working on this I think) and maybe a way to put it back 
> in...

Indeed.


On Tue, 16 Feb 2010, Stef Epardaud wrote:
> 
> Now that I think about it, is it possible to replace a regular form's 
> file input list (the list of File objects selected from an input of type 
> "file" with multiple files enabled) with "processed" files? Like data 
> coming out of a canvas, possibly by storing them locally (via storage) 
> to get a File instance.

There is no way to add a File to <input type=file>, no.


> Or are we then limited to sending those processed files via 
> XmlHttpRequest?

Yup. That's not a huge burden though. :-)


On Tue, 16 Feb 2010, Stef Epardaud wrote:
> 
> The rationale is simple: a JS action can filter what the user types in a 
> <input type=text>, and still allow the <form> to be sent regularly 
> (non-AJAX). Why not the same for files? In my use-case I have two 
> different options, sending processed files using XMLHttpRequest, or with 
> a normal <form> submit. XMLHttpRequest gives me better progress 
> notification (how I wish we could get progress notifications for a 
> regular <form> when POSTing), but if the set of processed files is part 
> of a <form> with more input elements (in the case of photos, comments, 
> date, whatever), it is simpler to use the regular <form> submission, and 
> just filter the selected Files as the user adds them to the <input 
> type=file>, process them, then replace them in the <input type=file>. If 
> the user then changes his mind about the selected file list, hey it 
> still works.

I agree that that could make sense, but it's a very sensitive part of the 
spec (it deals with the user's filesystem), and for this reason I think we 
should be very conservative about what we do with it.


On Mon, 22 Feb 2010, David Levin wrote:
>
> I've talked with some other folks on WebKit (Maciej and Oliver) about 
> having a canvas that is available to workers. They suggested some nice 
> modifications to make it an offscreen canvas, which may be used in the 
> Document or in a Worker.

What are the use cases?


On Tue, 23 Feb 2010, Jeremy Orlow wrote:
> 
> I've gotten a couple responses back on use cases.  I'll admit that about 
> half a simply resize/rotate.  The others I've been told I cannot talk 
> about publicly.  I think my map tiles example is similar enough to a 
> bunch of them though that you can get at least the conceptual idea.  
> And my understanding is that in prototypes, it's been very hard to do 
> the (computationally complex) rendering without making the UI go janky.  
> There might be some more use cases that come up that I can generalize 
> into a public form, in which case I'll do my best.

Feel free to send me specific use cases privately if you prefer, though we 
will have to describe them publicly if we're to add features to a 
vendor-neutral standard to support them.


On Wed, 3 Mar 2010, FrantiÅ¡ek Å~XezáÄ~M wrote:
> 
> Description
> add overload of (or add similarly called) method "createImageData" to
> interface CanvasRenderingContext2D which would take two arguments:
> - encodedImageBinaryData
> - dataMimeType
> which are rather self explanatory.
> 
> Reason
> The reason is to be able to supply output of the future File API
> standard (http://www.w3.org/TR/FileAPI/) into canvas.
> 
> Use case
> Usually there is a lot of image processing during image upload. This
> processing takes place at the server and it can consume a lot of
> system resources. The method I propose would enable basic client side
> image processing by loading image to canvas tag. Then the modified
> image could be sent to server instead of the original one. This would
> save a lot of server's resources and sometimes even a lot of
> bandwidth.
> 
> Another possibilities
> There are already solutions for this use case, but they are based on
> technologies like Flash or Silverlight. With this proposal, no plugin
> would be needed.

You don't need all that -- just create an Image object and then sets its 
src="" to the File.URL attribute.

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'