[whatwg] Canvas feedback (various threads)

[Ian Hickson]

On Thu, 3 Feb 2011, Boris Zbarsky wrote:
>
> It looks like CSS rgba colors with an alpha value of 0 are serialized as 
> rgba() with "0" as the alpha value. in at least Gecko, Webkit, and 
> Presto.
> 
> It also looks like canvas style color with an alpha value of 0 are 
> serialized as rgba() with "0.0" as the alpha value in Gecko 3.6, Webkit, 
> and Presto.
> 
> In Gecko 2.0 we made a change to the canvas code to fix some bugs in the 
> serialization by the simple expedient of reusing the well-tested code 
> that CSS colors use.  This has the incidental benefit of more behavior 
> consistency for authors.  Unfortunately, this makes us not do what these 
> other UAs do, nor do what the current spec draft says.
> 
> While we can clearly special-case 0 in this code, it seems like an 
> authoring pitfall to have the two different serialization styles, so I 
> would prefer to have consistent behavior between the two.  What do other 
> UA vendors think of standardizing on "0" as the serialization for canvas 
> color alpha channels when they're transparent?  Alternately, what about 
> using "0.0" for CSS colors?

On Fri, 4 Feb 2011, Anne van Kesteren wrote:
> 
> Either way is fine. Note however that when there is no alpha-channel 
> involved the differences are much greater between CSS and <canvas>. At 
> least in Gecko/WebKit. I.e. rgb() vs #rrggbb. That probably cannot be 
> changed though.

Given Anne's point (that this is already different in other contexts), and 
given the current deployed interop on this issue, I'm reluctant to change 
this. I agree that it's unfortunate that we needlessly differ from CSS 
here. It's another example of why it's important for us to define 
everything, and that we never leave anything up to UAs to decide. :-)


On Mon, 17 Jan 2011, carol.szabo at nokia.com wrote:
>
> I propose changing the drawing model steps 2 to 5 the following way:
> 
> Step 2: Multiply the alpha channel of every pixel in A by globalAlpha. 
> (Prior Step 5)
>
> Step 3: When shadows are drawn, render the shadow from image A, using 
> the current shadow styles, creating image B.
>
> Step 4: When shadows are drawn, composite image B onto image A using 
> destination-over compositing operation.
> 
> This algorithm is less expensive then the prior one (it saves one 
> multiplication of the AlphaChannel over the entire B bitmap), and treats 
> the image/object drawn and its shadow as one entity. Which results in 
> the shadow being preserved for composite operations such as copy and 
> source-in and produces less strange results in operations such as xor 
> and destination-out when shadows are drawn.
> 
> The this algorithm yields the same result as the current version of the 
> spec for source-over, but for completeness supports shadows in all 
> modes. Indeed the way the current spec exists, many non-source-over 
> modes including xor yield very strange results if shadows are used. I do 
> not care that much if the spec has my proposal in it or whether it specs 
> that shadows are not rendered in modes other then source over, but it 
> would be nice to hear an agreement from browser implementors on this.

On Tue, 18 Jan 2011, Robert O'Callahan wrote:
> 
> [...] if we don't have good use cases for using shadows with 
> non-"source-over" operators (I don't), let's just say that shadows don't 
> draw for non-"source-over" operators. That would reduce spec and 
> implementation complexity.

I'm happy to do either of these, but I'm very relunctant to change the 
spec away from what browsers do. Currently, browsers pretty much agree on 
how shadows work in composition, they just disagree over what gets 
composited (and even then it's only really WebKit that disagrees).

If there is interest in changing this, I think the best thing would be for 
browser vendors to indicate a commitment to change it, so that I can make 
sure I'm not changing the spec away from what browsers want to implement.


On Tue, 23 Nov 2010, Tab Atkins Jr. wrote:
>
> Right now, canvas gradients interpolate their colors in 
> non-premultiplied space; that is, the raw values of r, g, b, and a are 
> interpolated independently.  This has the unfortunate effect that colors 
> darken as they transition to transparent, as "transparent" is defined as 
> "rgba(0,0,0,0)", a transparent black.  Under this scheme, the color 
> halfway between "yellow" and "transparent" is "rgba(127,127,0,.5)", a 
> partially-transparent dark yellow, rather than "rgba(255,255,0,.5)".*
> 
> The rest of the platform has switched to using premultiplied colors for 
> interpolation, because they react better in cases like this**. CSS 
> transitions and CSS gradients now explicitly use premultiplied colors, 
> and SVG ends up interpolating similarly (they don't quite have the same 
> problem - they track opacity separate from color, so transitioning from 
> "color:yellow;opacity:1" to "color:yellow;opacity:0" gives you 
> "color:yellow;opacity:.5" in the middle, which is the moral equivalent 
> of "rgba(255,255,0,.5)").
> 
> It would be unfortunate for canvas gradients to be the only part of the 
> platform that interpolates colors in a different and generally uglier 
> way here.  I suspect that this can be changed without any real compat 
> impact; the change will be a minor visual tweak to the middle half of 
> gradients that go from a solid color to transparent (the initial and 
> final quarter won't be noticeably different, solid->solid transitions 
> aren't affected, and other transitions trace a shorter line and thus 
> diverge from the "pretty" behavior less).
> 
> I realize that this basically just depends on whether browsers are 
> willing to change their current implementation.  Implementors, does this 
> sounds like a change you can get behind?  We already changed canvas 
> shadows to match behavior with CSS shadows; this is a much smaller 
> change for spec-equivalence.
> 
> * An even more obvious example can be seen by drawing a 
> white->transparent gradient in a white rect on canvas.  This should 
> pretty obviously just result in more white, right?  Instead, you get a 
> symmetrical fade from white to gray and back to white.  (The reason it 
> reverses halfway through is because it's now less than half opacity, so 
> the background white is winning when it's composed with the gradient 
> color.  The gradient is actually still getting darker until it hits 
> black at the end, it just contributes less and less of that darkness to 
> the final color of the pixels.)
> 
> ** Quick primer for those who don't understand color math well enough to 
> follow the conversation (like me from a few days ago):
> 
> "Premultiplied" means that you multiply the color components by the 
> alpha component before doing compositing operations.  It turns out that 
> this lets you get away with some simpler and faster math when 
> compositing partially-transparent colors while getting equivalent 
> results, so it's a common optimization.
> 
> The effect of this is that the space of colors gets "compressed" as the 
> alpha shrinks.  If the alpha is .5, then the total value range for the 
> color components is only 0-127.  If the alpha is .1, the range is 0-25.  
> This doesn't produce a material change in behavior, due to the way the 
> compositing math works.  The only downside is that if you lose precision 
> in the exact color, if you want to extract that back out from the 
> premultiplied version - there are only 26^3 possible colors at alpha=.1, 
> as opposed to the 255^3 colors at alpha=1.
> 
> A nice benefit of this, though, is that tracing a straight line between 
> two colors in premultiplied space gives you a more attractive 
> transitions than doing so in non-premultiplied (that is, normal rgba) 
> space.  As I noted above, the color halfway between yellow and 
> transparent (defining "transparent" as transparent black, or 
> rgba(0,0,0,0)) is rgba(127,127,0,.5) in non-premultiplied space, which 
> has a dark yellow as its color component.  In premultiplied space, the 
> midpoint is the 4-tuple (127,127,0,.5), which translates to the color 
> rgba(255,255,0,.5) when you extract it back into normal rgba space.
> 
> Of course, in normal rgba space you could fix this by explicitly 
> transitioning from yellow to transparent yellow (in other words, from 
> rgba(255,255,0,1) to rgba(255,255,0,0)), but that's more work.  In 
> premultiplied space, all fully-transparent colors are equivalent, and 
> naive transitions always "do the right thing".

On Tue, 23 Nov 2010, Philip Taylor wrote:
> 
> If you define the gradient as interpolating from solid yellow to 
> transparent black, I'd expect that it *should* be semi-transparent 
> blackish-yellow in the middle.
> 
> If you want it to be pure yellow, don't use a keyword which is 
> explicitly specified as transparent black - define the gradient from 
> rgba(255,255,0,1) to rgba(255,255,0,0) instead. Then you'll get 
> rgba(255,255,0,0.5) in the middle.

On Tue, 23 Nov 2010, L. David Baron wrote:
> 
> Sure, you can solve that particular case.  However, if neither of
> the endpoints is precisely transparent, and you're changing both
> color and alpha components, you'll still get an ugly effect with the
> current rules, and one that you can't work around.  (I experimented
> with both methods when implementing CSS transitions of colors, and
> went with premultiplied.
> http://dbaron.org/css/test/2009/transitions/transitions-alpha makes
> it look like WebKit has now switched to premultiplied.)

Looking at that demo, it seems that premultiplied removes possible options 
from the author. How do you go from red to actually transparent black, 
getting darker as you transition? Do you have to give a nearly-transparent 
black (alpha=0.01) to get around it or some such? That seems weird.


> It's not only the 'transparent' keyword; it affects all cases of 
> gradients between colors with different alpha values and different color 
> values.  And in cases where one of the endpoint alphas is not 0, it's 
> not possible to get the correct (premultiplied) result with a gradient 
> computed in nonpremultiplied space.

Can you elaborate on that? I'm interested in seeing the cases that you 
can't do in one or the other of the colour spaces we're discussing. If one 
is a strict superset of the other, then it would make sense to specify 
that we use that one. If you can do the same gradients in both, then 
interoperability seems more important.


On Tue, 23 Nov 2010, Jonas Sicking wrote:
> 
> For what it's worth, I suspect most people don't think of "transparent" 
> as "transparent black", but rather as "fully see-through".

On Tue, 23 Nov 2010, Tab Atkins Jr. wrote:
> 
> The fact that "transparent" means "transparent black" is a technical 
> detail that is usually irrelevant.  A fully transparent color doesn't 
> really have a "color" at all.  I'm highly technical, involved in the 
> relevant specs, and understand the underlying math, and I *still* get 
> tripped up over the fact that "transparent" is black.

On Thu, 25 Nov 2010, Anne van Kesteren wrote:
> 
> The people at Opera responsible for the graphics layer agree with 
> Philip's point of view.

On Fri, 26 Nov 2010, Simon Fraser wrote:
> 
> This would be hard for WebKit, which relies on Core Graphics for 
> gradients on some platforms. CG doesn't allow us to interpolate in 
> premultiplied space.
> 
> [Of course, some of the new radial gradient stuff is also not doable 
> using CG, so maybe we'll have to have a slower non-CG gradient code path 
> anyway].

On Fri, 26 Nov 2010, Boris Zbarsky wrote:
> 
> But CSS gradients are already requiring interpolation in premutiplied 
> space, right?

On Fri, 26 Nov 2010, Simon Fraser wrote:
> 
> I think you're thinking of CSS Transitions, which we decided should run 
> in premultiplied.

On Fri, 26 Nov 2010, Boris Zbarsky wrote:
> 
> No.  http://dev.w3.org/csswg/css3-images/#color-stop-syntax currently 
> says:
> 
>   Between two color-stops, the line's color is linearly interpolated
>   between the colors of the two color-stops, with the interpolation
>   taking place in premultiplied RGBA space.
> 
> and there was related discussion on www-style, iirc.

On Sun, 28 Nov 2010, Tab Atkins Jr. wrote:
> 
> Note, though, that that's a relatively recent change that I made based 
> on dbaron's feedback.  That said, in the thread where I made the change, 
> Simon expressed that he'd like Transitions and Gradients to work the 
> same way, and I read him as preferring premultiplied.

The canvas gradient spec is pretty uniformly and interoperably implemented 
on this front:

   http://software.hixie.ch/utilities/js/canvas/?c.clearRect(0%2C%200%2C%20640%2C%20480)%3B%0Ac.save()%3B%0Avar%20gradient%20%3D%20c.createLinearGradient(0%2C0%2C640%2C480)%3B%0Agradient.addColorStop(0%2C%20'rgba(255%2C255%2C255%2C1)')%3B%0Agradient.addColorStop(1%2C%20'rgba(0%2C0%2C0%2C0)')%3B%0Ac.fillStyle%20%3D%20gradient%3B%0Ac.fillRect(0%2C0%2C640%2C480)%3B%0Ac.restore()%3B%0A

It's easy to work around this issue:

   http://software.hixie.ch/utilities/js/canvas/?c.clearRect(0%2C%200%2C%20640%2C%20480)%3B%0Ac.save()%3B%0Avar%20gradient%20%3D%20c.createLinearGradient(0%2C0%2C640%2C480)%3B%0Agradient.addColorStop(0%2C%20'rgba(255%2C255%2C255%2C1)')%3B%0Agradient.addColorStop(1%2C%20'rgba(255%2C255%2C255%2C0)')%3B%0Ac.fillStyle%20%3D%20gradient%3B%0Ac.fillRect(0%2C0%2C640%2C480)%3B%0Ac.restore()%3B%0A

I'm happy to change the spec on this, but I'm not going to change it ahead 
of the implementations. If you want this changed, I recommend getting the 
browser vendors to change this.


On Sat, 13 Nov 2010, Evgeny Burzak wrote:
> 
> I faced some troubles when using default image data, because of in old 
> Explorers used extremely ineffective memory manager, so that I made some 
> tests and figure out that fastest and less memory consumption is array 
> with pixel colors encoded in 32-bit integer. Here is the test: 
> http://burzak.com/proj/fxcanvas/tests/data-structure-comparison.html
> 
> But, the test results looks similar for all vendors! On average the 
> difference in processing speed is about 2-3 times (five times in Firefox 
> 4). This is due to the fact that loops take less time (width x height * 
> 4 vs. width x height) and arrays with less elements take less memory.  
> Though I realize that main idea for data structure was simplicity, but 
> in this case it seems simplicity is evil, not good.

The data storage for ImageData need not be much bigger, since the browser 
can use whatever storage mechanism it needs, including structured arrays 
of 32bit ints or indeed simply a single flat bitmap of the data. I expect 
this is an area browsers will compete in to optimise JS execution and 
memory usage over time.


> Another thing is reusable canvas path.  I've added experimental class
> CanvasPath and and some related methods to Canvas context.
> For example:
> var p = new CanvasPath() // I think first argument can be string from
> SVG "d" attribute
> p.moveTo(0, 0)
> p.lineTo(10, 20)
> ctx.beginPath()
> ctx.appendPath(p)
> ... and so on. If you are interested in it, I can describe my idea futher...
> This is quite useful thing, at least for me.

Yeah, this is actually already on the list of things to add to the next 
version of canvas. We're waiting for browsers to catch up with 
implementing a lot of the other features in HTML first.


On Sat, 13 Nov 2010, Boris Zbarsky wrote:
> 
> At least Gecko and Webkit implement canvas imagedata as something more 
> like a WebGL typed array, not a JS array.  So in particular, the memory 
> usage is the same or better as for your "compact" array version (in the 
> case of Gecko actually 2x better, since your "compact" array uses 8 
> bytes per color, while the native imagedata uses 4 bytes per color). Yes 
> that gives a significant speedup due to less memory traffic and better 
> cache locality.

Indeed.


On Sun, 14 Nov 2010, Evgeny Burzak wrote:
>
> Yes, this is local problem for canvas emulator and IE.

Hopefully this will become moot with IE9, which apparently implements 
<canvas> natively.


> So which format should I use? A lot of people using WinXP and IE 6-8 at 
> this moment. There are no typed arrays, no fast JS and so on... :( But 
> they may be our clients and don't worry about that. If they see message 
> "Script takes too much time", they say goodbye, it's not for me. This 
> situation will be (probably) five years long or more. So while is it 
> possible to add conversion methods to the typed array? Something like 
> fromArray32() and fromArray8() ?

Unfortunately pretty much nothing we can do here will get implemented and 
distributed in less than 5 years, so it can't help you. As a general rule, 
you can't work around lack of implementations of specs by adding features 
to specs.


On Mon, 15 Nov 2010, Charles Pritchard wrote:
> On Sun, 3 Oct 2010, Charles Pritchard wrote:
> > > Having worked quite awhile with WebApps APIs, Canvas included, I've 
> > > concluded that HTML can be implemented within the web stack. It's my 
> > > firm belief that the Web Apps specifications can and should be 
> > > proven complete.
> >
> > If by "complete" you mean "self-hosting", then: why? That seems like a 
> > very arbitrary goal.
> > 
> > If not, what do you mean?
>
> I intended that section 4, "The elements of HTML", with minor 
> exceptions, should be displayable with Canvas, and operable with a 
> minimal canvas+webapps profile.

But why?


> > > I'm concerned that the issue is being avoided because it originated 
> > > from a project you disagree with; and has biased your judgment of 
> > > additional use cases or possible remedies.
> >
> > Good lord no. This is merely a prioritisation issue. I'm sure we'll 
> > add lots of metrics over time.
>
> But I'm not requesting a discussion on various aspects of fonts.
>
> I'm pointing to the non-availability of one particular metric, as a 
> blocking issue in my ability to keep a string of text on the same 
> baseline.

I understand. I agree that we should support this in due course. As I said 
before:

> > The problem is that anytime we add anything to canvas, implementors 
> > get so excited that they drop what they're doing and implement it (in 
> > some cases, overnight!). This takes resources away from other 
> > features. If we're to get the whole platform to improve, we need to 
> > make sure that everything gets a chance to be implemented. This means 
> > we can't just be adding stuff to canvas all the time.
>
> You're making a slippery slope argument, I don't think it fits: I'd 
> asked for one property to be added to the specs document; as an 
> extension of the textBaseline attribute already in the document.

Lots of people are asking for just one property. Why is yours a higher 
priority than theirs?


> Currently, I can't use fillText with two separate font sizes, and 
> underline them.

Sure you can:

   http://software.hixie.ch/utilities/js/canvas/?c.clearRect(0%2C%200%2C%20640%2C%20480)%3B%0Ac.save()%3B%0Avar%20x%20%3D%2020%3B%0Avar%20y%20%3D%20200%3B%0Ac.textBaseline%20%3D%20'baseline'%3B%0Ac.font%20%3D%20'80px%20Verdana'%3B%0Ac.fillText('Hello'%2C%20x%2C%20y)%3B%0Avar%20dx%20%3D%20c.measureText('Hello%20').width%3B%0Ac.font%20%3D%20'40px%20Zapfino'%3B%0Ac.fillText('World'%2C%20x%2Bdx%2C%20y)%3B%0Adx%20%2B%3D%20c.measureText('World').width%3B%0Ac.fillRect(x%2C%20y%2B5%2C%20x%2Bdx%2C%202)%0Ac.restore()%3B%0A

But I agree that there are other things you can't do, like put a 
background colour behind the text that covers all the text (Zapfino in 
particular goes way outside its em box, for instance). It's something that 
we will address, we're just waiting for browser vendors to have 
implemented more of the current spec before adding more features. It's a 
prioritisation issue, that's all.


> > > Nobody wants to see another vendor-specific extension; can we try to 
> > > form an agreement on this, so we can avoid that?
> >
> > On the contrary, we _do_ want to see vendor-specific extensions. 
> > That's how we get implementation experience and how the Web improves.
> > 
> > Standardisation is the penultimate stage in a feature's development, 
> > after studying author practices, experimental implementations in 
> > vendor-specific features, and studying use cases, and ahead only of 
> > the final convergence of browser implementations
> 
> Your enthusiasm for the "final convergence" is charming.
> 
> I made a poor generalization. Most of us do not want to see vendor-specific
> extensions which stick.
> Example:
> moz..transform = webkit..transform = transform = ...;

Sure, extensions should be a short-term thing.


On Sat, 18 Dec 2010, Glenn Maynard wrote:
> 
> It does make a lot of sense to do asynchronous reads from file data 
> generated from a canvas.
> 
> For a larger canvas, generating a PNG can take some time.  By making a 
> copy (or COW) of the canvas, reading PNG data from it can be done 
> asynchronously.  This would keep the whole page from halting for half a 
> second (or more; PNG can be expensive at higher compression levels) 
> while the data is generated.
> 
> For example, this would be very important if an image editor is backing 
> up the user's work in the background, as Gmail does with message text.  
> If the user is drawing on the canvas, having the UI hitch for 500ms (or 
> even 100ms) would be disruptive.
> 
> The hard part is that you can't have a Blob whose size isn't known in 
> advance, so this would probably need a separate class, making it a much 
> larger undertaking.  (The same would be needed for things like 
> deflating/inflating data: can't slice it and don't know the final size.)  
> But, the async reads and compression themselves do make a lot of sense.

I've noted in the spec that we should have an asynchronous canvas.toBlob() 
in a future version. I haven't added it yet because -- at the risk of 
sounding like a broken record -- we're waiting for browsers to catch up 
with implementing a lot of the other features in HTML first.

I agree that it would make a lot of sense, though.


On Wed, 29 Dec 2010, Boris Zbarsky wrote:
> 
> Assuming the new text is in the "CSS Modules" section, there's a minor 
> typo: the text "as it closing the open construct" should not have "it" 
> there.

Fixed, thanks.


On Fri, 7 Jan 2011, Charles Pritchard wrote:
>
> Let me know if this has been discussed before:
> 
> Loading an html page containing:
> <canvas><img src="fallback.jpg" /></canvas>
> loads the fallback.jpg image, even when canvas is supported.
> 
> Is this intentional, or simply the easiest route for the moment?

Intentional. Images for <img> elements are always loaded, even if they're 
not in the document at all (e.g. just in script), because people 
historically used them for "image overs".



(Over the past couple of months there have been a number of e-mails and 
threads about CSS issues and asking questions of implementators but not 
suggesting changes to the specs I edit. I have not responded to those 
e-mails. If you sent an e-mail on <canvas> to the WHATWG list before this 
week and haven't received a reply from me but wanted one, let me know.)

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'