[whatwg] Spec-based canvas tests

[Philip Taylor]

I was wondering how accurately the <canvas> specification matches with
the current implementations, so I've done a bit of work on testing the
first parts of it (up to Transformations, i.e. the easy/unexciting
bits) - the results suggest there isn't great consistency yet, so
hopefully the tests are worthwhile.

http://canvex.lazyilluminati.com/tests/tests/ has the data. There's an
annotated copy of the spec, with inline links to the relevant tests
for each automatically-testable feature and property that is described
(unless I missed any, which is not unlikely). The tests are grouped in
some arbitrary hierarchy, and the page for each category runs all the
descendant tests to provide a quick overview of the results. There's
also a page with the results I recorded from Firefox 2 and 3, Opera
9.10 (the current 9.20 is identical), and IE with ExplorerCanvas.
(Unfortunately I don't have access to a suitable version of Safari.)

Some of the tests are just looking at limits and other situations
where I guessed problems might occur, so it's not a particularly
methodical approach, and in some of the cases I had to guess what
would be an acceptable result when the spec doesn't specify it (which
I assume is a problem with the spec, particularly when implementations
differ); but I'm attempting to cover at least the points where you
would expect certain behaviour after reading the specification.

The tests are all automated as much as possible (when the browser
supports the right features to check the output), and the test
generation is also largely automated, since I'm lazy and don't want to
do any avoidable work - the system seems to work well for me so far,
though I have no idea whether anyone else likes this format.

I'm currently thinking it would be good to expand this to the
subsequent parts of the canvas spec - I don't know how much time I'll
have for this, but I expect I'll do some more at some point in the
future, so any suggestions on what I'm doing wrong or how to do things
better would be useful!

-- 
Philip Taylor
excors at gmail.com