[whatwg] Cross browser automated tests

[Sylvain Pasche]

Hi,

I'm working on an experimental project to build a set of cross browser
automated tests. The idea would be to have a repository of browser
independent automated tests. Existing test suites could be imported into it.

These tests could be run automatically on today's available browsers 
with the results published. Spec writers or web developers could check 
what feature is implemented on what browser and browser vendor could use 
that for their internal regression tests.

Would WHATWG be interested in such a project?

As a starting point, the testing environment and API would need to be 
defined. I'm thinking of the following requirements:
* Possible for browser vendors to integrate these tests in their testing 
frameworks
* Minimal modifications necessary for migrating existing non automated 
test cases like the ones on http://www.hixie.ch/tests or 
http://tc.labs.opera.com/. They should still report PASS/FAIL messages 
when run in a browser like they do now, which is convenient for testing 
test cases individually.

To meet these goals, such an API should be as unobtrusive as possible:
* One .js file to include
* A set of functions that can be called for performing the checks and 
life cycle functions (end of tests, ...)

Once this is defined, existing non automated test cases could be 
modified and imported in the repository. That could be lot of work if 
I'm looking at all the tests linked from 
http://wiki.whatwg.org/wiki/Test_cases, but I think there would be lots 
of benefit.

The other big part of the work is to develop the framework for running 
these tests automatically and reporting the results. I've already done 
some experiments by running automated tests from Mozilla and WebKit. The 
results are visible on http://www.browsertests.org/.

I started a thread on Mozilla [1] and WebKit lists [2]. There were some 
interesting discussions with Jeff Walden on the Mozilla list already.


Sylvain

[1] 
http://groups.google.com/group/mozilla.dev.quality/browse_thread/thread/b2a959c7547b9877
[2] https://lists.webkit.org/pipermail/webkit-dev/2008-May/003943.html