[whatwg] Cross browser automated tests
sylvain.pasche at gmail.com
Tue May 20 17:11:55 PDT 2008
I'm working on an experimental project to build a set of cross browser
automated tests. The idea would be to have a repository of browser
independent automated tests. Existing test suites could be imported into it.
These tests could be run automatically on today's available browsers
with the results published. Spec writers or web developers could check
what feature is implemented on what browser and browser vendor could use
that for their internal regression tests.
Would WHATWG be interested in such a project?
As a starting point, the testing environment and API would need to be
defined. I'm thinking of the following requirements:
* Possible for browser vendors to integrate these tests in their testing
* Minimal modifications necessary for migrating existing non automated
test cases like the ones on http://www.hixie.ch/tests or
http://tc.labs.opera.com/. They should still report PASS/FAIL messages
when run in a browser like they do now, which is convenient for testing
test cases individually.
To meet these goals, such an API should be as unobtrusive as possible:
* One .js file to include
* A set of functions that can be called for performing the checks and
life cycle functions (end of tests, ...)
Once this is defined, existing non automated test cases could be
modified and imported in the repository. That could be lot of work if
I'm looking at all the tests linked from
http://wiki.whatwg.org/wiki/Test_cases, but I think there would be lots
The other big part of the work is to develop the framework for running
these tests automatically and reporting the results. I've already done
some experiments by running automated tests from Mozilla and WebKit. The
results are visible on http://www.browsertests.org/.
I started a thread on Mozilla  and WebKit lists . There were some
interesting discussions with Jeff Walden on the Mozilla list already.
More information about the whatwg