[whatwg] Cross browser automated tests

[Maciej Stachowiak]

On May 20, 2008, at 7:51 PM, Ian Hickson wrote:

> On Wed, 21 May 2008, Sylvain Pasche wrote:
>>
>> I'm working on an experimental project to build a set of cross  
>> browser
>> automated tests. The idea would be to have a repository of browser
>> independent automated tests. Existing test suites could be imported  
>> into
>> it.
>>
>> Would WHATWG be interested in such a project?
>
> I think that would be great; if you do go ahead with this, you are  
> welcome
> to use the WHATWG wiki and other resources. I encourage you to use the
> implementors mailing list for this:
>
>   http://www.whatwg.org/mailing-list#imps
>
>
>> To meet these goals, such an API should be as unobtrusive as  
>> possible:
>> * One .js file to include
>
> I recommend not embedding any JS, but instead requiring that the  
> following
> two lines be used to report results (or something like them):
>
>   if (parent.reportResults)
>     parent.reportResults(...);
>
> ...and using <iframe>s to embed the tests one after another. I use
> something similar for my performance tests:
>
>   http://www.hixie.ch/tests/adhoc/perf/
>
> This makes the test even easier to maintain, and also makes it a lot
> easier to reimplement the harness or to share tests between  
> hardnesses.

This approach would not be so good for the WebKit regression test  
harness, which loads pages one at a time in an offscreen window  
without any use of frames.

Our regression test framework can run large numbers of tests  
significantly faster than any browser-based harness I know of, and it  
would be nice if any shared pool of regression tests could be  
compatible with it.

WebKit's test setup is actually pretty advanced and can test many  
things that one might expect to require user interaction or a central  
web server, such as event handling, back/forward history, window  
opening, cross-site security, drag and drop, XMLHttpRequest in the  
face of different server behavior, etc.

Regards,
Maciej