[imps] Cross browser automated tests: API proposal

[Sylvain Pasche]

Hi,

I wrote down some of my investigations and first proposal which is 
visible there (sorry for my English):

http://wiki.whatwg.org/Browser_Tests_Project

There's nothing fancy there, that's basically taking the cross browser 
parts of both Mozilla and WebKit testing frameworks.

Mozilla Mochitests and WebKit LayoutTest approach are rather different. 
The LayoutTest approach will make it easier for converting existing non 
automated tests, you just have to create a thetest-expected.txt file 
alongside the test with the innerText value of the document at the end 
of the test.

For the Mochitests, the additional constraint not present in the 
existing Mozilla tests is to notify the end of tests. I saw that when 
running tests on Opera, you can get false positive when an exception is 
raised and not caught because "onerror" is not implemented (same for 
WebKit). I think calling finish() at the end is better than enclosing 
everything in try catch or counting the number of assertions.

As I mentioned, if there's interest a version of the API which does not 
require MochiKit could be created.


For the LayoutTests, something that can be tricky is that not all 
browser implement innerText, which is used for serializing the page 
content to text. The trick used by the WebKit layoutTestController 
Firefox extension or what I did for running the tests in a frame is to 
iterate over the nodes and ignore <script> elements. However this is 
very far from the innerText implementation on WebKit (and I guess IE) 
that only dumps visible elements, adds newlines around block elements 
and does others things. With the innerText approximation above we can 
still compare against the -expected.txt if all newlines and white space 
are removed, but making a diff between the actual and expected output is 
not meaningful.

Still lots of open questions, and some parts are not very detailed but 
that's a first start.


Sylvain