[imps] Cross browser automated tests

[Sylvain Pasche]

Adam Barth wrote:
> 
> The WebKit LayoutTestController might be a reasonable starting point
> for a testing API.  Just calling parent.reportResults might be
> limiting for security testing because a frame might not be able to
> access its parent.  For example, the LayoutTestController has a
> globally readable and writable boolean flag that helps cross-domain
> frames signal that the test is complete.

One fact to keep in mind is that all browsers won't be able to provide 
something like a LayerTestController object to web pages (but having 
both WebKit and Firefox do that is already a good coverage).

However, testing some features requires that a privileged API is 
available for the testcases (synthesizing privileged events, clearing 
caches, ...).  One idea would be to offer a generic privileged API that 
browsers could implement to some degree.  Then, tests should be flagged 
in some way to describe what privileged API they require to be run. 
They would be skipped on browsers not implementing that.  To have the 
widest browser coverage, tests should only require these privileged API 
when needed.

For the WebKit security tests, being able to run them on a browser where 
something like the LayoutTestController is not available seems 
difficult.  In the experiments I did to run the WebKit tests, I loaded 
them in a frame, and extracted the page text to compare against the 
-expected.txt file.  This approach won't work directly for testing top 
level frames.

>> ...and using <iframe>s to embed the tests one after another.
> 
> This is also limiting for security tests because some behavior must be
> tested in a top-level frame.
> 
>> This makes the test even easier to maintain, and also makes it a lot
>> easier to reimplement the harness or to share tests between hardnesses.
> 
> One of our goals for the project is to integrate with Firefox's
> testing harness.  Ideally, the Firefox source tree could check out the
> WebKit LayoutTests unmodified and run them as part of their
> per-checkin regression testing.

I had a look at your extension.  Out of curiosity, did you develop some 
tool like run-webkit-tests for loading pages inside Firefox and 
comparing the actual and expected .txt files?


Sylvain