[whatwg] [editing] New conformance tests

[Aryeh Gregor]

On Fri, Sep 9, 2011 at 12:48 PM, Bjartur Thorlacius
<svartman95 at gmail.com> wrote:
> It freezes surf as well, and then crashes it (though that's clearly a bug in
> surf and maybe JavaScriptCore). Opera runs it fine, but the chrome is slowed
> down severely. The results are quite a bit bigger than my copy buffer, and
> polite ML messages, though :) Maybe my Opera build is just plain outdated,
> but I felt the numbers might be interesting either way.
>
> Opera 10.60 Internal. Build 6386 for Linux.
> Compiled on Jun 30 2010 by gcc 4.3.2 (ABI: 1002) for GNU libc 2.7.
>
> Time elapsed: 2:38.857 min.
> Summary
>
> Found 56720 tests
> 13685 Pass
> 43035 Fail

Apparently I need a faster CPU:

Time elapsed: 15:38.332 min.Summary
Found 56734 tests14107 Pass42627 Fail

By way of comparison, in Firefox 8.0a2:

Time elapsed: 4:51.723 min.
Summary

Found 56734 tests
14759 Pass
41975 Fail

Annoyingly, it looks like Opera is failing some tests solely due to
not following the HTML5 serialization algorithm.  I rely on innerHTML
heavily for the tests.  I could try to work around these failures, but
hopefully IE and Opera will migrate to HTML5 serialization soon enough
that it won't be worth the effort.
On Tue, Sep 13, 2011 at 6:25 PM, Ehsan Akhgari <ehsan at mozilla.com> wrote:
> This looks really interesting.  Do you think this conformance suite is in a
> good stage for browser vendors to integrate it in their test suites?  I'm
> interested in integrating this into Mozilla's test suite to make sure that
> we don't regress anything covered by these tests without realizing it.

It's very likely that some of the tests are wrong, or that they match
the spec but the spec is wrong.  With that kept in mind, yes, it
should be suitable for integration into browser test suites.  It uses
James Graham's testharness.js, so I should think you'd want to make
some bridge that lets that integrate into your test suites, then plug
these in.  testharness.js is designed for this use-case, but as far as
I know, only Opera uses it that way at this point.  You might want to
speak to James about it if you want to work on integrating it.  There
are lots more tests that use testharness.js, such as these:

http://dvcs.w3.org/hg/html/file/tip/tests/submission/AryehGregor/reflection
http://dvcs.w3.org/hg/webapps/file/tip/DOMCore/tests/submissions/Ms2ger

And more are being written regularly.  This is now the preferred test
format for the HTML and WebApps WGs, so it would be great if browsers
supported it.