[whatwg] [editing] New conformance tests

[Ehsan Akhgari]

On 11-09-14 5:24 PM, Aryeh Gregor wrote:
> Apparently I need a faster CPU:
>
> Time elapsed: 15:38.332 min.Summary
> Found 56734 tests14107 Pass42627 Fail
>
> By way of comparison, in Firefox 8.0a2:
>
> Time elapsed: 4:51.723 min.
> Summary
>
> Found 56734 tests
> 14759 Pass
> 41975 Fail
>
> Annoyingly, it looks like Opera is failing some tests solely due to
> not following the HTML5 serialization algorithm.  I rely on innerHTML
> heavily for the tests.  I could try to work around these failures, but
> hopefully IE and Opera will migrate to HTML5 serialization soon enough
> that it won't be worth the effort.

Maybe we can split these tests into multiple files, with one master file 
which loads them all, one by one, in iframes or something?

I just tried running them under a debug build of Firefox, and they take 
about three times as long as an optimized build for me (it was a bit 
more than 3 minutes for me, but I have a pretty fast machine!)

> On Tue, Sep 13, 2011 at 6:25 PM, Ehsan Akhgari<ehsan at mozilla.com>  wrote:
>> This looks really interesting.  Do you think this conformance suite is in a
>> good stage for browser vendors to integrate it in their test suites?  I'm
>> interested in integrating this into Mozilla's test suite to make sure that
>> we don't regress anything covered by these tests without realizing it.
>
> It's very likely that some of the tests are wrong, or that they match
> the spec but the spec is wrong.  With that kept in mind, yes, it
> should be suitable for integration into browser test suites.

These are all fine.  I'm planning to write a script which gathers the 
current status of the tests, and then use that every time that I update 
the test suite to get the expected result set.  I'll I care about at 
this point is that we don't change something in Gecko which changes 
these expected test results (either making some tests that passed before 
fail or vice versa) without understanding why.

 > It uses
> James Graham's testharness.js, so I should think you'd want to make
> some bridge that lets that integrate into your test suites, then plug
> these in.  testharness.js is designed for this use-case, but as far as
> I know, only Opera uses it that way at this point.  You might want to
> speak to James about it if you want to work on integrating it.  There
> are lots more tests that use testharness.js, such as these:
>
> http://dvcs.w3.org/hg/html/file/tip/tests/submission/AryehGregor/reflection
> http://dvcs.w3.org/hg/webapps/file/tip/DOMCore/tests/submissions/Ms2ger
>
> And more are being written regularly.  This is now the preferred test
> format for the HTML and WebApps WGs, so it would be great if browsers
> supported it.

Definitely.  I will send him an email CCing you guys to ask him about 
the best way to approach this problem.

In the mean time, do you know about the licensing of the stuff under 
http://dvcs.w3.org/hg/html/file/tip/tests?  I want to make sure that 
they have a compatible license so that we can import them into our tree.

Cheers,
Ehsan