[whatwg] Encoding sniffing algorithm - update proposal

[Leif Halvard Silli]

I have just written a document on how implementations prioritize 
encoding info for HTML documents.[1] (As that document shows, I have 
not tested Safari 6.) Based on my findings there, I would like to 
suggest that the spec's encoding sniffing algorithm should be updated 
to look as follows:

Revised encoding sniffing algorithm proposal:

NEW! 0. document is XML format - opt out of the algorithm.
        [This step is already implicit in the spec, but it would
        make sense to explicitly include it to make sure that
        one could e.g. write test cases to see that it is step
        is implemented. Currently Safari, Chrome and Opera do 
        not 100% implement this step.]
         
NEW! #. Alternative: The BOM signa­ture could go here instead of 
        in step 5. There is a bug to move the BOM hereto and make
        it override anything else. What speaks against this are:
          a) that Firefox, IE10 and Opera do not currently have
             this behavior.
          b) this revision of the sniffing algorithm, especially
             the revision in step 6 (required UTF-8 detection),
             might make the BOM-trumps-everything-else override
             less necessary
        What speaks for this override:
          a) Safari, Chrome and legacy IE implement it.
          b) some legacy content may depend on it

     1. user override.
        (PS: The spec should clarify whether user override is
             cacheable.)

NEW! 2. iframe inherits user override from parent browsing context
        [Currently not mentioned in the spec, despite that "all"
         UAs do have this step for HTML docs.]

     3. explicit charset attribute in Content-Type header.

     4. BOM signa­ture [or as the second step, see above]

     5. native markup label <meta charset=UTF-8>

NEW! 6. UTF-8 detection.
        I think we should separate UTF-8 detection from other
        detection in order to make this step obligatory.
        The newness here is only the limitation to UTF-8
        detection plus that it should be obligatory. 
        (Thus: If it is not detected as UTF-8, then
        the parser proceeds to next step in the algorithm.)
        This step would make browsers lean more strongly 
        towards UTF-8.

NEW! 7. parent browsing context default.
        The current spec does not mention this step at all,
        despite that both Opera, IE, Safari, Chrome, Firefox
        do implement it.

        Regarding 6. and 7., then the order is important. Chrome
        does for instance perform UTF-8 detection, but it does it
        only /after/ the parent browsing context. Whereas everyone
        else (Opera 12 by default, Firefox for some locales - don't
        know if there are others) let it happen before the 'parent
        browsing context default'.

NEW! 8. info on “the likely encoding”
        The main newness is that this step is placed _after_ 
        the (revised) UTF-8 detection and after the (new) parent
        browsing context default.
        The name 'the likely encoding' is from the current spec
        text. I am a bit uncertain about what it means in the 
        current spec, though. So I move here what I think make
        sense. The steps under this point should perhaps be
        optional:

        a. detection of other charsets than UTF-8
           (e.g the optional Cyrillic detection in
           Firefox or legacy Asian encoding detection.
           The actual detection might happen in step 6,
           but it should only be made to count here.)
        b. markup label of the sister language
           <?xml version="1.0" encoding="UTF-8"?>
           (Opera/Webkit/Chrome currently have this directly
           after the native encoding label step - step 5.
        c. Other things? What does "likely encoding" current
           refer to, exactly?

     9. locale default

[1] 
http://malform.no/blog/white-spots-in-html5-s-encoding-sniffing-algorithm
[2] To the question of whether the BOM should trump everything else, 
then I think it it would be more important to get the other parts of 
this algorithm right. If we do get the rest of it right, then the 'BOM 
should trump' argument, becomes less important.
-- 
Leif Halvard Silli