[whatwg] Encoding sniffing algorithm

[Ian Hickson]

On Fri, 27 Jul 2012, Leif Halvard Silli wrote:
>
> I have just written a document on how implementations prioritize 
> encoding info for HTML documents.[1] (As that document shows, I have not 
> tested Safari 6.) Based on my findings there, I would like to suggest 
> that the spec's encoding sniffing algorithm should be updated to look as 
> follows:
> 
> Revised encoding sniffing algorithm proposal:
> 
> NEW! 0. document is XML format - opt out of the algorithm.
>         [This step is already implicit in the spec, but it would
>         make sense to explicitly include it to make sure that
>         one could e.g. write test cases to see that it is step
>         is implemented. Currently Safari, Chrome and Opera do 
>         not 100% implement this step.]

I don't understand the relevance of the algorithm to XML. Why would anyone 
even look at this algorithm if they were parsing XML?


> NEW! #. Alternative: The BOM signa­ture could go here instead of 
>         in step 5. There is a bug to move the BOM hereto and make
>         it override anything else. What speaks against this are:
>           a) that Firefox, IE10 and Opera do not currently have
>              this behavior.
>           b) this revision of the sniffing algorithm, especially
>              the revision in step 6 (required UTF-8 detection),
>              might make the BOM-trumps-everything-else override
>              less necessary
>         What speaks for this override:
>           a) Safari, Chrome and legacy IE implement it.
>           b) some legacy content may depend on it

Not sure what this means.


>      1. user override.
>         (PS: The spec should clarify whether user override is
>              cacheable.)

This seems to be entirely a user interface issue.


> NEW! 2. iframe inherits user override from parent browsing context
>         [Currently not mentioned in the spec, despite that "all"
>          UAs do have this step for HTML docs.]

That's a UI issue much like whether it's remembered or not. But I've added 
a non-normative note.


> NEW! 6. UTF-8 detection.
>         I think we should separate UTF-8 detection from other
>         detection in order to make this step obligatory.
>         The newness here is only the limitation to UTF-8
>         detection plus that it should be obligatory. 
>         (Thus: If it is not detected as UTF-8, then
>         the parser proceeds to next step in the algorithm.)
>         This step would make browsers lean more strongly 
>         towards UTF-8.

Without a specific algorithm to detect UTF-8, this is meaningless.


> NEW! 7. parent browsing context default.
>         The current spec does not mention this step at all,
>         despite that both Opera, IE, Safari, Chrome, Firefox
>         do implement it.

Added. (Some comprehensive testing of this would be good, e.g. comparing 
it to each of the earlier and later steps, considering it with different 
ways of giving the encoding, differnet locales, etc.)


>         Regarding 6. and 7., then the order is important. Chrome
>         does for instance perform UTF-8 detection, but it does it
>         only /after/ the parent browsing context. Whereas everyone
>         else (Opera 12 by default, Firefox for some locales - don't
>         know if there are others) let it happen before the 'parent
>         browsing context default'.

Can you elaborate on this?


> NEW! 8. info on “the likely encoding”
>         The main newness is that this step is placed _after_ 
>         the (revised) UTF-8 detection and after the (new) parent
>         browsing context default.
>         The name 'the likely encoding' is from the current spec
>         text. I am a bit uncertain about what it means in the 
>         current spec, though. So I move here what I think make
>         sense. The steps under this point should perhaps be
>         optional:
> 
>         a. detection of other charsets than UTF-8
>            (e.g the optional Cyrillic detection in
>            Firefox or legacy Asian encoding detection.
>            The actual detection might happen in step 6,
>            but it should only be made to count here.)

I don't understand your reasoning on the desired ordering here.


>         b. markup label of the sister language
>            <?xml version="1.0" encoding="UTF-8"?>
>            (Opera/Webkit/Chrome currently have this directly
>            after the native encoding label step - step 5.

No idea what this means.


>         c. Other things? What does "likely encoding" current
>            refer to, exactly?

The spec gives an example.

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'