[whatwg] Surrogate pairs and character references

[Øistein E. Andersen]

On 15 Sep 2009, at 03:03, Ian Hickson wrote:

> [...] [R]egardless of the environment, �� and  
> 𐀀 are not the
> same -- the first has two invalid characters U+D800 and U+DC00, the  
> second
> has one character U+10000.

That works well as long as the concepts are couched in abstract terms,  
but how is one expected to prevent adjacent surrogates from coalescing  
in a UTF-16 environment?

Firefox circumvents the problem by substituting U+FFFD for the  
surrogates; other browsers make no attempt to prevent coalescence.

> I'm not really sure how to make that clearer in the spec.

(Let us first determine what should happen.)

> I suppose we
> could just change the spec and say that surrogate characters (whether
> literal characters, e.g. in UTF-8, or from character references) all  
> get
> converted to U+FFFD?.

That seems to be the only reasonable option if handling  
�� as U+FFFD U+FFFD is deemed desirable and sufficiently  
compatible with existing documents.  It would simplify things a bit in  
non-UTF-16 environments (as compared to my interpretation of the  
current text) without much added complexity in UTF-16 environments.

> [\xD800�] should give U+FFFD U+DC00. It's not clear to me why  
> that is not clear. :-)
> Could you walk me through the spec interpreting it in such a way  
> that you
> get any other result?

See below.

> The spec says "Bytes or sequences of bytes in the original byte stream
> that could not be converted to Unicode characters must be converted to
> U+FFFD REPLACEMENT CHARACTER code points".

I take it you mean that \xD800� should turn into \xFFFD�  
at this point, which is only supported by the quoted text if "bytes or  
sequences of bytes" representing surrogates "[cannot] be converted to  
Unicode characters" or, to put it differently, if surrogates are not  
"Unicode characters".

Unfortunately for this reading, the term "Unicode character" does not  
seem to be defined in HTML5 or in Unicode, and the following paragraph  
(which appears shortly after the one you quoted) clearly includes  
surrogate code points within the concept of "Unicode character":

"Any occurrences of any characters in the ranges [...] U+D800 to U 
+DFFF, [...] are parse errors. (These are all control characters or  
permanently undefined Unicode characters.)"

Moreover, this paragraph would be pointless if the characters  
mentioned therein could never occur at all.

***

The use of "Unicode character" without a definition is fine in other  
parts of HTML5, but clearly not sufficiently precise in this instance.  
If you want to exclude (unpaired) surrogate code points only, the  
appropriate term to use would probably be "Unicode scalar value".

-- 
Øistein E. Andersen