[whatwg] Encoding: big5 and big5-hkscs
Øistein E. Andersen
liszt at coq.no
Sun Apr 8 18:08:20 PDT 2012
On 8 Apr 2012, at 18:03, Philip Jägenstedt wrote:
> On Sat, 07 Apr 2012 16:04:55 +0200, Øistein E. Andersen <liszt at coq.no> wrote:
>>  <http://coq.no/character-tables/eten1.pdf> <http://coq.no/character-tables/eten1.js>
> What is the source for the mappings in eten1.pdf?
There is no authoritative source for those Unicode mappings, I am afraid. Lunde's CJKV opus was my main source, but it does not include Unicode mappings. Browser implementations and the HKSCS standard were also taken into account, but the final mapping does not reflect any one source.
> I assume that E-Ten was originally just some Big5 fonts with no defined mappings to Unicode?
Yes, I think so.
>> Suggested change: map C6CD to U+5E7A.
> These are the existing mappings:
> C6CD =>
> opera-hk: U+2F33 ⼳
> firefox: U+5E7A 幺
> chrome: U+F6DD
> firefox-hk: U+5E7A 幺
> opera: U+2F33 ⼳
> chrome-hk: U+2F33 ⼳
> internetexplorer: U+F6DD
> hkscs-2008: <U+2F33> ⼳
> At least on the Web, this isn't a question of HK vs non-HK mappings. Other than Firefox, which (de-facto) specs or implementations use U+5E7A?
I have now had a closer look at my notes (<http://coq.no/character-tables/chinese-traditional/en>). My argument for U+5E7A goes as follows:
Of the 214 Kangxi radicals, 186 appear (as normal Han character) in CNS 11643 Planes 1 or 2, whereas 25 appear in Plane 3 and 3 are missing altogether. Big5 only covers Planes 1 and 2, which means that 28 Kangxi radicals (which may be rare in running text, but are nevertheless important) are missing. The E-Ten 1 extension encodes 25 of the missing radicals in the range C6BF--C6D7. Unlike CNS 11643 and Unicode, Big5 does not encode radicals twice (as radicals and normal characters). This means that Big5 with the E-Ten 1 extension contains 211 of the 214 Kangxi radicals, all mapped to normal Han characters, and no codepoints mapped to Unicode Kangxi Radicals in the range U+2F00--U+2FD5.
In summary: although E-Ten 1 was not defined in terms of Unicode, it is clear that the 25 radicals were all meant to map to normal Han characters, not to the special radical characters found in CNS 11643 and Unicode.
Enter HKSCS. 20 of the E-Ten 1 Kangxi radical mappings (along with the rest of E-Ten 1 and E-Ten 2, or almost) are adopted, but the remaining 5 are instead given new codepoints elsewhere. Whatever the reason be, 4 of the 5 unused E-Ten positions are simply left undefined in the HKSCS standard, which is not much of a problem for a unified HK/non-HK Big5 encoding. Unfortunately, the position C6CD was not left undefined, but instead mapped to U+2F33 (⼳), the Unicode Kangxi Radical version of U+5E7A (幺), thus introducing not only the only Unicode Kangxi Radical into the HKSCS standard, but also a Unicode mapping that is incompatible with previous Big5 versions. I wish I knew why.
> Possibly, one could argue that U+2F33 normalizes (NFKC) to U+5E7A, but it's not the only hanzi in HKSCS-2008 that normalizes into something else:
> 8BC3 => <U+2F878> 屮 => <U+5C6E> 屮
> 8BF8 => <U+F907> 龜 => <U+9F9C> 龜
> 8EFD => <U+2F994> 芳 => <U+82B3> 芳
> 8FA8 => <U+2F9B2> 䕫 => <U+456B> 䕫
> 8FF0 => <U+2F9D4> 貫 => <U+8CAB> 貫
> C6CD => <U+2F33> ⼳ => <U+5E7A> 幺
> 957A => <U+2F9BC> 蜨 => <U+8728> 蜨
> 9874 => <U+2F825> 勇 => <U+52C7> 勇
> 9AC8 => <U+2F83B> 吆 => <U+5406> 吆
> 9C52 => <U+2F8CD> 晉 => <U+6649> 晉
> A047 => <U+2F840> 咢 => <U+54A2> 咢
> FC48 => <U+2F894> 弢 => <U+5F22> 弢
> FC77 => <U+2F8A6> 慈 => <U+6148> 慈
The other pairs all contain characters that look slightly different, whereas U+5E7A and U+2F33 look the same (and, I believe, are supposed to look the same), the only difference being that the former is a normal Han character whereas the latter carries the additional semantics of being a Kangxi radical.
> I'm not sure what the conclusion is...
I am not entirely sure either. It seems clear that the mapping from C6CD to U+2F33 makes no sense for non-HKSCS Big5 (which does not encode U+5E7A anywhere else), but it does not seem to make much sense for Big5-HKSCS either, which suggests that I might be missing something.
>> On Fri Apr 6 06:42:26 PDT 2012, Philip Jägenstedt <philipj at opera.com> wrote:
>>> Also, a single mapping fails the Big5-contra[di]ction test:
>>> F9FE =>
>>> opera-hk: U+FFED ￭
>>> firefox: U+2593 ▓
>>> chrome: U+2593 ▓
>>> firefox-hk: U+2593 ▓
>>> opera: U+2593 ▓
>>> chrome-hk: U+FFED ￭
>>> internetexplorer: U+2593 ▓
>>> hkscs-2008: <U+FFED> ￭
>>> I'd say that we should go with U+FFED here, since that's what the [HKSCS-2008] spec
>>> says and it's visually close anyway.
>> Given that the goal is to define a unified Big5 (non-HK) and Big5-HKSCS encoding and that this seems to be a case of the HK standard going against everything and everyone else, perhaps more weight should be given to existing specifications and (non-HK-specific) implementations.
>> Suggested change: map F9FE to U+2593
> This is the only mapping where IE maps something other than PUA or "?" that my mapping doesn't agree on, so I don't object to changing it. Still, it would be very interesting to know why HKSCS-2008 changed it, do you know?
No, I am afraid not. I have been wondering as well, but I have not been able to find an explanation.
Lunde (if I remember correctly, 1st Edn) and Kano's 'Developing International Software' (1st Edn, 1995) both show something like U+2593, but it could of course be that popular non-Unicode (HK) Big5 fonts had glyphs more like U+FFED, which would make the HKSCS-2008 mapping less surprising. Do let me know if you discover any information on this.
>> Duplicates and reverse mappings:
> [...] it clearly needs to be defined what to do for these 100 code points that have multiple mappings to Big5. I extended my Python script to find these 100 duplicates and to check what Python did for 'big5', falling back to 'big5-hkscs'. This is what it produced:
> These are the ones where you (Øistein) disagree:
>> C6CF <= U+5EF4
>> C6D3 <= U+65E0
>> C6D5 <= U+7676
>> C6D7 <= U+96B6
> AFAICT this has nothing to do with compatibility mappings, so what's the reason for this?
As I wrote, '[o]nly these mappings will work for non-HK Big5 implementations.' My reasoning was that a random Big5 implementation would be more likely to include the E-Ten 1 extension than the HKSCS extension. On the other hand, these codepoints could be less than ideal if major Big5-HKSCS implementations follow the standard strictly and map to nothing.
>> F9E9 <= U+255E
>> F9EA <= U+256A
>> F9EB <= U+2561
>> F9F9 <= U+2550
> Python's big5-hkscs agrees, but Python's big5 does this instead:
> A2A5 <= U+255E
> A2A6 <= U+256A
> A2A7 <= U+2561
> A2A4 <= U+2550
> It seems safer to go with the big5 mappings, but checking what browsers do would be helpful.
Does this imply that Python's big5 (non-HK) implementation does not include the corresponding E-Ten 2 (forward) mappings for decoding either?
F9E9 => U+255E
F9EA => U+256A
F9EB => U+2561
F9F9 => U+2550
My reasoning was that the Unicode characters are line-drawing characters, which is indisputably the case for the Big5 E-Ten 2 characters (F9xx) as well, whereas the intended use for the Big5 characters at A2xx is less clear (looking at a pre-Unicode Big5 fonts might be useful). An additional reason for my suggestion is that the CP950 table in Kano's book leaves the four A2xx codepoints blank, which suggests that the F9xx positions are (or have been considered) preferable.
> How about the rest of my generated list, is that fine?
The remaining 84 characters are handled correctly, yes. Here is the original HKSCS list:
point of HKSCS Unified With
>> On Fri Apr 6 14:03:22 PDT 2012, Philip Jägenstedt <philipj at opera.com> wrote:
>>> There are 29 mappings to U+003F (?) in IE that no other browser has.
>> Are you referring to the ones at A3E2--A3FE? [...]
> Yes, that's the range. I think we should leave these undefined.
> I generated <http://people.opera.com/philipj/2012/04/08/big5-undefined-ie.txt> and had a look using various Chinese fonts in Windows 7. It looks like most fonts have a copy of the printable ASCII characters in U+F020 through U+F07E, and what looks like parts of windows-1252 or latin-1 up to U+F0FF.
> Exactly the 22 codepoints you list *are* Han characters in the MingLiu_HKSCS font, see <http://people.opera.com/philipj/2012/04/08/big5-mingliu-hkscs.png>. Presumably they were not in Unicode when HKSCS-2008 was defined, but if they have been added since I think we should simply map them. Unfortunately, I haven't been able to find them by searching by radicals in the Unihan database...
Perhaps they are 'Phantom Ideographs' (Lunde), i.e., 'ideographs whose origins cannot be determined' and which will not be added to Unicode today, in which case there is not much we can do (but such characters are unlikely to see much use anyway).
Øistein E. Andersen
More information about the whatwg