[whatwg] Encoding: big5 and big5-hkscs

Sun Apr 8 10:03:58 PDT 2012

On Sat, 07 Apr 2012 16:04:55 +0200, Øistein E. Andersen <liszt at coq.no>  
wrote:

> On Fri Apr 6 14:03:22 PDT 2012, Philip Jägenstedt <philipj at opera.com>  
> wrote:
>
>> So, <http://people.opera.com/philipj/2012/04/06/big5-foolip.txt> is the
>> mapping I suggest, with 18594 defined mappings and 1188 U+FFFD.
>
> (Second byte 0xA1 appears as 0x7F in the mapping file.)

Oops, I blame Anne's get_bytes() which had an off-by-one error. This is  
the corrected version:

def get_bytes(index):
     row = 0xFE-0xA1 + RANGE + 1
     lead = (index // row) + 0x81
     cell = index % row
     trail = (cell + 0xA1 - RANGE) if cell >= RANGE else cell + 0x40
     return (lead, trail)

I've also updated big5-foolip.txt with this fix.

> Your table is very similar to my idea of an ideal Big5-HKSCS decoder,  
> which follows Unihan, official Big5-HKSCS mappings and real  
> implementations closely.  My version is documented in detail at  
> <http://coq.no/character-tables/chinese-traditional/en> and summarised  
> at <http://coq.no/character-tables/charset5/en>.
>
> Only 2 mappings differ, viz, C6CD (U+2F33 v. U+5E7A) and F9FE (U+FFED v.  
> U+2593), which is quite reassuring given that we have worked  
> independently and used somewhat different approaches.  The two divergent  
> mappings and related issues are discussed below.

Yes, it's very encouraging indeed that we got this close independently and  
with different methods!

>> [in summary:]
>> C6CF => U+5EF4 廴 [v.] U+2F35 ⼵
>> C6D3 => U+65E0 无 [v.] U+2F46 ⽆
>> C6D5 => U+7676 癶 [v.] U+2F68 ⽨
>> C6D7 => U+96B6 隶 [v.] U+2FAA ⾪
>> C6DE => U+3003 〃[v.] U+F6EE [PUA]
>> C6DF => U+4EDD 仝 [v.]  U+F6EF [PUA]
>
> To this list can be added:
>
> C6CD => U+5E7A v. U+2F33
>
> These seven characters are all part of the E-Ten 1 extension [1] to  
> Big5, which is included in all implementations of Big5 (with or without  
> HK extensions) that I have come across in browsers .  The official  
> Big5-HKSCS table includes the E-Ten 1 extension as well, but the seven  
> characters listed above appear elsewhere in the HKSCS extensions and are  
> handled specially to avoid encoding the same character more than once.
>
> 	[1] <http://coq.no/character-tables/eten1.pdf>   
> <http://coq.no/character-tables/eten1.js>

What is the source for the mappings in eten1.pdf? I assume that E-Ten was  
originally just some Big5 fonts with no defined mappings to Unicode?

> Five are Kangxi radicals and encoded twice in Unicode (once as radicals,  
> once as normal Han characters).  The official Big5-HKSCS table maps C6CD  
> to the Unicode Kangxi radical U+2F33 and does not list the remaining  
> four codepoints at all.  U+2F33 is the only Unicode Kangxi radical  
> included in the official Big5-HKSCS table.  As you have noticed, some  
> Big5-HKSCS implementations follow this idea for the remaining four as  
> well.  It seems better to follow non-HK Big5 implementations here and  
> map all five to normal Unicode Han characters.
>
> For the last two characters, U+3003 and U+4EDD, there is only one  
> possible Unicode mapping, so duplicates are impossible to avoid (without  
> using PUA characters).  The official Big5-HKSCS table does not map C6DE  
> and C6DF to anything.
>
> Suggested change:  map C6CD to U+5E7A.

These are the existing mappings:

C6CD =>
opera-hk: U+2F33 ⼳
firefox: U+5E7A 幺
chrome: U+F6DD 
firefox-hk: U+5E7A 幺
opera: U+2F33 ⼳
chrome-hk: U+2F33 ⼳
internetexplorer: U+F6DD 
hkscs-2008: <U+2F33> ⼳

At least on the Web, this isn't a question of HK vs non-HK mappings. Other  
than Firefox, which (de-facto) specs or implementations use U+5E7A?

Possibly, one could argue that U+2F33 normalizes (NFKC) to U+5E7A, but  
it's not the only hanzi in HKSCS-2008 that normalizes into something else:

8BC3 => <U+2F878> 屮 => <U+5C6E> 屮
8BF8 => <U+F907> 龜 => <U+9F9C> 龜
8EFD => <U+2F994> 芳 => <U+82B3> 芳
8FA8 => <U+2F9B2> 䕫 => <U+456B> 䕫
8FF0 => <U+2F9D4> 貫 => <U+8CAB> 貫
C6CD => <U+2F33> ⼳ => <U+5E7A> 幺
957A => <U+2F9BC> 蜨 => <U+8728> 蜨
9874 => <U+2F825> 勇 => <U+52C7> 勇
9AC8 => <U+2F83B> 吆 => <U+5406> 吆
9C52 => <U+2F8CD> 晉 => <U+6649> 晉
A047 => <U+2F840> 咢 => <U+54A2> 咢
FC48 => <U+2F894> 弢 => <U+5F22> 弢
FC77 => <U+2F8A6> 慈 => <U+6148> 慈

I'm not sure what the conclusion is...

> On Fri Apr 6 06:42:26 PDT 2012, Philip Jägenstedt <philipj at opera.com>  
> wrote:
>
>> Also, a single mapping fails the Big5-contra[di]ction test:
>>
>> F9FE =>
>> opera-hk: U+FFED ￭
>> firefox: U+2593 ▓
>> chrome: U+2593 ▓
>> firefox-hk: U+2593 ▓
>> opera: U+2593 ▓
>> chrome-hk: U+FFED ￭
>> internetexplorer: U+2593 ▓
>> hkscs-2008: <U+FFED> ￭
>>
>> I'd say that we should go with U+FFED here, since that's what the  
>> [HKSCS-2008] spec
>> says and it's visually close anyway.
>
> Given that the goal is to define a unified Big5 (non-HK) and Big5-HKSCS  
> encoding and that this seems to be a case of the HK standard going  
> against everything and everyone else, perhaps more weight should be  
> given to existing specifications and (non-HK-specific) implementations.
>
> Suggested change:  map F9FE to U+2593

This is the only mapping where IE maps something other than PUA or "?"  
that my mapping doesn't agree on, so I don't object to changing it. Still,  
it would be very interesting to know why HKSCS-2008 changed it, do you  
know?

> Duplicates and reverse mappings:
>
> big5-foolip.txt currently provides two different codepoints for 100  
> Unicode characters.
>
> 6 are mentioned above.  84 result from compatibility mappings defined in  
> the official HKSCS-2008 specification, cf. [2].  This leaves 10:
>
> 	[2] <http://coq.no/character-tables/o-h-comp.pdf>  
> <http://coq.no/character-tables/o-h-comp.js>
>
> U+5341 (十, 'ten') and U+5345 (卅 'thirty') are encoded twice in Big5,  
> once as numerals and once as standard Han characters.  (U+5344 卄  
> 'twenty' is only encoded once in Big5, but was added to HKSCS and is now  
> one of the 84 compatibility mappings.)
>
> According to Lunde, the four codepoints F9FA--F9FD (ETen-2 extension)  
> are supposed to encode double-stroked circle segments which appear to be  
> missing from Unicode (I am not sure whether they have ever been proposed  
> for inclusion).  They are currently mapped to single-stroked circle  
> segments instead, but those are already encoded at A2E7, A2A1--A2A3  
> (original Big5).
>
> The four codepoints F9F9, F9E9, F9EB, F9EA (ETen-2 extension) encode  
> line-drawing characters with a double horizontal line.  These appear to  
> be encoded at A2A4--A2A7 (original Big5) already, and it is not clear to  
> me whether the characters at A2A4--A2A7 are supposed to be different or  
> whether ETen chose to encode them again to have a full set of  
> line-drawing characters in one location.
>
> Suggested reverse mappings:
>
> C6CF <= U+5EF4
> C6D3 <= U+65E0
> C6D5 <= U+7676
> C6D7 <= U+96B6
> C6DE <= U+3003
> C6DF <= U+4EDD
> C6CD <= 5E7A (if the mapping of C6CD is changed)
> Rationale:  Only these mappings will work for non-HK Big5  
> implementations, and these characters appear to be important not only in  
> Hong Kong.
>
> A451 <= U+5341
> A4CA <= U+5345
>
> A27E <= U+256D
> A2A1<= U+256E
> A2A3 <= U+256F
> A2A2 <= U+2570
>
> F9F9 <= U+2550
> F9E9 <= U+255E
> F9EB <= U+2561
> F9EA <= U+256A
>
> (The 84 compatibility mappings should obviously only be used to decode  
> and never as reverse mappings.)

Anne, how do you plan to define encoders for tables with duplicate  
mappings? Have you collected data for what browsers currently do?

In any event, it clearly needs to be defined what to do for these 100 code  
points that have multiple mappings to Big5. I extended my Python script to  
find these 100 duplicates and to check what Python did for 'big5', falling  
back to 'big5-hkscs'. This is what it produced:

8FB6 <= U+880F
90C4 <= U+96B6
91BE <= U+9F17
9242 <= U+8503
9361 <= U+5F0C
9455 <= U+7250
947A <= U+7468
96EE <= U+701E
9975 <= U+732A
9CE4 <= U+975D
9DEF <= U+5605
9DFB <= U+5ED0
A05F <= U+936E
A0D4 <= U+89A9
A0DC <= U+60A4
A1B2 <= U+3003
A259 <= U+5159
A25A <= U+515B
A25B <= U+515E
A25C <= U+515D
A260 <= U+74E9
A261 <= U+7CCE
A27E <= U+256D
A2A1 <= U+256E
A2A2 <= U+2570
A2A3 <= U+256F
A2A4 <= U+2550
A2A5 <= U+255E
A2A6 <= U+256A
A2A7 <= U+2561
A2CD <= U+5344
A451 <= U+5341
A4CA <= U+5345
A55D <= U+5305
A7FB <= U+675E
A9E4 <= U+62D0
A9F0 <= U+62CE
AACC <= U+8005
ABEC <= U+6062
ADC5 <= U+5029
ADEB <= U+537F
AFB0 <= U+79E3
B05F <= U+8D77
B0B0 <= U+507D
B3A3 <= U+90FD
B440 <= U+5A77
B4B8 <= U+6674
B4E4 <= U+6E2F
B4FC <= U+6E1D
B54E <= U+716E
B5AE <= U+7B51
B5D7 <= U+83C1
B7EC <= U+745C
B9B0 <= U+50ED
BAE6 <= U+7BB8
BAFC <= U+7DD2
BCB5 <= U+6490
BF47 <= U+6FB6
BFA6 <= U+7E1D
BFAE <= U+8028
BFCC <= U+89A6
C052 <= U+975C
C0E7 <= U+71DF
C554 <= U+97FF
C5F7 <= U+77D7
C95C <= U+5C10
C969 <= U+4EDD
C9DB <= U+5E75
C9FC <= U+6C4A
CA52 <= U+9097
CB58 <= U+6C9C
CDE7 <= U+4FBB
CFF1 <= U+7809
D0C0 <= U+91D4
D256 <= U+6D67
D4D1 <= U+5A67
D8F4 <= U+5F58
DB5D <= U+83CF
DB79 <= U+840F
DC52 <= U+9104
DE72 <= U+7162
DECD <= U+75F9
E07C <= U+8F0B
E3C8 <= U+84A8
E6AB <= U+7479
E6D0 <= U+799B
E8CD <= U+99D6
E959 <= U+5B28
EBC9 <= U+8F36
EDCA <= U+7C06
EFF9 <= U+7201
F1E3 <= U+9F16
F5E8 <= U+7E87
F86D <= U+9DF0
F9C4 <= U+9B2E
F9D7 <= U+92B9
FBFD <= U+5EF4
FCD3 <= U+65E0
FD64 <= U+60DE
FEC1 <= U+7676

These are the ones where you (Øistein) disagree:

> C6CF <= U+5EF4
> C6D3 <= U+65E0
> C6D5 <= U+7676
> C6D7 <= U+96B6

AFAICT this has nothing to do with compatibility mappings, so what's the  
reason for this?

> F9E9 <= U+255E
> F9EA <= U+256A
> F9EB <= U+2561
> F9F9 <= U+2550

Python's big5-hkscs agrees, but Python's big5 does this instead:

A2A5 <= U+255E
A2A6 <= U+256A
A2A7 <= U+2561
A2A4 <= U+2550

It seems safer to go with the big5 mappings, but checking what browsers do  
would be helpful.

How about the rest of my generated list, is that fine?

> On Fri Apr 6 14:03:22 PDT 2012, Philip Jägenstedt <philipj at opera.com>  
> wrote:
>
>> There are 29 mappings to U+003F (?) in IE that no other browser has.
>
> Are you referring to the ones at A3E2--A3FE?  IE decodes (or used to  
> decode) the control pictures at A3C0--A3E0 as C0 control characters in  
> plain text, but replace(s) them with question marks in HTML.  It looks  
> like this treatment has been extended to the the remaining A3xx  
> codepoints (after the euro), perhaps without a good reason.

Yes, that's the range. I think we should leave these undefined.

>> The remaining mappings are to PUA or U+FFFD in all browsers [...].  
>> Mapping
>> these to U+FFFD unless anyone finds pages using these byte sequences  
>> seems
>> the only sane option.
>
> Agreed.  Do any of these ever render in a meaningful way (e.g., in IE on  
> a Windows machine with HK locale and appropriate HKSCS PUA fonts)?
>
> The following 22 codepoints are 'reserved for backwards compatibility'  
> in the HKSCS-2008 standard, but no Unicode mappings are provided:
>
> 9EAC
> 9EC4
> 9EF4
> 9F4E
> 9FAD
> 9FB1
> 9FC0
> 9FC8
> 9FDA
> 9FE6
> 9FEA
> 9FEF
> A054
> A057
> A05A
> A062
> A072
> A0A5
> A0AD
> A0AF
> A0D3
> A0E1
>
> I assume some systems will render at least these as potentially  
> meaningful Han characters.

I generated  
<http://people.opera.com/philipj/2012/04/08/big5-undefined-ie.txt> and had  
a look using various Chinese fonts in Windows 7. It looks like most fonts  
have a copy of the printable ASCII characters in U+F020 through U+F07E,  
and what looks like parts of windows-1252 or latin-1 up to U+F0FF.

Exactly the 22 codepoints you list *are* Han characters in the  
MingLiu_HKSCS font, see  
<http://people.opera.com/philipj/2012/04/08/big5-mingliu-hkscs.png>.  
Presumably they were not in Unicode when HKSCS-2008 was defined, but if  
they have been added since I think we should simply map them.  
Unfortunately, I haven't been able to find them by searching by radicals  
in the Unihan database...

-- 
Philip Jägenstedt
Core Developer
Opera Software