> I was told that the Unified CJK thing adopted a different "philosophy" from the rest of Unicode
What would that philosophy be about? Sounds apocryphal. Unicode has never done "unification" like that for other languages/scripts?
i/ı/i, ö/ø/ø̈/oͤ: Same same, different codepoints.
Search and sorting is a mess everywhere. Depending on your locale, ö sorts either after o or after z. Sometimes it's semantically and phonetically equivalent to o wrt search but moreoften not. https://en.wikipedia.org/wiki/%C3%96
Well, Unicode didn't unify 門/门 either, but the appearance of the second glyph changes a lot depending on whether you use a Chinese font or a Japanese one. Meanwhile they did unify all the graphical variants of Latin characters with identical semantics, it's just that everyone is so used to the result that you hardly notice not being able to control whether g has a hook or a looped tail, and nobody expects to copy some German blackletter text into a French document and have it show up visually distinct, without explicitly setting the font.
I was involved in this before Unicode be a e Unicode when it started as discussions with Xerox and Apple—Xerox at that time had their XCCS standard. Those discussions directly morphed into Unicode. [[I was even there when we picked the name Unicode for what we had home to produce.]]
Conceptually we long debated unifying everything and in an ideal perfect world we would have done it. The reason was one primary goal for a new standard was to make it easily parseable and having unique rather than repeated codes was key to that. Sadly in the end we did not unify everything only to get buy-in from all major countries to support. That’s even why you see the roman/asciii characters repeated within Unicode itself—like as romaji. This was all well in good until we came to CJK and the number characters with semantical overlap was huge that this was more seriously considered—infact we started investigating this at Xerox before even thinking about Unicode and that work predated and influenced and leveraged the work done later.
Yes, and Cyrillic letters also look quite different in Bulgarian and Russian fonts. So what?
Is that true? Other than things like not seeing a Ы, Э or Ё in Bulgarian, the modern printed fonts look the same to me, though I am not from either country.
Some letters look different: https://en.wikipedia.org/wiki/Bulgarian_alphabet#/media/File...
Same thing, different code points is the opposite of han unification. Han unification is same code point, different thing.
Unicode has done the same unification for cuneiform languages. If you’re dealing with texts in Hittite, Sumerian, or Akkadian, you need to make sure you’re using the right fonts, otherwise the text will look entirely wrong.
Unicode merges some Chinese and Japanese characters into same code points and leave others as they are, on random per-character basis. So Chinese guys gets a ton of Japanese mixed texts and it causes tons of problems.
Japanese users are angry when system chooses Chinese fonts over Japanese, Chinese users apparently suffer with a font mishmash of Chinese and Japanese all the time like a mid-word capitalization. The official sanctioned solution is to just commit a genocide and nuke the offending language out of the system or to attach IVS to every Japanese characters which is like zero padding every single characters by a byte or two.
It's really putting everyone in pain and making absolutely no one happy, which sounds like what a good compromise tends to be, but LLMs seem to be struggling with there being multiple completely separate language families sharing codepoints but not character shapes or syntax. I've seen a smaller LLM model use made up verbs spilling over from the other language. I believe ML guys don't like datasets that do that kind of things. Same apply to image models.