Raw Materials -- pre-Unicode Japanese Character Sets

During the 19th century, the number of kanji required for literacy in Japan was perhaps about 4,000. Even at that time, there were many people calling out for a rationalization and pruning of the writing system. In the early 20th century, the Ministry of Education (now the Ministry of Culture) issued a list of common kanji and a new kana system. Newspapers also announced their own plans of restricting kanji to some sensible subset (although these subsets appear very large and baroque by modern standards). However, opposition from traditionalists effectively postponed reform until after World War 2. In 1945, the Yomiuri newspaper announced that the abolition of kanji would now finally be possible, which at the time wasn't too extreme a position -- others were advocating the total abandonment of the Japanese language!

Japanese character sets as we know them, therefore, have arisen from a background of rapid change and strong reformism.

Educational/Cultural Standards

Partly because of the mixed reception of the Toyo list, later lists such as the Joyo ('everyday use') list of 1981 have come as 'recommendations', followed to varying degrees by different organizations. The 1,945 characters of the Joyo list form a core group of kanji that seems unlikely to change much and which is included in every Japanese character set for computer use. However, there are several very common kanji that are not in Joyo.

The second main list of kanji referred to today is the Gakushuu ('educational') list. This is a list of 1,006 kanji divided into levels based (approximately) on frequency. This list specifies the minimum number of kanji to be known by students at various school levels. It is a subset of the Joyo list and therefore a subset of all common Japanese coded character sets.

In the immediate postwar period, the range of kanji used in newspapers and official publications was strongly restricted, by government-imposed guidelines, to the Joyo repertoire. This is no longer the case, now that reformism has died down, and a recent study suggested that only about 90% of the kanji occurring in a sample of major daily newspapers were Joyo kanji.

The number of kanji (and other characters) in common use in Japan appears to have been increasing since the 1980s, and some would argue that the Joyo and Gakushuu sets are less relevant now than they used to be. However, they are the nearest thing to a definition of 'commonly used kanji' and are likely to remain so for a long time.

Industrial Standards

In 1969, the JIS C 6220 standard was created, specifying two sets of characters: a localized ASCII (yen sign instead of backslash) and a set of half-width katakana, suitable for writing basic Japanese on dot-matrix era devices. The 'C' in the name refers to electronics; the standard was later renamed to JIS X 0201, where 'X' refers to information technology.

JIS X 0208 (originally JIS C 6226) of 1978 was the first JIS character set to include kanji. It specified 6,335 kanji, arranged by frequency into two levels. The arrangement of the JIS X 0208 kanji does not seem to be connected with that of the Joyo kanji. The initial release of this standard was extremely problematic, and it wasn't until the 1983 revision that it was reasonably error-free. The selection of kanji was also odd: the character set was assembled from various existing proprietary sets, but prewar forms and other variants were suppressed, while many unusual variants were included in order to write personal and place names -- although not enough to write all personal names. Many bizarre mistakes were made in transcribing names, resulting in several new kanji coming into existance. The result was that JIS X 0208 was a slowly-adopted and controversial standard, which competed with a number of proprietary character sets (e.g. Fujitsu's JEF).

The 1983 revision, while correcting most of the errors, also replaced some kanji present in the actual language with simplified forms that existed nowhere outside the Ministry. Also, because JIS X 0208 had been made without reference to either the Joyo (see above) or Jinmei (see below) lists, which were growing, various new characters had to be added. During the 80's the National Language Council (part of the Agency for Cultural Affairs, nothing to do with the Ministry of Economy, Trade and Industry) was called in to solve some of the problems that had arisen from JIS X 0208. This example of liaison between two unrelated departments was considered a great step forward.

The problematic nature of JIS X 0208 created many problems for Unicode. The needs of round-trip compatibility meant that some kanji which had been split into two variants in JIS X 0208 had to be given two Unicode code points, even though almost everyone would agree that the 'variants' are just slightly different glyphs (e.g. U+5294 and U+5292). Unification was also made harder by the need to work out which JIS X 0208 characters were really unique and which were just very eccentric forms of ordinary characters.

JIS X 0212, the 'hojokanji' (auxiliary kanji) list, was created in 1990 to silence the complaints about kanji missing from JIS X 0208. Unfortunately, by this time the encodings (eg Shift-JIS) that had already become widespread did not have room for the 5,801 kanji of the new set. Instead people were adding the kanji they needed into their own particular Shift-JIS implementations, a tendency which continues to this day. Although barely ever used, JIS X 0212 was quite controversial in the way it 'modernized' some kanji and created ambiguous variants which appear to cover a range of real-world variants.

In 1995, JIS X 0221 was defined. It was basically a Japanese translation of ISO/IEC 10646-1:1993, i.e. identical to Unicode 1.1. However, Unicode 1.1 was not used widely (if at all). Subsequent revisions of JIS X 0221 will presumably track the development of Unicode.

In 1996/7, JIS X 0208 was reviewed very effectively under the guidance of Kouji Shibano, with several results:

• A revision of JIS X 0208, reassigning the code points of several hundred kanji while only adding two new ones.
• Publication of the JIS Kanji Jiten, a tome explaining the JIS kanji in depth.
• The identification of hundreds of necessary kanji beyond those in JIS X 0208.

The newly identified kanji were collected into JIS X 0213, a standard intended both to satisfy those discontented with the JIS repertoire and to help get the new kanji into Unicode 3.1. This new JIS standard had many advantages: it was easy to integrate with Shift-JIS encoding, it was well-researched, and it was developed in close co-operation with Unicode. Despite these properties, it is rarely encountered.

There was a certain amount of disagreement as to whether JIS X 0213 was intended as a step on the way to Unicode, or as an alternative to Unicode. Unicode has round-trip compatibility with JIS X 0213 and is a superset of it, so the main reason to use JIS X 0213 now is probably compatibility with Shift-JIS encoding and Shift-JIS based fonts (which are still numerous). Some US companies refused to support JIS X 0213 until it could be made round-trip compatible with Unicode; whether this was dutiful attention to international standards or neo-colonial bullying is occasionally debated.

The relative obscurity of JIS X 0212, the tendency to just use Unicode 3.1 instead of JIS X 0213, and the deficiencies of JIS X 0208 mean that at the moment, when a system is said to use "JIS", a proprietary extention of JIS X 0208 is often what is meant.

Around the mid 90s the general anger of the Japanese IT and typography community seems to have shifted away from JIS and toward Unicode, which perhaps reflects the increasing irrelevance of new JIS standards. The accusations made against Unicode often seem like a slightly expanded version of those made toward JIS (i.e. issues relating to the unification of Japanese with non-Japanese Han characters were added).

Personal Names

Originally, the Jinmei list was determined without reference to any other standard list of kanji, and had no particular relationship to any other list. However, by 1983 it was a subset of the JIS X 0208 character set.

The original Jinmeiyou list was widely considered to be too narrow and it has been expanded at various points in the past. This reflects not only the discovery of various reasonable names that are not on the list, but also a gradual erosion of the language-reforming zeal of the Japanese government.

In the 2001, after the list had remained static for some years, a minor squabble over a curious name gradually developed into a reform movement. At issue was whether the kanji U+8235, 'rudder', could form part of a name. The Ministry eventually decided on a new revision of the Jinmei list, more extensive than any before, and made the following statement:

It is expected that the Jinmei kanji will at least double, and if possible increase to about the level of 1,000. Informal and incorrect characters will, as before, be forbidden but we will base our decisions on the following attitude: Except for very difficult kanji, everything that is requested should be allowed.

Since this liberalization, hundreds of new name kanji have been suggested ('strawberry' seems to be a common one). Most of these are already JIS kanji but some are not and there has been speculation -- just speculation -- that a new Joyo revision and a new JIS revision may be necessary if the Ministry accepts exotic kanji variants.

• Han Character Unification
• Round-Trip Compatibility with existing character sets

What is Han Character Unification?

You can unify the character repertoire, in which case characters from existing sets are examined, and if two or more are felt to represent the same abstract 'real life' character, they are represented with only one character in the new set.

You can not unify the character set, in which case every single character that you take from a source set becomes one distinct character in the set you build.

A non-unified character set is easy to build: you just add all the characters from existing sets together to make your new set. In practise, though, this results in a very large set with a lot of duplicates. For instance, if you were making a character set for Europe, you wouldn't include a capital 'A' for every single European language; instead you would unify the 'A's found in the various national character sets into a single 'A'.

Although unification is important in all areas of Unicode, it is most important with Han characters (i.e. chinese-derived ideograms such as Japanese kanji) because their sheer number means that it is impossible to keep a separate set for all the languages that use them. Han characters can have any of the following forms:

• Traditional Chinese, as used in Taiwan and in older documents elsewhere.
• Simplified Chinese: thousands of common Han characters have simplified forms used in mainland China.
• Japanese Kanji: Japanese uses thousands of Han characters, often with forms subtly different from the Chinese original.
• Korean Hanja: Korean uses a limited number of Han characters on a day-to-day basis, and a much larger number in historical/literart contexts. Again, the Korean form often differs slightly from the Chinese original.
• Vietnamese Chu Han: Vietnamese was once written with Chinese characters and special forms evolved for this purpose can still be required.
• Historical Forms:
Chinese 'seal script' and Japanese cursive calligraphy are two examples of situations where particular variants of a character, not part of the common language, are often used. Most people would say that these 'artistic' variants are not separate characters, in the same way that a complicated medieval manuscript 'a' is not a different character from a regular printed 'a'.

Ideally, if unification were perfect, Unicode would provide a single character, having a single code point, and the display system would come up with an appropriate Traditional Chinese, Simplified Chinese, Japanese, Korean, or Vietnamese glyph to represent it. However, unification has not been perfect...

Issues With Han Character Unification

When Unification is over-enthusiastic, and variants that are really thought of as separate are assigned the same code point, an unfortunate sequence of events tends to take place:

• Considering, for instance, the Han character for 'bone', Unicode scholars decide that there is one basic character involved, and assign it a code point, U+9AA8
• Users are aghast to find that their national variant of the character is not displayed on their screens, because there is no way to specify the particular variant of the character needed (and their font usually just displays a variant close to the reference glyph provided by Unicode).
• Hair-pulling ensues and new characters are assigned for local variants; in this case U+3947 for Simplified Chinese, U+586C for Traditional Chinese, U+397C for Japan and U+4D69 for Korea. In some cases this step happens only long afterward and the local variant has to be represented with surrogate code points. Note: This is not a special problem just for Asian characters. It is analogous to what Unicode does for ASCII: punctuation marks that have local standard forms are present as the 'general' ASCII form and as 'specific' forms, e.g. the hyphen (U+002D) which is specialized in U+2010 'hyphen', U+2011 'nonbreaking hyphen', U+2013 'en dash' and so on.
• The unlucky creator of text now has to decide whether they mean 'bone' in general or whether they mean 'bone, the Japanese version of it in particular', bearing in mind that the display system may or may not know what to do with the regional versions. Characters now exist whose range of meanings overlap or contain each other. This is inelegant and creates work.

Often, people who favor 'aggressive' unification of the sort that produced U+9AA8 argue that there is just one character involved and that what is annoying the users is just a glyph difference, which should be sorted out at the display level. Unicode was generally built on the following paradigm:

One Character has many Glyphs

One Character has one or more Variants, which have many Glyphs

A point worth noting is that sometimes, a pair of variants are interchangeable in one context but considered different in another context. For instance, the characters U+9AD8 and U+9AD9 would be considered interchangeable by all but the most finicky when used in Japanese text, but when used in a family name the difference is taken seriously.

In general, it is by no means easy to judge when a difference is between two characters, or between two variants of one character, or between two visual representations of one variant, and there is nowhere you can draw the line without irritating someone somewhere. Unicode has often drawn the line very aggressively (relegating entites that are very different in some situations to the status of mere glyphs), and then redrawn it later, and the result can be confusing for particular characters.

Mojikyo

Unicode is a Western standard and therefore less appropriate than Japanese standards

What is now usually just called 'Unicode' was developed in parallel by two groups; the ISO and the Unicode Consortium, the latter being (at the time) a group of mostly American industrial interests. Since the early 80's, the ISO had been discussing the possibility of an international character set. It proved difficult to decide whether Han characters in this set should be unified or not, although this was just one among many problems faced by ISO/IEC JTC1/SC2 (the snappily-named committee in question). By 1990, there was concern in the ISO about duplication of effort between the Unicode Consortium and JTC1/SC2, and a majority of SC2 voted (in Seoul, which thus became the Jerusalem of han unification) to coordinate ISO standards and work with the Unicode Consortium. Since then, the Unicode standard has been identical with the ISO's ISO10646 standard and the terms are usually used interchangeably.

People who are generally positive about Unicode like to point out that the Han unification effort in Unicode evolved from the efforts of a group called CJK-JRG (China, Japan, Korea Joint Research Group), formed in 1990 from the original JTC1/SC2, which met in Japan, Korea and China and was composed of representatives from those countries -- and was thus fully in the hands of han character users. On the other hand, Japanese people who oppose Unicode like to point out that the CJK-JRG actually favored the DIS 10646 proposal, a proposal for a non unified character set, which was thrown out in favor of unification with the Unicode Consortium's unified character set by the votes of American and European ISO members. More pro-Unicode people, however, like to point out that within the CJK-JRG, the Japanese voted against DIS 10646 because they aparrently preferred a unified character set. Those positive about Unicode, again, will mention that the editor of 10646, who favored the fusion between 10646 and Unicode that was eventually adopted, was in fact Japanese. In reply, anti-Unicode people will bitterly state that he was an employee of DEC Japan, a strongly pro-Unicode and pro-unification faction, and therefore not a free agent. In other words, there are plenty of facts to support either side.

Some claim that the work of unification was carried out by Americans unfamiliar with Han characters. This is a fabrication, since the actual unification was done by the CJK-JRG, as noted above. However, some claim (and this time with a foundation in reality) that in some cases the CJK-JRG was bullied or circumvented by the Unicode Consortium; for instance it is said that IBM managed to insert 32 characters from its own character set into the standard even after the CJK-JRG rejected them.

As these bitter references to DEC and IBM suggest, the original participants in the Unicode Consortium included many large US-based companies, whose expected return on the creation of Unicode was easier access to worldwide markets. The fusing of the ISO efforts with those of the Unicode Consortium, therefore, looked to some non-US observers like the selling out of international standards to US commercial interests. The issue of Han unification became identified with the idea of American corporations forcing globalization on a vulnerable Asian market, to an extent that often obscured academic and technical arguments (and, in many cases, still does).

With the release of JIS X 0213, a related issue appeared -- JIS X 0213 allocated separate characters for the Ainu language (the language of the pre-Japanese inhabitants of Japan), but Unicode preferred to represent them with combining characters, which has little impact on actual information processing but quite a lot on the people's feelings. Supporters of the decision would argue that the characters in question really are combined forms rather than unique signs with their own identity, and that Ainu was handled in the same way as many other non-ideographic Asian languages. Others would argue that the decision was railroaded through as the result of American companies refusing to use JIS X 0213 until it could be made compatible with Unicode.

Frequently Asked Questions

Are all Japanese characters in common use included in Unicode?

• Unicode is green
• JIS is red
• Other Japanese Government standards are orange
• Everything else is black

• 1670: K'ang Hsi dictionary compiled -- the most important attempt to standardize kanji. The forms in this dictionary were standard for centuries.
• 1946: Toyo kanji list defined.
• 1951: Jinmei kanji list defined (92 kanji).
• 1958: Gakushuu kanji list defined.
• 1969: JIS C 6220 (Japanese ASCII) was defined.
• 1976: Jinmei kanji list revised.
• 1978: JIS C 6226 (Basic kanji) was defined.
• 1983: JIS C 6226 was revised and heavily corrected.
• 1981: Jinmei kanji list revised.
• 1981: Joyo kanji list defined.
• 1983: Shift-JIS developed.
• 1984: TRON project started.
• 1987: JIS C 6220 was renamed to JIS X 0201.
• 1987: JIS C 6226 was renamed to JIS X 0208 and revised heavily.
• 1988: TRON association founded and begins to popularise TRON.
• 1990: JIS X 0208 was revised.
• 1990: JIS X 0212 (auxiliary kanji) was defined.
• 1990: Jinmei (name) kanji list reaches its current form (Joyo + 285 kanji).
• 1991: Unicode Consortium founded, Unicode 1.0 published.
• 1993: Unicode Consortium and ISO publish Unicode 1.1.
• 1995: JIS X 0221 (JIS Unicode) was defined.
• 1996: Unicode 2.0 published, the first useful (and used) version of Unicode.
• 1997: Mojikyo institute begins to compile the Mojikyo character set.
• 1997: JIS X 0208 was revised again.
• 1997: Jinmei kanji list has one character added.
• 1997: JIS Kanji Jiten, a definitive resource on the JIS kanji, published.
• 1998: Unicode 2.1 published.
• 2000: JIS X 0213 (more auxiliary kanji) was defined.
• 2000: Unicode 3.0 published.
• 2001: Unicode 3.1 has round-trip compatibility with JIS X 0213.
• 2002: Unicode 3.2 published, adding many Han characters.
• 2002: Revised JIS Kanji Jiten published.
• 2002: Expansion of Jinmei list announced.
• 2003: Unicode 4.0 published, adding many historical Han characters.

Appendix B: Glossary

Jikei

Jitai

JIS X 0213, which defines various terms, defines Jitai rather differently: The abstract ideal of the shape of a graphical representation of a graphical character. (my rough translation).

It is possible for displayed kanji to differ from each other by jitai only (a pair of variants displayed in the same font), by jikei only (one variant, displayed in Mincho and in gothic style) or by both.

Dezain

Mojizukei

Itaiji

Moji

Defined in JIS X 0213 as One unit in a collection of elements used in the organization, control and representation of data. My translation is unreliable as usual, but it's interesting that the writers don't use any text or language related terms in the definition -- it's seen as a data entity, not a language entity.

Zukeimoji

Ryakuji, Zokuji