Enabling Unicode Sinhala (සිංහල) in GNU/Linux HOWTO

2.4.1. Sinhala Technical List (in English for developers)

• Archive: http://sourceforge.net/mailarchive/forum.php?forum_name=sinhala-technical

• Subscribe: https://lists.sourceforge.net/lists/listinfo/sinhala-technical

2.4.2. Sinhala Unicode List (in Sinhala for developers and users)

• http://groups.google.com/group/sinhala-unicode

3.1.1. Debian 6.0 (Squeeze) and Above (may work on older versions)

1. As root/superuser run:

   apt-get install ttf-sinhala-lklug ibus im-switch ibus-m17n m17n-db m17n-contrib

2. From your user account (i.e. not root) run:

   rm -f ~/.xinput.d/* ; im-switch -z all_ALL -s ibus

3. Logout and login again. Environment variables need to be set/updated (NO NEED TO REBOOT).

4. From your user account (i.e. not root) select your keyboard layouts by running:

   ibus-setup

In Debian Wheezy IBus may depend on im-config. im-config conflicts with im-switch, so you will need to run im-config instead of im-switch.

3.1.2. Fedora 10 (Cambridge) and Above (may work on older versions)

1. As root/superuser run:

   yum groupinstall sinhala-support

2. From your user account (i.e. not root), select and configure the Input Method by running:

   im-chooser

3.1.3. Ubuntu 9.10 (Karmic) and Above (may work on older versions)

1. The Universe repository should already be enabled (https://wiki.ubuntu.com/AlwaysEnableUniverseMultiverse). If not, first enable the Universe repository.

2. As root/superuser run:

   apt-get install ttf-sinhala-lklug ibus im-switch ibus-m17n m17n-db m17n-contrib language-pack-si-base

3. From your user account (i.e. not root) run:

   rm -f ~/.xinput.d/* ; im-switch -z all_ALL -s ibus

4. Logout and login again. Environment variables need to be set/updated (NO NEED TO REBOOT).

5. From your user account (i.e. not root) select your keyboard layouts by running:

   ibus-setup

3.1.4. How to test

1. Visit http://si.wikipedia.org/ and see if the Sinhala letters render correctly.

2. Copy and paste some of the content from Sinhala Wikipedia to Open Office Writer. Then highlight the Sinhala text and choose the LKLUG font to display them.

3. To test typing, press Control-space whilst you are running a GNOME application. Then select one of the Sinhala input methods.

3.2.1. Fonts

If your distribution does not contain a Sinhala font package, then download a Unicode Sinhala font:

• http://sinhala.sourceforge.net/files/lklug.ttf

If you are using a modern GNU/Linux version and it has fontconfig installed, all you have to do is make a .fonts directory in your home directory:

mkdir ~/.fonts

and copy the True/Open Type font into that directory.

If you want to make the font available to all users of the system, become root and copy the font to:

/usr/share/fonts

In both the above cases, run:

fc-cache -fv

To check which font file provides the Sinhala support, run:

fc-list :lang=si file

Immediately you'll be able to read Unicode Sinhala in these programs (You may have to restart the program.):

• Anything gtk2/gtk3 based

  • evolution

  • gedit

  • gucharmap

  • Firefox/Mozilla (built with gtk2, FreeType2 and Pango support)

If you have Pango 1.8.2 and greater, you will have full SLS1134 Sinhala support. Harfbuzz does not support Sinhala nor Indic scripts well at the moment (29/08/2011).

3.2.2. Input Methods

There are a number of different input method infrastructures available on GNU/Linux. In general, first try IBus/m17n and as a last resort XKB.

To test multi-lingual input methods in gtk2/gtk3 based programs, run:

gedit

To check which input method systems are available, run:

im-switch  -l

To switch between input method systems, run:

im-switch -z all_ALL -c

Generally choose:

ibus

3.2.3. Keyboard Layouts

If you are unfamiliar with the Wijesekera keyboard layout, here are some recommended keyboard layouts. If you download any of the m17n keyboard layout files, copy it to the correct location and relogin in-order to use it.

/usr/share/m17n/si-trans.mim

/usr/share/m17n/si-phonetic-dynamic.mim

/usr/share/X11/xkb/symbols/lk

3.2.4. Character Maps

You can use a Unicode Character Map program to copy and paste Sinhala characters into your program/document. Available programs are:

• gucharmap

3.2.5. Databases

The ability to alphabetically sort words in a database is essential. Till recently databases containing Sinhala words could not be sorted according to the Sinhala sorting order as established by SLS1134 - Part 1: Collation Sequence. Sinhala words can be sorted in MySQL (from version 5.2).

Instead of running this query:

SELECT * FROM table1 ORDER BY column1;

run a slightly modified query that looks like:

SELECT * FROM table1 ORDER BY column1 COLLATE utf8_sinhala_ci;

3.2.6. Locales

The Sinhala locale for Sri Lanka is already included in most distributions. If you wish to view the user interface in Sinhala, then you must first enable the si_LK locale.

/usr/share/i18n/SUPPORTED

/usr/share/i18n/locales/

/etc/locale.gen

# sid_ET UTF-8
si_LK UTF-8
# sk_SK ISO-8859-2 

/usr/sbin/locale-gen

LANG=si_LK.UTF-8 gedit
LANG=fr_FR.UTF-8 gedit

/usr/share/locale/<ISO639 2 character lang code>/LC_MESSAGES/

/etc/default/locale

LC_ALL=si_LK.utf8 sort

3.2.7. Open Office

By default Open Office chooses the first Sinhala font regardless of its completeness. Therefore, you may have to highlight the Sinhala text and select a complete Unicode Sinhala font such as the LKLUG font.

3.2.8. LaTeX

TeX Live, 2009 release, should contain a version of XeTeX with Unicode Sinhala support.

A less preferable option is to download sintex:

• http://www.ucsc.cmb.ac.lk/People/cik/sintex-0.2.0.tar.gz

4.1.1. List of feature tags

• http://partners.adobe.com/public/developer/opentype/index_tag3.html

4.1.2. Glyph Naming

• http://partners.adobe.com/public/developer/en/opentype/aglfn13.txt

• http://partners.adobe.com/public/developer/opentype/index_glyph.html

4.1.3. Indic

• http://www.microsoft.com/typography/SpecificationsOverview.mspx

• Indic FAQ at Unicode.org

• Indic Working Group at Unicode.org

4.1.4. How GNU FreeFont obtained Sinhala glyphs

• http://mail.gnu.org/archive/html/freefont-bugs/2003-03/msg00000.html

• http://mail.gnu.org/archive/html/freefont-bugs/2003-04/msg00001.html

• http://mail.gnu.org/archive/html/freefont-bugs/2003-05/msg00005.html

• http://mail.gnu.org/archive/html/freefont-bugs/2003-08/msg00002.html

• http://lists.gnu.org/archive/html/freefont-bugs/2009-02/msg00000.html

• http://savannah.gnu.org/forum/forum.php?forum_id=6518

4.1.5. Development Info

If you wish to use TeX font glyphs to create a new font, mftrace will be useful. In most circumstances fontforge alone will be sufficient for most of your needs.

4.2.1. Pango

Pango's Indic renderer is based on ICU's Indic renderer.

The original patch to add Sinhala support was created by Harsha Senanayake for ICU [1]and later ported to Pango. The Pango patch was ported to the latest version of Pango by Chamath Keppitiyagama. It was submitted to bugzilla by Anuradha Ratnaweera[2]. Harshula Jayasuriya modified the Pango state table & ZWJ handling [3] & [4].

The Pango code for Sinhala and Indic rendering is common and can be found in the Pango source at:

modules/indic/

One of the most important files to understand is:

modules/indic/indic-ot-class-tables.c

Particularly how the function:

indic_ot_find_syllable()

works.

Next have a look at the file:

modules/indic/indic-ot.c

and the function:

indic_ot_reorder()

4.2.2. ICU

Owen Taylor (Pango) submitted the Pango Sinhala patch to the ICU project [5]. Eric Mader (ICU) ported the Pango patch to ICU and checked-in the changes to ICU 3.6. Then Eric added the state table & ZWJ modifications from Pango to ICU 3.6 [6] & [7].

4.2.3. Harfbuzz

Harfbuzz does not support Sinhala nor Indic scripts well at the moment (29/08/2011). The Indic module (hb-ot-shape-complex-indic.cc) needs a lot more work to be useful.

4.3.1. Debian

Since Debian 4.0 (Etch), Pango is enabled by default.

4.3.2. Ubuntu

/usr/bin/mozilla-firefox 

##
## Set MOZ_ENABLE_PANGO
##
MOZ_ENABLE_PANGO=1
export MOZ_ENABLE_PANGO

/usr/bin/mozilla-firefox 

if [ "x${MOZ_DISABLE_PANGO}" = x ]; then
    if egrep '^(bn|gu|hi|kn|ml|mr|ne|pa|ta|te)_' \
        /var/lib/locales/supported.d/*[^~] >/dev/null 2>&1; then
        MOZ_DISABLE_PANGO=0
    else
        MOZ_DISABLE_PANGO=1
    fi
    export MOZ_DISABLE_PANGO
fi
if [ "x${MOZ_DISABLE_PANGO}" = x0 ]; then
    unset MOZ_DISABLE_PANGO
fi

MOZ_DISABLE_PANGO=0

# MOZ_DISABLE_PANGO=0 mozilla-firefox

4.3.3. Ubuntu 6.10 & Above

Ubuntu 6.10 users can enable Pango in Firefox by setting an environment variable:

MOZ_DISABLE_PANGO=0

Or by simply installing the Ubuntu package:

language-pack-si-base

4.3.4. Fedora Core

Since Fedora Core 4, Pango is enabled in Firefox by default. In order to disable Pango in Firefox an environment variable has to be set:

MOZ_DISABLE_PANGO=1

You can see the difference by running Firefox at the command line like so:

# MOZ_DISABLE_PANGO=1 firefox

Have a look at:

/usr/bin/firefox 

for an explanation.

##
## Set MOZ_ENABLE_PANGO is no longer used because Pango is enabled by default
## you may use MOZ_DISABLE_PANGO=1 to force disabling of pango
##
#MOZ_DISABLE_PANGO=1
#export MOZ_DISABLE_PANGO

##
## In order to better support certain scripts (such as Indic and some CJK 
## scripts), Fedora builds its Firefox, with permission from the Mozilla 
## Corporation, with the Pango system as its text renderer.  This change 
## is known to break rendering of MathML, and may negatively impact 
## performance on some pages.  To disable the use of Pango, set 
## MOZ_DISABLE_PANGO=1 in your environment before launching Firefox.
##
#
# MOZ_DISABLE_PANGO=1
# export MOZ_DISABLE_PANGO
#

4.3.5. Epiphany Browser

Changelog:

2006-01-27  Christian Persch  <chpe at cvs dot gnome dot org>
        * src/ephy-main.c: (main):
        Disable pango rendering by default, unless MOZ_ENABLE_PANGO env
        var is set. Bug #328844.

src/ephy-main.c:

        /* Work around bug #328844, and avoid the gecko+pango performance problem */
        env = g_getenv ("MOZ_ENABLE_PANGO");
        enable_pango = env != NULL &&
                       env[0] != '\0' &&
                       g_ascii_strtoull (env, NULL, 10) != 0;
        if (eel_gconf_get_boolean (CONF_GECKO_ENABLE_PANGO))
        {
                g_print ("NOTE: Enabling gecko pango renderer; this may cause performance degradation.\n"
                         "You can set " CONF_GECKO_ENABLE_PANGO " to \"false\" to disable it.\n");
        }
        else if (!enable_pango)
        {
                g_setenv ("MOZ_DISABLE_PANGO", "1", TRUE);
        }

Epiphany also has a file, data/epiphany-pango.schemas containing a list of locales which require Pango to be enabled by default.

4.4.1. Open Office 2.0.4

Whilst working on the patches for adding Sinhala support to ICU, the renderer of Open Office, I observed that the ZWJ characters do not appear to reach ICU [12]. I demonstrated this at the Red Hat Sri Lanka office (approx. 21/04/2006). Red Hat Sri Lanka then conveyed this to Red Hat India on 27/07/2006. A few days later Red Hat India opened a bug[13]. Then, Caolan McNamara found the Open Office file that filters ZWJ and ZWNJ [14].

The source file:

vcl/source/gdi/sallayout.cxx

contains a function:

inline bool IsControlChar( sal_Unicode cChar )

This function tells a caller that characters U+200B to U+200F are control characters.

In the source file:

linguistic/source/misc.cxx

two functions,

static INT16 GetOrigWordPos( const OUString &rOrigWord, INT16 nPos )

and

INT32 GetPosInWordToCheck( const OUString &rTxt, INT32 nPos )

call

inline bool IsControlChar( sal_Unicode cChar )

when doing lingustic analysis for what appears to be spelling purposes. Even found some comments written in, I assume, German.

In the source file:

vcl/source/gdi/sallayout.cxx

there is a function:

void ImplLayoutArgs::AddRun( int nCharPos0, int nCharPos1, bool bRTL )

which calls the function:

inline bool IsControlChar( sal_Unicode cChar )

it's purpose is to:

// add a run after splitting it up to get rid of control chars

It should be noted that this function handles RTL text in a different way to LTR text. My initial reaction is that should not be the case. However, I have not looked into it any further.

Compiling Open Office 2.0.4 on Debian Etch on a Pentium M 2.13 GHz with 1 GiB RAM took approximately 10 hours and required 10 GBs of additional hard drive space for the source and the compiled files.

4.4.2. Open Office 2.1

Open Office 2.1 does not filter ZWJ, therefore, it supports Unicode Sinhala.

4.5.1. Syllable Segmentation [17]

Sinhala letters which define the start of a new 'syllable':

1. All independent vowels (U+0d85 - U+0d96)

2. Kombuva (U+0dd9) - except if preceded by a kombuva.

3. All consonants (U+0d9a - U+0dc6) - except if preceded by kombuva or kombuva deka (U+0ddb)

4. Kunddaliya (U+0df4)

5. All non-Sinhala characters/codepoints - except ZWJ (U+200D)

4.5.2. Keyboard

You can use showkey in linux to display the scancode.

• 57 - space

• 56 - left alt

• 100 - right alt

• 29 - left ctrl

• 97 - right ctrl

• 42 - left shift

• 54 - right shift

Look in the linux source:

drivers/char/keyboard.c

Look for the function:

getkeycode()

4.5.3. XKB - adding a new keyboard layout

All you need to do is just copy the keyboard layout file into the correct directory:

/etc/X11/xkb/symbols/

or

/etc/X11/xkb/symbols/pc/

or

/usr/share/X11/xkb/symbols

However, for completeness some files in these directories:

/etc/X11/

or

/usr/X11R6/lib/X11/

or

/usr/share/X11/

need to be modified, namely these files:

xkb/rules/{xorg,xfree86}
xkb/rules/{xorg,xfree86}.lst
xkb/rules/{xorg,xfree86}.xml
xkb/symbols.dir

To test a loaded keyboard layout:

setxkbmap -print | xkbcomp -w 10 -xkb - <outfile>

4.5.4. IBus

IBus can be used as the frontend, which is exposed to the user, and the backend that maps keycodes to codepoints. Or IBus can be used as a frontend for other backends. e.g. m17n can be a backend via the ibus-m17n engine.

ibus 1.3.99.20110817 and ibus-m17n 1.3.3 support Surrounding Text Support (STS).

4.5.5. m17n

The m17n backend keyboard layout definition file is a text file. The documentation can be found:

• http://www.m17n.org/common/m17n-docs-en/m17nDBFormat.html#mdbIM

4.5.6. xmodmap

The xmodmap keyboard layout is not fully functional, hence it is recommended you use the X Keyboard Extension keyboard layout. To familiarise yourself with this keyboard layout, read:

• http://www.nongnu.org/sinhala/doc/keymaps/sinhala-keyboard_3.html

1. Download the keyboard layout from:

   • sin.xmodmap

2. Then run xmodmap:

   xmodmap sin.xmodmap

4.5.7. gvim

To familiarise yourself with this keyboard layout, read:

• http://www.nongnu.org/sinhala/doc/keymaps/sinhala-keyboard_3.html

1. Download the keyboard layout and redirector from:

   • sinhala-phonetic_utf-8.vim

   • sinhala.vim

2. Copy the keyboard layout and redirector to ~/.vim/keymap/

3. Start gvim

4. Need to disable the menu so that you can use the 'alt' key:

   set guioptions-=m

5. Select the new keyboard layout, using the redirector, by typing:

   set keymap=sinhala

   or select the new keyboard layout directly by typing:

   set keymap=sinhala-phonetic_utf-8

   To toggle between the Sinhala keyboard layout and the standard ASCII keyboard layout, press <Ctrl> <6> whilst in insert mode.

4.6.1. Collation

The Sinhala letters in the Unicode chart can be categorised as:

• independent vowels: U+0D85 - U+0D96

• consonants: U+0D9A - U+0DC6

• dependent vowels: U+0DCA - U+0DF3

• consonant modifiers: U+0D82 - U+0D83

The collation order of the groups can be broadly described as:

1. independent vowels

2. consonant modifiers

3. consonants

4. dependent vowels.

The order of the groups and the order of the letters within the groups do not correspond to the collation order. Hence, tailoring is required.

4.6.2. Tailoring

• ICU User Guide: Collation Customization

• Change Management for the Unicode Collation Algorithm

/*
  SCCII Part 1 : Collation Sequence (SLS1134)
  2006/11/24
  Harshula Jayasuriya <harshula at gmail dot com>
  Language Technology Research Lab, University of Colombo / ICTA
*/
#if 1
static const char sinhala[]=
    "& \\u0D96 < \\u0D82 < \\u0D83"
    "& \\u0DA5 < \\u0DA4"
    "& \\u0DD8 < \\u0DF2 < \\u0DDF < \\u0DF3"
    "& \\u0DDE < \\u0DCA";
#else
static const char sinhala[]=
    "& \\u0D96 < \\u0D82 < \\u0D83 < \\u0D9A < \\u0D9B < \\u0D9C < \\u0D9D"
              "< \\u0D9E < \\u0D9F < \\u0DA0 < \\u0DA1 < \\u0DA2 < \\u0DA3"
              "< \\u0DA5 < \\u0DA4 < \\u0DA6"
              "< \\u0DA7 < \\u0DA8 < \\u0DA9 < \\u0DAA < \\u0DAB < \\u0DAC"
              "< \\u0DAD < \\u0DAE < \\u0DAF < \\u0DB0 < \\u0DB1"
              "< \\u0DB3 < \\u0DB4 < \\u0DB5 < \\u0DB6 < \\u0DB7 < \\u0DB8"
              "< \\u0DB9 < \\u0DBA < \\u0DBB < \\u0DBD < \\u0DC0 < \\u0DC1"
              "< \\u0DC2 < \\u0DC3 < \\u0DC4 < \\u0DC5 < \\u0DC6"
              "< \\u0DCF"
              "< \\u0DD0 < \\u0DD1 < \\u0DD2 < \\u0DD3 < \\u0DD4 < \\u0DD6"
              "< \\u0DD8 < \\u0DF2 < \\u0DDF < \\u0DF3 < \\u0DD9 < \\u0DDA"
              "< \\u0DDB < \\u0DDC < \\u0DDD < \\u0DDE < \\u0DCA";
#endif

4.6.3. MySQL

• Unicode and Other Funny Characters

• MySQL 5.0 Reference Manual :: 5 Database Administration :: 5.2 mysqld ? The MySQL Server :: 5.2.1 Option and Variable Reference

• MySQL 5.0 Reference Manual :: 10 Character Set Support

• MySQL Internals Manual :: 4 Important Algorithms and Structures :: 4.10 Character/Collation Sets

/*
  Collation language is implemented according to
  subset of ICU Collation Customization (tailorings):
  http://www.icu-project.org/userguide/Collate_Customization.html
  
  Collation language elements:
  Delimiters:
    space   - skipped
  
  <char> :=  A-Z | a-z | \uXXXX
  
  Shift command:
    <shift>  := &       - reset at this letter. 
  
  Diff command:
    <d1> :=  <     - Identifies a primary difference.
    <d2> :=  <<    - Identifies a secondary difference.
    <d3> := <<<    - Idenfifies a tertiary difference.
  
  
  Collation rules:
    <ruleset> :=  <rule>  { <ruleset> }
    
    <rule> :=   <d1>    <string>
              | <d2>    <string>
              | <d3>    <string>
              | <shift> <char>
    
    <string> := <char> [ <string> ]
  An example, Polish collation:
  
    &A < \u0105 <<< \u0104
    &C < \u0107 <<< \u0106
    &E < \u0119 <<< \u0118
    &L < \u0142 <<< \u0141
    &N < \u0144 <<< \u0143
    &O < \u00F3 <<< \u00D3
    &S < \u015B <<< \u015A
    &Z < \u017A <<< \u017B    
*/

typedef struct charset_info_st
{
  uint      number;
  uint      primary_number;
  uint      binary_number;
  uint      state;
  const char *csname;
  const char *name;
  const char *comment;
  const char *tailoring;
  uchar    *ctype;
  uchar    *to_lower;
  uchar    *to_upper;
  uchar    *sort_order;
  uint16   *contractions;
  uint16   **sort_order_big;
  uint16      *tab_to_uni;
  MY_UNI_IDX  *tab_from_uni;
  MY_UNICASE_INFO **caseinfo;
  uchar     *state_map;
  uchar     *ident_map;
  uint      strxfrm_multiply;
  uchar     caseup_multiply;
  uchar     casedn_multiply;
  uint      mbminlen;
  uint      mbmaxlen;
  uint16    min_sort_char;
  uint16    max_sort_char; /* For LIKE optimization */
  uchar     pad_char;
  my_bool   escape_with_backslash_is_dangerous;
  
  MY_CHARSET_HANDLER *cset;
  MY_COLLATION_HANDLER *coll;
  
} CHARSET_INFO;

4.7.1. Collation

To define a collation order for a script, edit:

/usr/share/i18n/locales/iso14651_t1_common

To test the changes, run:

LC_ALL=<LOCALE> sort

4.8.1. GNOME

• http://developer.gnome.org/projects/gtp/l10n-guide/

• http://developer.gnome.org/doc/tutorials/gnome-i18n/translator.html

• http://developer.gnome.org/projects/gtp/teams.html

• http://l10n.gnome.org/languages/si/

4.8.2. KDE

• KDE Localization

• http://l10n.kde.org/team-infos.php?teamcode=si

4.8.3. Debian

• http://www.debian.org/international/l10n/po/si

4.8.4. Fedora

• http://fedoraproject.org/wiki/L10N/Teams/Sinhala

4.8.5. Ubuntu

• https://translations.launchpad.net/ubuntu/jaunty/+lang/si

4.9.1. Debian

• Debian Developers' Corner

• Debian New Maintainers' Guide

• Debian Developer's Reference

• Debian Policy Manual

• The Debian GNU/Linux FAQ

• Debian Library Packaging guide

• Work-Needing and Prospective Packages

• To understand issues involving config.guess and config.sub, refer to /usr/share/doc/autotools-dev/README.Debian.gz .

• In debian/rules, for the target binary-arch, you'll probably have to uncomment dh_install. Or you can just change the DESTDIR of:

  $(MAKE) DESTDIR=$(CURDIR)/debian/tmp install

• pbuilder

• To extract the content of a .deb file:

  ar -x <name>.deb

• To create a Debianized source tree from a Debian source package:

  dpkg-source -x <name>.dsc

• To generate a .deb from within a Debianized source tree:

  dpkg-buildpackage -b -uc -us

• To generate the .dsc and the .diff.gz run:

  dpkg-source -b <dir>

  on the untarred Debianized source tree. Remember to leave the renamed original source tarball of the package in the parent directory.

• Don't forget to sign the source package:

  debsign -m<maintainer> <name>.dsc

• irc.oftc.net, channel #debian-mentors

4.9.2. Ubuntu

• Packaging Guide

• MOTU/Packages/New

• pbuilder HOWTO