If you’re interested, I just did a major overhaul of my script notes on Bengali in Unicode. There’s a new section about which characters to use when there are multiple options (eg. RRA vs. DDA+nukta), and the page provides information about more characters from the Bengali block in Unicode (including those used in Bengali’s amazingly complicated currency notation prior to 1957).

In addition, this has all been squeezed into the latest look and feel for script notes pages.

The new page is at a new location. There is a redirect on the old page.

Hope it’s useful.

>> Read it

I finally got to the point, after many long early morning hours, where I felt I could remove the ‘Draft’ from the heading of my Myanmar (Burmese) script notes.

This page is the result of my explorations into how the Myanmar script is used for the Burmese language in the context of the Unicode Myanmar block. It takes into account the significant changes introduced in Unicode version 5.1 in April of this year.

Btw, if you have JavaScript running you can get a list of characters in the examples by mousing over them. If you don’t have JS, you can link to the same information.

There’s also a PDF version, if you don’t want to install the (free) fonts pointed to for the examples.

Here is a summary of the script:

So, what next. I’m quite keen to get to Mongolian. That looks really complicated. But I’ve been telling myself for a while that I ought to look at Malayalam or Tamil, so I think I’ll try Malayalam.

The page design helps give the impression – there are no clunky boxes around the video itself – but there’s also no need to view in a different area, or switch to another tool, or even wait for a download to get started – it’s just there as part of the page, but a part that moves and produces sound. Kind of like the moving paper in Harry Potter’s world.

It’s great how technology marches on sometimes.

[Update: Since I wrote the above the video has acquired grey panels around the edges for controls, which I think is a shame. It's still pretty good technology though. ]

For example, I was in a discussion last week about XML, and the problems caused by the fact that XML 1.0 is currently tied to a specific version of Unicode, and a very old version at that (2.0). This affects what characters you can use for things such as element and attribute names, enumerated lists for attribute values, and ids. Note that I’m not talking about the content, just those names.

I spoke about this at a W3C Technical Plenary some time back in terms of how this bars people from using certain aspects of XML applications in their own language if they use scripts that have been added to Unicode since version 2.0. This includes over 150 million people speaking languages written with Ethiopic, Canadian Syllabics, Khmer, Sinhala, Mongolian, Yi, Philippine, New Tai Lue, Buginese, Cherokee, Syloti Nagri, N’Ko, Tifinagh and other scripts.

This means, for example, that if your language is written with one of these scripts, and you write some XHTML that you want to be valid (so you can use it with AJAX or XSLT, etc.), you can’t use the same language for an id attribute value as for the content of your page. (Try validating this page now. The previous link used some Ethiopic for the name and id attribute values.)

But there’s another issue that hasn’t received so much press – and yet I think, in it’s own way, it can be just as problematic. Scripts that were supported by Unicode 2.0 have not stood still, and additional characters are being added to such scripts with every new Unicode release. In some cases these characters will see very general use. Take for example, the Bengali character U+09CE BENGALI LETTER KHANDA TA.

With the release of Unicode 4.1 this character was added to the standard, with a clear admonition that it should in future be used in text, rather than the workaround people had been using previously.

This is not a rarely used character. It is a common part of the alphabet. Put Bengali in a link and you’re generally ok. Include a khanda ta letter in it, though, and you’re in trouble. It’s as if English speakers could use any word in an id, as long as it didn’t have a ‘q’ in it. It’s a recipe for confusion and frustration.

Similar, but much more far reaching, changes will be introduced to the Myanmar script (used for Burmese) in the upcoming version 5.1. Unlike the khanda ta, these changes will affect almost every word. So if your application or protocol froze its Unicode support to a version between 3.0 and 5.0, like IDNA, you will suddenly be disenfranchising Burmese users who had been perfectly happy until now.

Here are a few more examples (provided by Ken Whistler) of characters added to Unicode after the initial script adoption that will raise eyebrows for people who speak the relevant language:

• 01F6 LATIN SMALL LETTER N WITH GRAVE: shows up in NFC pinyin data for Chinese.
• 0219 LATIN SMALL LETTER S WITH COMMA BELOW: Romanian data.
• 0450 CYRILLIC SMALL LETTER IE WITH GRAVE: Macedonian in NFC.
• 0653..0655 Arabic combining maddah and hamza: Implicated in NFC normalization of common Arabic letters now.
• 0972 DEVANAGARI LETTER CANDRA A: Marathi.
• 097B DEVANAGARI LETTER GGA: Sindhi.
• 0B35 ORIYA LETTER VA: Oriya.
• 0BB6 TAMIL LETTER SHA: Needed to spell sri.
• 0D7A..0D7F Malayalam chillu letters: Those will be ubiquitous in Malayalam data, post Unicode 5.1.
• and a bunch of Chinese additions.

So the moral is this: decouple your application, protocol or specification from a specific version of the Unicode Standard. Allow new characters to be used by people as they come along, and users all around the world will thank you.

I was travelling from Vienna to Budapest (probably about 25 years ago) and I had decided to go through Sopron, rather than Hegyeshalom, so I could see something a little more off the beaten track. This was a month-long InterRail trip, so I was able to follow my whim and jump on whatever train I wanted. The train connections worked, and I found myself heading south from Vienna.

Eventually, the train passed into Hungary and stopped. I needed a visa, so I got off with a bunch of other (Hungarian looking) people, and traipsed over to a small outbuilding, where I found myself at the back of a queue of people jostling bags of various sizes and dressed and coiffed in what looked to me to be a very Eastern European fashion. Looking out of the window, everything was grey. I could see rail tracks and points and small, grey buildings but also several very tall towers with machine gun nests perched on top (quite large looking machine guns). The queue moved slowly, and I was surprised at one point to see my train pulling away and disappearing. It seemed a bit odd (and I was glad I’d brought all my stuff with me), but I figured this was probably normal, and I’d just have to catch another train.

I finally arrived at the desk and asked for a visa. The guy behind the desk started talking to me in a somewhat animated fashion, but I had no idea what he was saying. I hadn’t learned German yet, and Hungarian was completely incomprehensible to me. I kept trying to explain, politely, in English, that I needed a visa. Finally, he gave me an exasperated look and called someone out of a nearby room. The guy who emerged was huge, bald and intimidatingly business-like. (Some time later I saw the film Midnight Express, and realised that the prison guard and he could have been the same person.) He shouted at me “Nicht visa!”. And I tried to explain, in English, that, yes, I had no visa, but would like to obtain one, please. This didn’t appear to get across clearly, because he simply repeated “Nicht visa!!” several times, increasing in volume.

Finally, the tension broke and gave way to action. He motioned for me to follow him out of the building, and we started walking away across a couple of sets of railway tracks. I noticed, feeling slightly less at ease but still hopeful, that I was flanked by a soldier with a gun on either side. They weren’t exactly giving me encouraging looks, and as I glanced up at the machine gun towers and at the surrounding barbed wire, I began to wish I knew what was happening.

Soon we arrived at the end of a short train. The very last carriage of this train looked like something you’d expect to see in a Wild West film. It had a kind of standing area at each end with a railing, a door into the carriage and steps leading down to the ground on either side. I was ushered up one set of steps and into what turned out to be an empty carriage. The door was shut behind me, and within a minute or so, as I remember it, the train started moving off, in the same direction my earlier train had disappeared. So I wasn’t just being sent back across the border.

That last realisation started to trouble me a little, since I still had no visa and no idea what was happening. It didn’t help that there was a small round window in the door at each end of the carriage, through which I could see guards sitting on the steps at each corner, all holding machine guns at the ready. As the towers slid away behind us, night started to fall.

Twenty five years has dulled the memory of some of what happened next, but eventually I got off at a small station, having reached the end of the line. The guards were gone, and the station turned out to be quite modern and clean looking. I still couldn’t understand anything anyone was saying, so I still had no idea where I was, but I was able figure out that I was somehow back in Austria. It was much later that I was to realise that Sopron is on a peninsular that sticks into Austria, and I had come in one side and been sent out the other.

I slept that night on the floor of the main station building, and the next morning set off to find someone who spoke English and could tell me where I was – and just as importantly how to get into Hungary. The town was quite small, maybe just a village. In spite of that it took me a while, but I eventually came across a chap in a supermarket who was able to explain to me that visas are not issued on entry into Hungary by train via Sopron. I was ahead of him there. He also offered to drive me to the border, telling me that I would be able to get a visa at the road entry point.

It’s nice to think about that person whenever I relive this story. He really went out of his way, leaving work to assist a complete stranger, with no fuss or thought for reward. I wonder whether he remembers me. I doubt it. Of course, these days he may even be reading this blog post…

So it was that, eventually, I got the stamp in my passport that I needed, and somehow found my way onto another train heading for Budapest. Well, it wasn’t quite the end of the fun. That continued when I tried to meet up with my father in the capital. But that, as they say, is another story…

If you type A0 (the hex code for a non-breaking space) in the Hexadecimal code points field, then press Convert, you will get a blank space in the Characters field that should be U+00A0 NO-BREAK SPACE. Then press Convert or View Names above this Characters field and you’ll find that what was supposed to be a NBSP has changed into an ordinary space. IE7, Opera and Safari all continue to show the character in the field as a NBSP.

(However, all four browsers substitute an ordinary space when you copy and paste the text from the Characters field into something else.)

I tried this with a range of other types of space , but had no such behaviour (try it). They all remained themselves.

Anyone know what this is about?

Monday, November 26, 2007 is the day thousands of Standardistas (people who support web standards) will wear a Blue Beanie to show their support for accessible, semantic, and hopefully internationalized web content.

I haven’t got a blue hat, so I cheated a little by borrowing bits of the cover of Jeffrey Zeldman’s great book, “Designing with Web Standards”. That’s me under the hat though.

(If you’re wondering, the text on the left says the same as the text top right, in Arabic, Urdu, Inuktitut, Simplified Chinese, Traditional Chinese, Khazakh, Greek, Dzonkha, Ethiopian, Hebrew, Hindi, Nepali, Japanese, Korean, Hungarian, Punjabi, Thai and Venda.)

See the Flickr pool.

I’ve been brushing up on the Myanmar script, since major changes are on the way with Unicode 5.1.

I upgraded my myanmar picker to handle the new characters, and I edited my notes on how the script works.

The new characters will make a big difference to how you author text in Unicode, and people will need to update currently existing pages to bring them in line with the new approach. The changes should make it much easier to create content in Burmese, in addition to addressing some niggly problems with making the script work correctly. One reason the changes were sanctioned is that there is currently very little Burmese content out there in Unicode.

I’ll be updating my character by character notes later too.

The only problem with all this is that existing fonts will all need to be changed to support the new world order (or myanmar order). I found one font that is already 5.1 ready from the Myanmar Unicode & NLP Research Center. So if you don’t want to download that font, you’ll need to read the PDF version of my notes on the script.

That would be a pity, however, since i had some fun adding javascript to the article today, so that it displays a breakdown, character by character, of each example as you mouse over it (using images, so you see it properly).

Strč prst skrz krk.

How amazing is that? A whole sentence without vowels! (Means “Put your finger down your throat.” – I’m wondering whether that has something to do with the missing vowels…)

See a video of Jirka pronouncing it.

I received a query from someone asking:

I try to edit lao and thai text with XMetal 5.0, but nothing is displayed but squares. In fact, Unicode characters seems to be correctly saved in the XML file and displayed in Firefox (for example), but i can’t get a correct display in XMetal. Is it a font problem ?

There are two places this needs to be addressed:

1. in the plain text view
2. in the tags-on view

For the plain text view, it is a question of setting a font that shows Lao and Thai (or whatever other language/script you need) in Tools>Options>Plain Text View>Font. You can only set one font at a time, so a wide ranging Unicode font like Arial Unicode MS or Code2000 may be useful for Windows users.

For the tags-on view (which is the view I use most of the time) you need to edit the CSS file that sets the editor’s styling for the DOCTYPE you are working with. This may be in one of a number of places. The one I edit is C:\Program Files\Blast Radius\XMetaL 4.6\Author\Display\xhtml1-transitional.css.

I added the following to mine. I chose fonts I have on my PC and sets font sizes relative to the size I set for my body element. You should, of course, choose your own fonts and sizes.

[lang="am"] { font-family: "Code2000", serif; font-size: 120%; }
[lang="ar"] {font-family: "Traditional Arabic", sans-serif; font-size: 200%; }
[lang="bn"] {font-family: SolaimanLipi, sans-serif; font-size: 200%; }
[lang="dz"] { font-family: "Tibetan Machine Uni", serif; font-size: 140%; }
[lang="he"] {font-family: "Arial Unicode MS", sans-serif; font-size: 120%;}
[lang="hi"] {font-family: Mangal, sans-serif;  font-size: 120%;}
[lang="kk"] {font-family: "Arial Unicode MS", sans-serif;  }
[lang="iu"] {font-family: Pigiarniq, Uqammaq, sans-serif; font-size: 120%; }
[lang="ko"] { font-family: Batang, sans-serif; font-size: 120%;}
[lang="ne"] {font-family: Mangal, sans-serif;  font-size: 120%; }
[lang="pa"] { font-family: Raavi, sans-serif; font-size: 120%;}
[lang="te"] {font-family: Gautami, sans-serif; font-size: 140%;}
[lang="my"] {font-family: Myanmar1, sans-serif; font-size: 200%;}
[lang="th"] {font-family: "Cordia New", sans-serif; font-size: 200%; }
[lang="ur"] { font-family: "Nafees Nastaleeq", serif; font-size: 130%;}
[lang="ve"] { font-family: "Arial Unicode MS", sans-serif; }
[lang="zh-Hans"] { font-family: "Simsun", sans-serif; font-size: 140%; }
[lang="zh-Hant"] { font-family: "Mingliu", sans-serif; font-size: 140%; }

Note that I would have preferred to say :lang(am) { font-family… } etc, but XMetal 4.6 seems to require you to specify the attribute value as shown above. (You also have to specify class selectors as [class="myclass"] {…} rather than .myclass {…}.)