Dochula Pass, Bhutan

Picture of the page in action.

An updated version of the Unicode Character Converter web app is now available. This app allows you to convert characters between various different formats and notations.

Significant changes include the following:

  • It’s now possible to generate EcmaScript6 style escapes for supplementary characters in the JavaScript output field, eg. \u{10398} rather than \uD800\uDF98.
  • In many cases, clicking on a checkbox option now applies the change straight away if there is content in the associated output field. (There are 4 output fields where this doesn’t happen because we aren’t dealing with escapes and there are problems with spaces and delimiters.)
  • By default, the JavaScript output no longer escapes the ASCII characters that can be represented by \n, \r, \t, \’ and \”. A new checkbox is provided to force those transformations if needed. This should make the JS transform much more useful for general conversions.
  • The code to transform to HTML/XML can now replace RLI, LRI, FSI and PDI if the Convert bidi controls to HTML markup option is set.
  • The code to transform to HTML/XML can convert many more invisible or ambiguous characters to escapes if the Escape invisible characters option is set.
  • UTF-16 code points are all at least 4 digits long.
  • Fixed a bug related to U+00A0 when converting to HTML/XML.
  • The order of the output fields was changed, and various small improvements were made to the user interface.
  • Revamped and updated the notes

Many thanks to the people who wrote in with suggestions.

Picture of the page in action.
>> Use the app

This app allows you to see how Unicode characters are represented as bytes in various legacy encodings, and vice versa. You can customise the encodings you want to experiment with by clicking on change encodings shown. The default selection excludes most of the single-byte encodings.

The app provides a way of detecting the likely encoding of a sequence of bytes if you have no context, and also allows you to see which encodings support specific characters. The list of encodings is limited to those described for use on the Web by the Encoding specification.

The algorithms used are based on those described in the Encoding specification, and thus describe the behaviour you can expect from web browsers. The transforms may not be the same as for other conversion tools. (In some cases the browsers may also produce a different result than shown here, while the implementation of the spec proceeds. See the tests.)

Encoding algorithms convert Unicode characters to sequences of double-digit hex numbers that represent the bytes found in the target character encoding. A character that cannot be handled by an encoder will be represented as a decimal HTML character escape.

Decoding algorithms take the byte codes just mentioned and convert them to Unicode characters. The algorithm returns replacement characters where it is unable to map a given byte to the encoding.

For the decoder input you can provide a string of hex numbers separated by space or by percent signs.

Green backgrounds appear behind sequences where all characters or bytes were successfully mapped to a character in the given encoding. Beware, however, that the character mapped to may not be the one you expect – especially in the single byte encodings.

To identify characters and look up information about them you will find UniView extremely useful. You can paste Unicode characters into the UniView Edit Buffer and click on the down-arrow icon below to find out what they are. (Click on the name that appears for more detailed information.) It is particularly useful for identifying escaped characters. Copy the escape(s) to the Find input area on UniView and click on Dec just below.

It has always been possible to pass a string to the converter in the URI, but that was never documented.

Now it is, and you can pass a string using the q parameter. For example, http://rishida.net/tools/conversion/?q=Crêpes. You can also pass a string with escapes in it, but you will need to be especially careful to percent escape characters such as &, + and # which affect the URI syntax. For example, http://rishida.net/tools/conversion/?q=CrU%2B00EApes.

Analyser: http://rishida.net/tools/analysestring/

Converter: http://rishida.net/tools/conversion/

The string analyser tool provides information about the characters in a string. One difference in this version is a new section “Data input as graphics”, where you see a horizontal sequence of graphics for each of the characters in the string you are analysing. This can be useful to get a screen snap of the characters. Of course, there is no combining or ligaturing behaviour involved – just a graphic per character.

You can reverse the character order for right-to-left scripts.

Another difference is that you can explode example text in the notes. Take this example: if you click on the Arabic word for Koran (red word near the bottom of the notes), you’ll see a pop-up window in the bottom right corner of the window that lists the characters in that word.

The other change is that the former “Related links” section in the sidebar is now called “Do more”, and the links carry the string you are analysing to the Converter or UniView apps.

Oh, and the page now remembers the options you set between refreshes, which makes life much easier.

The converter tool converts between characters and various escaped character formats. It was changed so that the “View names” button sends the characters to the string analyser tool. This means that you’ll now see graphics for the characters, and that, once on the string analyser page, you can change the amount of information displayed for each character (including showing font-based characters, if you need to).

I also fixed a bug related to the UTF-8 and UTF-16 input. Including spaces after the code values no longer fires off a bug.

PS: The string analyser tool has graphics for all new Unicode 6.0 characters, however I haven’t updated the data for those characters yet. I was planning to do so with the next release of UniView, which should be in October, when the final Unicode database is available.

A new version of this very popular tool is now available, in a new location. Although it is currently labeled ‘beta’, I recommend that you use that instead, and change any links and bookmarks to the new location. There are a number of new features.

There is also a vastly improved code base. If you are one of the many people who have contacted me to ask how I coded the conversions, please take a look at the new javascript code. It is much cleaner and more compact.

New features include:

* New mixed input field and position of some fields changed.
* New field for conversion of 0x… notation hex escapes.
* Enabled invisible and ambiguous characters to be made visible in the XML output.
* Added support for all HTML entities in HTML/XML input.
* All code rewritten to use characters as the internal representation, rather than code points. Also, code is much smaller and cleaner, partly through use of regular expression matching.
* Various filters available for conversion, such as allowing ASCII or Latin1 characters to remain unconverted in NCR output.
* New icon to quickly select all contents of a field.

There is also a new demonstration feature.

If there are no issues raised/remaining in a couple of months, I’ll remove the beta tag.

>> Use it !

Picture of the page in action.

This web-based tool helps you convert between a number of Unicode escape and code formats.

Changes in the new version:

  • Convert from JavaScript, Java and C escape notation, and to JavaScript/Java escapes (with switch to show C-style supplementary characters)
  • Convert to and from CSS escape notation
  • Convert from HTML/XML code with escapes to code with just characters
  • Convert < > ” or & in HTML/XML code to entities
  • Option to show ASCII characters when converting to NCRs
  • View a set of characters in UniView by clicking on the View in UniView button

For CSS output I chose the 6-figure version with no optional space, since I thought it was clearest. I’ve had a request to change it to the shortest form (4 or 6 figures) followed by space. If other people prefer that, I may change it.

Update: Markus Scherer convinced me to change the CSS output. So rather than 6-figure escapes with no space, the output now contains 6-figure escapes followed by a space for supplementary characters, and 4-figure escapes followed by a space elsewhere.

>> Use it !

Picture of the page in action.

This web-based tool helps you convert between Unicode character numbers, characters, UTF-8 and UTF-16 code units in hex, percent escapes, Unicode U+hex notation, and Numeric Character References (hex and decimal).

Changes in the new version:

  • Convert to and from Unicode U+hex notation
  • Get a list of Unicode names for a sequence of characters by clicking on the View Names button
  • You now have to click a button to start the conversion, rather than remove focus from the input area. This provides better control and a more intuitive approach.

It also allows you to separate a sequence of characters by spaces. Paste the characters into the Characters field and click Convert. Then click Convert immediately in the Unicode U+hex notation field. (The latter field is the only one that changes the data after an initial conversion.)

Start the app

This dynamic HTML app helps you convert between Unicode character numbers, characters, UTF-8 and UTF-16 code units in hex, percent escapes, and Numeric Character References (hex and decimal).

This new version adds some useful things:

  • You can now convert to and from percent escaped forms. When converting to percent escapes, characters allowed in URI syntax are not converted. When converting from percent escapes you can only use characters allowed in URIs.
  • You can also now convert from a mixture of characters and escapes in the bottom two fields.