Python and Unicode Panjabi (ਪੰਜਾਬੀ)

Documenting the weird ways of Python 2.7 programming language when dealing with Unicode characters. Examples in Panjabi text. Python 3 is a liberator, but still far away. Come follow along.

    $ # Let's bhangra!

    $  unicode('panjab')
    >  u'panjab'

    ## $  unicode('ਪੰਜਾਬ')

    UnicodeDecodeError Traceback (most recent call last)
    /Users/askang/Desktop/<ipython-input-2-ef57dc919a30> in <module>()
    ----> 1 unicode('ਪੰਜਾਬ')

    UnicodeDecodeError: 'ascii' codec can't decode byte 0xe0 in position 0: ordinal not in range(128)

    $  unicode(u'ਪੰਜਾਬ')
    >  u'\u0a2a\u0a70\u0a1c\u0a3e\u0a2c'

    $  # String vs Unicode

    $  s = 'punjab'

    $  type(s)
    >  str

    $  s = u'punjab'

    $  s
    >  u'punjab'

    $  type(s)
    >  unicode

    >  # Western Bhangra Fusion

    $  s = u'panjab ਪੰਜਾਬ'

    $  type(s)
    >  unicode

    $  s = u'panjab ਪੰਜਾਬ ਇੰਟਰਨੈਸ਼ਨਲ'

    $  s
    >  u'panjab \u0a2a\u0a70\u0a1c\u0a3e\u0a2c \u0a07\u0a70\u0a1f\u0a30\u0a28\u0a48\u0a36\u0a28\u0a32'

    $  type(s)
    >  unicode

    $  # Thus Python 2.\* stores all non-ascii characters internally as code point representations. Glyphs are someone else's headache.

    $  s.count('a')
    >  2

    $  s.count(u'a')
    >  2

    $  s.count(u'ੲ')
    >  0

    $  s.count(u'ਇ')
    >  1

    $  s.count(u'ਇੰ')
    >  1

    $  # ੲ (GURMUKHI IRI \u0A72) and ਇ (GURMUKHI LETTER I \u 0A07). Consonent vs Vowel. In my opinion poor choice because we are going to run into issues when mining data.

    $  unichr(38677)
    >  u'\u9715'

    $  ord(u'\u9715')
    >  38677

    In [25] # ord() returns entity number for unicode hexadecimal, starting at zero for the first element of unicode table (unassigned)

    $  s = u'punjab'

    $  s = u'punjab ਪੰਜਾਬ'

    $  type(s)
    >  unicode

    $  s
    >  u'punjab \u0a2a\u0a70\u0a1c\u0a3e\u0a2c'

    $  s.encode('utf-8')
    >  'punjab \xe0\xa8\xaa\xe0\xa9\xb0\xe0\xa8\x9c\xe0\xa8\xbe\xe0\xa8\xac'

    $  se = s.encode('utf-8')

    $  type(se)
    >  str

    $  # encode() converts a unicode into a string.

    $  sd = se.decode('utf-8')

    $  sd
    >  u'punjab \u0a2a\u0a70\u0a1c\u0a3e\u0a2c'

    $  type(sd)
    >  unicode

    $  # decode() converts string back into encoding, utf-8 in this case

Unicode literals in source code

Try running the following script -

#!/usr/bin/env python

s = u'Panjab International ਪੰਜਾਬ ਇੰਟਰਨੈਸ਼ਨਲ'
print 'Unicode String: ', s

we get

$ python python_text.py
> File "python_text.py", line 4
> SyntaxError: Non-ASCII character '\xe0' in file python_text.py on line 4,
> but no encoding declared;
> see http://www.python.org/peps/pep-0263.html for details

Python’s default encoding is assumed to be latin-1. Lets declare encoding as utf-8.

#!/usr/bin/env python
# -_- coding: utf-8 -_-
s = u'Panjab International ਪੰਜਾਬ ਇੰਟਰਨੈਸ਼ਨਲ'
print 'Unicode String: ', s

Now lets run this program.

$ python python_text.py
> Unicode String: Panjab International ਪੰਜਾਬ ਇੰਟਰਨੈਸ਼ਨਲ

Code point properties

#!/usr/bin/env python

# -_- coding: utf-8 -_-

import unicodedata

s = u'Panjab International ਪੰਜਾਬ ਇੰਟਰਨੈਸ਼ਨਲ'

print "====================================================="
print "index", '\t', 'ord(c)', '\t', "category", '\t', "name"
print "====================================================="

for index, char in enumerate(s):
print index, '\t', ord(char), '\t', unicodedata.category(char), '\t', '\t', unicodedata.name(char)

returns

\$❯ python python_text.py
=====================================================
index ord(c) category name
=====================================================
0 80 Lu LATIN CAPITAL LETTER P
1 97 Ll LATIN SMALL LETTER A
2 110 Ll LATIN SMALL LETTER N
3 106 Ll LATIN SMALL LETTER J
4 97 Ll LATIN SMALL LETTER A
5 98 Ll LATIN SMALL LETTER B
6 32 Zs SPACE
7 73 Lu LATIN CAPITAL LETTER I
8 110 Ll LATIN SMALL LETTER N
9 116 Ll LATIN SMALL LETTER T
10 101 Ll LATIN SMALL LETTER E
11 114 Ll LATIN SMALL LETTER R
12 110 Ll LATIN SMALL LETTER N
13 97 Ll LATIN SMALL LETTER A
14 116 Ll LATIN SMALL LETTER T
15 105 Ll LATIN SMALL LETTER I
16 111 Ll LATIN SMALL LETTER O
17 110 Ll LATIN SMALL LETTER N
18 97 Ll LATIN SMALL LETTER A
19 108 Ll LATIN SMALL LETTER L
20 32 Zs SPACE
21 2602 Lo GURMUKHI LETTER PA
22 2672 Mn GURMUKHI TIPPI
23 2588 Lo GURMUKHI LETTER JA
24 2622 Mc GURMUKHI VOWEL SIGN AA
25 2604 Lo GURMUKHI LETTER BA
26 32 Zs SPACE
27 2567 Lo GURMUKHI LETTER I
28 2672 Mn GURMUKHI TIPPI
29 2591 Lo GURMUKHI LETTER TTA
30 2608 Lo GURMUKHI LETTER RA
31 2600 Lo GURMUKHI LETTER NA
32 2632 Mn GURMUKHI VOWEL SIGN AI
33 2614 Lo GURMUKHI LETTER SHA
34 2600 Lo GURMUKHI LETTER NA
35 2610 Lo GURMUKHI LETTER LA

Python 3

In Python 3, all strings are sequences of Unicode characters.

(py3)Desktop ❯ python
Python 3.2.3 (default, Jul 18 2012, 15:57:31)
Type \e to get an external editor.

$  s = 'punjab'
$  type(s)
>  <class 'str'>
$  s = "ਪੰਜਾਬ"
$  type(s)
>  <class 'str'>
$  s
>  'ਪੰਜਾਬ'

From Dive into Python 3

In Python 3, all strings are sequences of Unicode characters. There is no such thing as a Python string encoded in UTF-8, or a Python string encoded as CP-1252. “Is this string UTF-8?” is an invalid question. UTF-8 is a way of encoding characters as a sequence of bytes. If you want to take a string and turn it into a sequence of bytes in a particular character encoding, Python 3 can help you with that. If you want to take a sequence of bytes and turn it into a string, Python 3 can help you with that too. Bytes are not characters; bytes are bytes. Characters are an abstraction. A string is a sequence of those abstractions.

Other Resources

Get on my private email list.

Occasional Emails. No Spam.