Why does unicode to string only work with try/except?

Question

Just when I thought I had my head wrapped around converting unicode to strings Python 2.7 throws an exception.

The code below loops over a number of accented characters and converts them to their non-accented equivalents. I've put in an special case for the double s.

#!/usr/bin/python
# -*- coding: utf-8 -*-
import unicodedata

def unicodeToString(uni):
  return unicodedata.normalize("NFD", uni).encode("ascii", "ignore")

accentList = [
#(grave accent)
u"à",
u"è",
u"ì",
u"ò",
u"ù",
u"À",
u"È",
u"Ì",
u"Ò",
u"Ù",

#(acute accent)
u"á",
u"é",
u"í",
u"ó",
u"ú",
u"ý",
u"Á",
u"É",
u"Í",
u"Ó",
u"Ú",
u"Ý",

#(arrete accent) 
u"â",
u"ê",
u"î",
u"ô",
u"û",
u"Â",
u"Ê",
u"Î",
u"Ô",
u"Û",

#(tilde )
u"ã",
u"ñ",
u"õ",
u"Ã",
u"Ñ",
u"Õ",

#(diaresses)
u"ä",
u"ë",
u"ï",
u"ö",
u"ü",
u"ÿ",
u"Ä",
u"Ë",
u"Ï",
u"Ö",
u"Ü",
u"Ÿ",

#ring 
u"å",
u"Å",

#ae ligature
u"æ",
u"Æ", 

#oe ligature
u"œ",
u"Œ",

#c cidilla
u"ç",
u"Ç",

# D stroke?
u"ð",
u"Ð",

# o slash
u"ø",
u"Ø",

u"¿", # Spanish ?
u"¡", # Spanish !
u"ß"  # Double s
]

for i in range(0, len(accentList)):
  try:
    u = accentList[i]
    s = unicodeToString(u)
    if u == u"ß":
      s = "ss"
    print("%s -> %s" % (u, s))
  except:
    pass

Without the try/except I get an error:

File "C:\Python27\lib\encodings\cp437.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode character u'\xc0' in  position 0
: character maps to <undefined>

Is there anything I can do to make the code run without using the try/except? I'm using Sublime Text 2.

From a script, in a folder on d drive. Running Python 2.7, Windows XP box. — Ghoul Fool, Sep 06 '15 at 14:19
Are you using a cmd prompt to run it? I am not overly familiar with windows but I think cp437 is not the correct codepage — Padraic Cunningham, Sep 06 '15 at 14:20
Sublime needs to know of which encoding your generated data is. Try to add `LANG ` variable to your sublime-build settings. http://stackoverflow.com/questions/15166076/sublime-text-2-encoding-error-with-python3-build — HelloWorld, Sep 06 '15 at 14:42
@Padraic Cunningham yes, running the script from a Command Prompt window. — Ghoul Fool, Sep 06 '15 at 15:01
There is also a lib that will do what you want https://pypi.python.org/pypi/Unidecode — Padraic Cunningham, Sep 06 '15 at 17:25
related: [What's the fastest way to strip and replace a document of high unicode characters using Python?](http://stackoverflow.com/q/2854230/4279) — jfs, Sep 07 '15 at 04:17
please, limit your questions to a single issue e.g., "how to save the Python source code as utf-8 in Sublime Text 2?" should be asked as a separate question. — jfs, Sep 07 '15 at 04:18
The `coding:` spec needs to be the first or second line of the file. If you have a blank line at the top, it won't work. — Mark Ransom, Sep 08 '15 at 19:26
Coding spec? I'm new at this, I thought the shebang line had to be first? — Ghoul Fool, Sep 08 '15 at 19:28
Yes, the shebang has to be first - but it's optional, you don't need it unless you want to execute the script via the command line on a *nix system without starting Python manually. Given that you're running this on Windows, you don't need it (although it shouldn't hurt). — Mark Ransom, Sep 08 '15 at 20:20

score 2 · Accepted Answer · edited May 23 '17 at 12:09

2

try/except does not make Unicode work. It just hides errors.

To fix the UnicodeEncodeError error, drop try/except and see Python, Unicode, and the Windows console.

edited May 23 '17 at 12:09

Community

1
1

answered Sep 07 '15 at 04:15

jfs

346,887
152
868
1,518

Why does unicode to string only work with try/except?

1 Answers1