Wanted Python to create a UTF-8 File, got an ANSI one. Why?

Question

I have the following function:

def storeTaggedCorpus(corpus, filename):
    corpusFile = codecs.open(filename, mode = 'w', encoding = 'utf-8')
    for token in corpus:
        tagged_token = '/'.join(str for str in token)
        tagged_token = tagged_token.decode('ISO-8859-1')
        tagged_token = tagged_token.encode('utf-8')
        corpusFile.write(tagged_token)
        corpusFile.write(u"\n")
    corpusFile.close()

And when I execute it, I've got the following error:

(...) in storeTaggedCorpus
    corpusFile.write(tagged_token)
  File "c:\Python26\lib\codecs.py", line 691, in write
    return self.writer.write(data)
  File "c:\Python26\lib\codecs.py", line 351, in write
    data, consumed = self.encode(object, self.errors)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)

So i went to debug it, and discovered that the created file was encoded as ANSI, not UTF-8 as declared in corpusFile = codecs.open(filename, mode = 'w', encoding = 'utf-8'). If the corpusFile.write(tagged_token) is removed, this function will (obviously) work, and the file will be encoded as ANSI. If instead I remove tagged_token = tagged_token.encode('utf-8'), it will also work, BUT the resulting file will have encoding "ANSI as UTF-8" (???) and the latin characters will be mangled. Since I'm analizing pt-br text, this is unacceptable.

I believe that everything would work fine if the corpusFile opened as UTF-8, but I can't get it to work. I've searched the Web, but everything I found about Python/Unicode dealt with something else...s So why this file always ends up in ANSI? I am using Python 2.6 in Windows 7 x64, and those file encodings were informed from Notepad++.

Edit — About the corpus parameter

I don't know the encoding of the corpus string. It was generated by PlaintextCorpusReader.tag() method, from NLTK. The original corpus file was encoded in UTF-8, according to Notepad++. The tagged_token.decode('ISO-8859-1') is just a guess. I've tried to decode it as cp1252, and got the same mangled characters from ISO-8859-1.

Answer 1

When you open the file with codec.open('w', encoding='utf8'), there is no point in writing byte arrays (str objects) into the file. Instead, write unicode objects, like this:

corpusFile = codecs.open(filename, mode = 'w', encoding = 'utf-8')
# ...
tagged_token = '\xdcml\xe4ut'
tagged_token = tagged_token.decode('ISO-8859-1')
corpusFile.write(tagged_token)
corpusFile.write(u'\n')

This will write platform-dependent End-Of-Line characters.

Alternatively, open a binary file and write byte arrays of already-encoded strings:

corpusFile = open(filename, mode = 'wb')
# ...
tagged_token = '\xdcml\xe4ut'
tagged_token = tagged_token.decode('ISO-8859-1')
corpusFile.write(tagged_token.encode('utf-8'))
corpusFile.write('\n')

This will write platform-independent EOLs. If you want a platform-dependent EOL, print os.sep instead of '\n'.

Note that the encoding naming in Notepad++ is misleading: ANSI as UTF-8 is what you want.

Answer 2

Try writing the file with a UTF-8 signature (aka BOM):

def storeTaggedCorpus(corpus, filename):
    corpusFile = codecs.open(filename, mode = 'w', encoding = 'utf-8-sig')
    for token in corpus:
        tagged_token = '/'.join(str for str in token)
        # print(type(tagged_token)); break
        # tagged_token = tagged_token.decode('cp1252')
        corpusFile.write(tagged_token)
        corpusFile.write(u"\n")
    corpusFile.close()

Note that this will only work properly if tagged_token is a unicode string. To check that, uncomment the first comment in the above code - it should print <type 'unicode'>.

If tagged_token is not a unicode string, then you will need to decode it first using the second commented line. (NB: I've assumed a "cp1252" encoding, but if you're certain it's "iso-8859-1", then of course you will need to change it.)

Answer 3

If you are seeing "mangled" characters from a file, you need to ensure that whatever you are using to view the file understands that the file is UTF-8-encoded.

The files created by this code:

import codecs
for enc in "utf-8 utf-8-sig".split():
    with codecs.open(enc + ".txt", mode = 'w', encoding = enc) as corpusFile:
        tagged_token = '\xdcml\xe4ut'
        tagged_token = tagged_token.decode('cp1252') # not 'ISO-8859-1'
        corpusFile.write(tagged_token) # write unicode objects
        corpusFile.write(u'\n')

are identified thusly:

Notepad++ (version 5.7 (UNICODE)) : UTF-8 without BOM, UTF-8
Firefox (7.0.1): Western(ISO-8859-1), Unicode (UTF-8)
Notepad (Windows 7): UTF-8, UTF-8

Putting a BOM in your UTF-8 file, while deprecated on Unix systems, gives you a much better chance on Windows that other software will be able to recognise your file as UTF-8-encoded.