python unable to load a json file with utf-8 encoding

Question

With the following python code:

filePath = urllib2.urlopen('xx.json')
fileJSON = json.loads(filePath.read().decode('utf-8'))

Where the xx.json looks like:

{
    "tags": [{
        "id": "123",
        "name": "Airport",
        "name_en": "Airport",
        "name_cn": "机场",
        "display": false
    }]
}

I see the following exception:

fileJSON = json.loads(filePath.read().decode('utf-8'))
    File "/usr/lib64/python2.7/json/__init__.py", line 339, in loads
        return _default_decoder.decode(s)
    File "/usr/lib64/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    File "/usr/lib64/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
    ValueError: No JSON object could be decoded

The code works before the Chinese characters are added to the json file, when I also added the .decode('utf-8') behind the read() as well.

I am not sure what needs to be done?

Odd. `urllib2.urlopen('xxx.json')` raises `ValueError: unknown url type: xxx.json` for me. `'file:xxx.json'` works for me. — Robᵩ, Jan 09 '18 at 18:54
Guess: your *actual* code has lines between `urlopen()` and `json.loads()`. In particular, you've already `.read()` the contents of the file and this `.read()` is returning an empty string. — Robᵩ, Jan 09 '18 at 19:00
2nd guess. Your `xxx.json` file is encoded with a BOM. Can you run `file xxx.json` and show us the result? — Robᵩ, Jan 09 '18 at 19:07
This is yet another case that Python 3 handles better (clearer errors when it fails, and better defaults so it doesn't fail). — Josh Lee, Jan 09 '18 at 19:13
I updated the file with the real file that I was using. The two lines are consecutive i.e. the streams are not read in between the two lines I had. — jamesdeath123, Jan 09 '18 at 19:22
@Robᵩ I updated to a real file, could you please check again? — jamesdeath123, Jan 09 '18 at 19:29
@larsks I updated the file to the one that I actually run into problem with. Could you please check? — jamesdeath123, Jan 09 '18 at 19:29

Josh Lee · Accepted Answer · 2018-01-09T19:34:29.480

$ wget https://s3.amazonaws.com/wherego-sims/tags.json 
$ file tags.json 
tags.json: UTF-8 Unicode (with BOM) text, with CRLF line terminators

This file begins with a byte order mark (EF BB BF), which is illegal in JSON (JSON Specification and usage of BOM/charset-encoding). You must first decode this using 'utf-8-sig' in Python to get a valid JSON unicode string.

json.loads(filePath.read().decode('utf-8-sig'))

For what it's worth, Python 3 (which you should be using) will give a specific error in this case and guide you in handling this malformed file:

json.decoder.JSONDecodeError: Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)

Namely, by specifying that you wish to discard the BOM if it exists (again, it's not conventional to use this in UTF-8, particularly with JSON which is always encoded in UTF-8 so it is worse than useless):

>>> import json
>>> json.load(open('tags.json', encoding='utf-8-sig'))

Good digging. I was beginning to suspect something along those lines — smac89, Jan 09 '18 at 19:33

python unable to load a json file with utf-8 encoding

1 Answers1