How do you pull JSON array from API in Python?

Question

I am having issues pulling a JSON array from http://api.divesites.com. When I manually set the data as a string in Python 3.7, I can see the array.

    import json

def main():
    hardcode = """{"request":{"str":null,"timestamp":1572865590,"loc":{"lat":"50.442","lng":"-4.08279999999999"},"mode":"sites","dist":"9","api":1},"sites":[{"currents":null,"distance":"7.47","hazards":null,"lat":"50.3378","name":"Fort Bovisand","water":null,"marinelife":null,"description":null,"maxdepth":null,"mindepth":null,"predive":null,"id":"24387","equipment":null,"lng":"-4.1285"},{"currents":null,"distance":"7.93","hazards":null,"lat":"50.3352","name":"Plymouth Breakwater","water":null,"marinelife":null,"description":null,"maxdepth":null,"mindepth":null,"predive":null,"id":"24388","equipment":null,"lng":"-4.1485"}],"version":1,"loc":{"lat":"50.442","lng":"-4.08279999999999"},"result":true}
        """
    print("Original Data: ---", get_data(hardcode))
    print("Site Data:----", get_data(hardcode)['sites'])

def get_data(data):
    return json.loads(data)

if __name__=='__main__':
    main()

Output

Original Data: --- {'request': {'str': None, 'timestamp': 1572865590, 'loc': {'lat': '50.442', 'lng': '-4.08279999999999'}, 'mode': 'sites', 'dist': '9', 'api': 1}, 'sites': [{'currents': None, 'distance': '7.47', 'hazards': None, 'lat': '50.3378', 'name': 'Fort Bovisand', 'water': None, 'marinelife': None, 'description': None, 'maxdepth': None, 'mindepth': None, 'predive': None, 'id': '24387', 'equipment': None, 'lng': '-4.1285'}, {'currents': None, 'distance': '7.93', 'hazards': None, 'lat': '50.3352', 'name': 'Plymouth Breakwater', 'water': None, 'marinelife': None, 'description': None, 'maxdepth': None, 'mindepth': None, 'predive': None, 'id': '24388', 'equipment': None, 'lng': '-4.1485'}], 'version': 1, 'loc': {'lat': '50.442', 'lng': '-4.08279999999999'}, 'result': True}
Site Data:---- [{'currents': None, 'distance': '7.47', 'hazards': None, 'lat': '50.3378', 'name': 'Fort Bovisand', 'water': None, 'marinelife': None, 'description': None, 'maxdepth': None, 'mindepth': None, 'predive': None, 'id': '24387', 'equipment': None, 'lng': '-4.1285'}, {'currents': None, 'distance': '7.93', 'hazards': None, 'lat': '50.3352', 'name': 'Plymouth Breakwater', 'water': None, 'marinelife': None, 'description': None, 'maxdepth': None, 'mindepth': None, 'predive': None, 'id': '24388', 'equipment': None, 'lng': '-4.1485'}]

However when I try and perform a GET request (using urllib.request.urlopen()), I only receive the objects.

import urllib.request
import json

def get_jsonparsed_data(url):
    response = urllib.request.urlopen(url)
    data = response.read().decode("utf-8")
    return json.loads(data)

def main():
    url = 'http://api.divesites.com/?mode=sites&lat=-50.350874&lng=175.849890&dist=12'
    print("Original Data: ---", get_jsonparsed_data(url))
    print("Sites: ----", get_jsonparsed_data(url)['sites'])



if __name__=='__main__':
    main()

Output

Original Data: --- {'request': {'str': None, 'timestamp': 1572866211, 'loc': {'lat': '-50.350874', 'lng': '175.849890'}, 'mode': 'sites', 'dist': '12', 'api': 1}, 'sites': [], 'version': 1, 'loc': {'lat': '-50.350874', 'lng': '175.849890'}, 'result': True}
Site Data:---- []

Am I doing something wrong, or do I need an extra step?

https://stackoverflow.com/questions/12965203/how-to-get-json-from-webpage-into-python-script — planet260, Nov 04 '19 at 11:46
see your `Original Data` - there is `'sites': [],` - so you get empty list from server and `Site Data` shows you this empty list. All work correctly. — furas, Nov 04 '19 at 11:51
If I use `lat=39.8333&lng=-98.5833` then code gives me many results in `sites`. The same for `lat=50.442&lng=-4.08279999999999`. Simply you have to use different values `lat`, `lng` to get some `sites` — furas, Nov 04 '19 at 12:00

score 1 · Answer 1 · answered Nov 04 '19 at 12:34

Solved it. I spent so long focusing on debugging the code I forgot to check the final API call. Turns out I either didn't have enough distance set, or the wrong starting longitude and latitude. This was different to my hard coded JSON.

The URL should have been: http://api.divesites.com/?mode=sites&lat=50.442&lng=-4.08279999999999&dist=9

Stupid mistake. Thanks for helping though. It got me to look.

score 0 · Answer 2 · answered Nov 04 '19 at 11:55

0

First, I'd use the requests library instead and then you can find the solution here

answered Nov 04 '19 at 11:55

Dinac23

173
2
10

problem has nothing to do with `requests` or `JSON` - code works correctly with different values `lat`, `lng` – furas Nov 04 '19 at 11:59

How do you pull JSON array from API in Python?

2 Answers2