scraping through python selenium or beautifulsoup

Question

i want to scrape the url below https://turo.com/search?airportCode=EWR&customDelivery=true&defaultZoomLevel=11&endDate=04%2F05%2F2019&endTime=11%3A00&international=true&isMapSearch=false&itemsPerPage=200&location=EWR&locationType=Airport&maximumDistanceInMiles=30&sortType=RELEVANCE&startDate=03%2F05%2F2019&startTime=10%3A00

i want all the href link of the cars listed so that i can scrape forward but i am unable to get it

please help

Answer 1

You can pull in the json response with the query and work with that:

import requests
from pandas.io.json import json_normalize

base_url = 'https://turo.com'

url = 'https://turo.com/api/search?'
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36',
'Accept': '*/*',
'Accept-Encoding': 'gzip, deflate, br',
'Accept-Language': 'en-US,en;q=0.9',
'Connection': 'keep-alive',
'Host': 'turo.com',
'Referer': 'https://turo.com/search'}

params = {
'airportCode': 'EWR',
'customDelivery': 'true',
'defaultZoomLevel': '11',
'endDate': '04/05/2019',
'endTime': '11:00',
'international': 'true',
'isMapSearch': 'false',
'itemsPerPage': '200',
'location': 'EWR',
'locationType': 'Airport',
'maximumDistanceInMiles': '30',
'sortType': 'RELEVANCE',
'startDate': '03/05/2019',
'startTime': '10:00'}

response = requests.get(url, headers=headers, params=params)
data = response.json()

search_id = data['searchId']
print (search_id)

for ele in data['list']:
    link = ele['vehicle']['url']
    print (base_url + link)




links_list = [ base_url + ele['vehicle']['url'] for ele in data['list'] ]




# If you want to Manipulate a table of the data
df = json_normalize(data['list'])

Output:

https://turo.com/rentals/cars/nj/jersey-city/ford-mustang/385739
https://turo.com/rentals/suvs/nj/jersey-city/mazda-cx-9/84783
https://turo.com/rentals/cars/nj/chatham-township/mercedes-benz-s-class/439410
https://turo.com/rentals/cars/nj/edgewater/bmw-3-series/360338
https://turo.com/rentals/cars/nj/edgewater/mercedes-benz-s-class/528415
https://turo.com/rentals/cars/nj/jersey-city/mercedes-benz-c-class/546598
https://turo.com/rentals/cars/nj/wallington/bmw-3-series/523288
https://turo.com/rentals/cars/nj/edgewater/mercedes-benz-c-class/402723
https://turo.com/rentals/cars/nj/east-brunswick/dodge-charger/474370
https://turo.com/rentals/cars/nj/dover/mini-cooper/562685
https://turo.com/rentals/cars/nj/east-brunswick/dodge-challenger/245816
https://turo.com/rentals/cars/nj/city-of-orange/bmw-3-series/441085
https://turo.com/rentals/cars/nj/aberdeen-township/chevrolet-corvette/329734
https://turo.com/rentals/suvs/nj/montvale/nissan-rogue/551766
https://turo.com/rentals/cars/nj/haledon/ford-mustang/527092
https://turo.com/rentals/suvs/nj/montvale/toyota-rav4/551789
https://turo.com/rentals/cars/nj/wallington/bmw-3-series/506047
https://turo.com/rentals/suvs/nj/englewood/toyota-rav4/205796
https://turo.com/rentals/cars/nj/jersey-city/audi-tt-rs/454914
https://turo.com/rentals/suvs/nj/newark/jeep-grand-cherokee/547858
https://turo.com/rentals/suvs/nj/scotch-plains/acura-mdx/494289
https://turo.com/rentals/suvs/nj/jersey-city/dodge-journey/371145
https://turo.com/rentals/cars/nj/jersey-city/tesla-model-s/450268
https://turo.com/rentals/cars/nj/ridgefield-park/bmw-m3/479442
https://turo.com/rentals/suvs/nj/edgewater/bmw-x3/509043
https://turo.com/rentals/suvs/nj/lodi/kia-sorento/535604
https://turo.com/rentals/suvs/nj/scotch-plains/bmw-x5/555187
https://turo.com/rentals/suvs/nj/west-new-york/audi-q5/317033
https://turo.com/rentals/suvs/nj/jersey-city/honda-cr-v/523284
https://turo.com/rentals/suvs/nj/old-bridge-township/acura-mdx/384309
https://turo.com/rentals/cars/pa/wind-gap/bmw-3-series/558761
https://turo.com/rentals/suvs/nj/jersey-city/toyota-rav4/442146
https://turo.com/rentals/cars/nj/hillsborough-township/mercedes-benz-c-class/330064
https://turo.com/rentals/cars/nj/jersey-city/audi-a8/518856
https://turo.com/rentals/cars/nj/mount-olive-township/bmw-3-series/528020
https://turo.com/rentals/cars/il/lincolnwood/mercedes-benz-cls-class/207362
https://turo.com/rentals/cars/nj/jersey-city/bmw-5-series/515357
https://turo.com/rentals/cars/nj/newark/audi-a6/108287
https://turo.com/rentals/cars/nj/montvale/audi-a3/552084
https://turo.com/rentals/suvs/nj/jersey-city/bmw-x6-m/325161
https://turo.com/rentals/minivans/nj/jersey-city/honda-odyssey/256481
https://turo.com/rentals/cars/nj/dover/bmw-1-series/562679
https://turo.com/rentals/suvs/nj/fort-lee/audi-q7/437196
https://turo.com/rentals/cars/nj/edgewater/jaguar-xf/528929
https://turo.com/rentals/cars/nj/jersey-city/bmw-3-series/309937
https://turo.com/rentals/cars/nj/jersey-city/bmw-1-series/517828
https://turo.com/rentals/suvs/nj/old-bridge-township/jeep-wrangler/506051
https://turo.com/rentals/suvs/nj/jersey-city/bmw-x5/533213
https://turo.com/rentals/suvs/nj/morristown/honda-pilot/310610
https://turo.com/rentals/cars/nj/jersey-city/chevrolet-camaro/207528
https://turo.com/rentals/minivans/nj/jersey-city/honda-odyssey/506466
https://turo.com/rentals/suvs/nj/edgewater/acura-mdx/522686
https://turo.com/rentals/cars/nj/bloomfield/porsche-911/153991
https://turo.com/rentals/suvs/nj/west-new-york/land-rover-range-rover-sport/295016
https://turo.com/rentals/suvs/nj/jersey-city/infiniti-fx/533200
https://turo.com/rentals/suvs/nj/jersey-city/bmw-x5/551272
https://turo.com/rentals/suvs/nj/newark/kia-sorento/554517
https://turo.com/rentals/suvs/nj/jersey-city/honda-cr-v/517423
https://turo.com/rentals/suvs/nj/rahway/jeep-grand-cherokee/410048
https://turo.com/rentals/cars/nj/clifton/dodge-challenger/513745
https://turo.com/rentals/cars/nj/jersey-city/bmw-z4/569205
https://turo.com/rentals/suvs/nj/union-city/acura-rdx/159384
https://turo.com/rentals/cars/nj/roselle/chevrolet-impala/389132
https://turo.com/rentals/suvs/nj/newark/ford-edge/543229
https://turo.com/rentals/cars/nj/wallington/bmw-3-series/560189
https://turo.com/rentals/suvs/nj/rahway/nissan-murano/491762
https://turo.com/rentals/suvs/nj/sayreville/ford-escape/561262
https://turo.com/rentals/cars/nj/randolph/bmw-3-series/476134
https://turo.com/rentals/cars/nj/jersey-city/dodge-charger/353789
https://turo.com/rentals/suvs/nj/bloomfield/land-rover-range-rover-sport/526192
https://turo.com/rentals/cars/nj/hillsborough-township/infiniti-g-sedan/466276
https://turo.com/rentals/suvs/nj/edgewater/lexus-rx-350/455895
https://turo.com/rentals/trucks/nj/jackson/toyota-tundra/506040
https://turo.com/rentals/cars/nj/jersey-city/tesla-model-3/458395
https://turo.com/rentals/cars/nj/jersey-city/mercedes-benz-e-class/533931
https://turo.com/rentals/suvs/nj/monroe-township/bmw-x3/526846
https://turo.com/rentals/suvs/nj/roselle-park/volkswagen-tiguan/534144
https://turo.com/rentals/suvs/pa/easton/mercedes-benz-glk-class/422203
https://turo.com/rentals/cars/nj/jersey-city/bmw-3-series/543432
https://turo.com/rentals/cars/nj/edison/infiniti-q50/559651
https://turo.com/rentals/suvs/nj/elizabeth/jeep-grand-cherokee/566712
https://turo.com/rentals/cars/nj/edison/infiniti-q50/448312
https://turo.com/rentals/suvs/nj/rahway/toyota-highlander/524049
https://turo.com/rentals/cars/nj/newark/ford-mustang/551401
https://turo.com/rentals/suvs/nj/montvale/porsche-macan/551814
https://turo.com/rentals/cars/nj/hackensack/porsche-cayman/514871
https://turo.com/rentals/cars/nj/city-of-orange/mercedes-benz-c-class/414725
https://turo.com/rentals/cars/nj/east-brunswick/ford-flex/340268
https://turo.com/rentals/cars/nj/burlington/tesla-model-s/316123
https://turo.com/rentals/cars/nj/jersey-city/tesla-model-3/552630
https://turo.com/rentals/suvs/nj/jackson/ford-explorer/534670
https://turo.com/rentals/cars/nj/haledon/honda-insight/421898
https://turo.com/rentals/cars/nj/elizabeth/bmw-5-series/548519
https://turo.com/rentals/suvs/nj/montvale/mercedes-benz-glc-class/552101
https://turo.com/rentals/suvs/nj/jersey-city/bmw-x5/357098
https://turo.com/rentals/trucks/nj/weehawken/ford-f-150/405386
https://turo.com/rentals/cars/nj/jersey-city/audi-a4/541799
https://turo.com/rentals/suvs/nj/edgewater/jeep-wrangler/219798
https://turo.com/rentals/trucks/nj/montvale/ford-f-150/551838
https://turo.com/rentals/cars/nj/little-falls/lexus-is-250/179987
https://turo.com/rentals/cars/nj/old-bridge-township/maserati-granturismo-convertible/488496
https://turo.com/rentals/cars/nj/jersey-city/jaguar-f-type/420320
https://turo.com/rentals/suvs/nj/east-brunswick/gmc-acadia/348063
https://turo.com/rentals/cars/nj/garfield/mercedes-benz-e-class/500408
https://turo.com/rentals/cars/nj/edison/mercedes-benz-c-class/315170
https://turo.com/rentals/cars/nj/bayonne/mercedes-benz-cls-class/480787
https://turo.com/rentals/cars/nj/hammonton/ford-mustang/534026
https://turo.com/rentals/cars/nj/south-amboy/bmw-7-series/138755
https://turo.com/rentals/cars/nj/jersey-city/bmw-i3/344512
https://turo.com/rentals/cars/pa/easton/pontiac-g3/343750
https://turo.com/rentals/cars/nj/jersey-city/bmw-5-series/362079
https://turo.com/rentals/cars/nj/montvale/bmw-4-series/551976
https://turo.com/rentals/cars/nj/haledon/honda-civic/369357
https://turo.com/rentals/cars/nj/phillipsburg/porsche-boxster/420778
https://turo.com/rentals/suvs/nj/city-of-orange/infiniti-qx30/427614
https://turo.com/rentals/suvs/nj/jersey-city/toyota-highlander/196708
https://turo.com/rentals/suvs/nj/jersey-city/mercedes-benz-glk-class/546930
https://turo.com/rentals/cars/nj/elizabeth/mercedes-benz-e-class/473235
https://turo.com/rentals/cars/nj/newark/volkswagen-jetta/297019
https://turo.com/rentals/suvs/nj/little-ferry/honda-cr-v/483880
https://turo.com/rentals/cars/nj/jersey-city/mazda-3/379389
https://turo.com/rentals/cars/nj/jersey-city/tesla-model-3/539962
https://turo.com/rentals/cars/nj/jackson/cadillac-cts/506062
https://turo.com/rentals/cars/ct/sharon/bmw-7-series/536091
https://turo.com/rentals/suvs/nj/east-orange/porsche-macan/467839
https://turo.com/rentals/cars/nj/roselle/nissan-versa/473960
https://turo.com/rentals/cars/nj/hoboken/mercedes-benz-e-class/533198
https://turo.com/rentals/suvs/nj/hillsborough-township/gmc-acadia/553441
https://turo.com/rentals/cars/nj/lodi/tesla-model-3/568337
https://turo.com/rentals/cars/nj/jersey-city/mazda-6/528355
https://turo.com/rentals/cars/nj/west-new-york/mercedes-benz-e-class/544361
https://turo.com/rentals/suvs/nj/edison/gmc-terrain/416012
https://turo.com/rentals/cars/nj/newark/ford-focus/284564
https://turo.com/rentals/cars/nj/old-bridge-township/bmw-5-series/560039
https://turo.com/rentals/cars/nj/linwood/toyota-camry/219654
https://turo.com/rentals/cars/nj/jersey-city/honda-fit/429840
https://turo.com/rentals/suvs/nj/jersey-city/bmw-x5/429301
https://turo.com/rentals/suvs/nj/montvale/jaguar-f-pace/551780
https://turo.com/rentals/suvs/nj/elizabeth/ford-explorer/361114
https://turo.com/rentals/cars/nj/jersey-city/infiniti-g37/69252
https://turo.com/rentals/cars/nj/jersey-city/tesla-model-3/483663
https://turo.com/rentals/minivans/nj/linden/toyota-sienna/565755
https://turo.com/rentals/suvs/nj/dover/jeep-wrangler/273594
https://turo.com/rentals/suvs/nj/jersey-city/mercedes-benz-glk-class/147897
https://turo.com/rentals/suvs/nj/montvale/audi-q7/552088
https://turo.com/rentals/cars/nj/montvale/nissan-altima/551762
https://turo.com/rentals/cars/nj/jersey-city/bmw-3-series/382246
https://turo.com/rentals/cars/nj/newark/chevrolet-malibu/337149
https://turo.com/rentals/cars/nj/fairview/honda-accord/441008
https://turo.com/rentals/cars/pa/easton/ford-fiesta/324654
https://turo.com/rentals/cars/nj/montvale/mercedes-benz-c-class/552099
https://turo.com/rentals/suvs/nj/west-new-york/land-rover-lr3/317945
https://turo.com/rentals/cars/nj/montvale/bmw-4-series/551975
https://turo.com/rentals/cars/pa/lansdale/volvo-s80/502585
https://turo.com/rentals/suvs/nj/montvale/alfa-romeo-stelvio/567102
https://turo.com/rentals/cars/pa/easton/toyota-yaris/286270
https://turo.com/rentals/cars/nj/jersey-city/hyundai-accent/406944
https://turo.com/rentals/minivans/nj/jackson/toyota-sienna/506024
https://turo.com/rentals/suvs/nj/jersey-city/land-rover-range-rover-evoque/253107
https://turo.com/rentals/suvs/nj/bloomfield/land-rover-range-rover-sport/307494
https://turo.com/rentals/cars/nj/montvale/mercedes-benz-e-class/552109
https://turo.com/rentals/minivans/nj/newark/honda-odyssey/535200
https://turo.com/rentals/cars/nj/jersey-city/ford-mustang/500739
https://turo.com/rentals/suvs/nj/elizabeth/toyota-rav4/503028
https://turo.com/rentals/cars/nj/montvale/audi-a6/552092
https://turo.com/rentals/suvs/nj/paterson/honda-cr-v/312733
https://turo.com/rentals/cars/nj/montvale/bmw-5-series/551973
https://turo.com/rentals/cars/nj/teaneck/tesla-model-s/520126
https://turo.com/rentals/suvs/nj/little-falls/acura-mdx/222780
https://turo.com/rentals/suvs/nj/wallington/bmw-x3/440188
https://turo.com/rentals/suvs/nj/belleville/jeep-cherokee/501526
https://turo.com/rentals/suvs/nj/harrison/volkswagen-touareg/531531
https://turo.com/rentals/cars/nj/jersey-city/ford-mustang/438281
https://turo.com/rentals/suvs/nj/roxbury-township/bmw-x1/562709
https://turo.com/rentals/cars/nj/montvale/mercedes-benz-c-class/551803
https://turo.com/rentals/cars/pa/easton/nissan-versa/276913
https://turo.com/rentals/suvs/nj/city-of-orange/kia-sorento/101234
https://turo.com/rentals/suvs/nj/montvale/bmw-x5/551968
https://turo.com/rentals/minivans/pa/easton/toyota-sienna/215018
https://turo.com/rentals/cars/nj/edgewater/hyundai-elantra-touring/476041
https://turo.com/rentals/cars/nj/haledon/nissan-altima/318098
https://turo.com/rentals/suvs/nj/paterson/toyota-rav4/553455
https://turo.com/rentals/cars/nj/union-city/hyundai-sonata/461816
https://turo.com/rentals/cars/nj/montvale/jaguar-xe/551782
https://turo.com/rentals/cars/nj/northvale/bmw-4-series/483229
https://turo.com/rentals/cars/nj/harrison/audi-a5/518618
https://turo.com/rentals/cars/nj/newark/chevrolet-camaro/409288
https://turo.com/rentals/suvs/nj/montvale/porsche-cayenne/551802
https://turo.com/rentals/suvs/ct/west-haven/lexus-gx-460/557327
https://turo.com/rentals/trucks/nj/south-river/dodge-ram-1500/537439
https://turo.com/rentals/suvs/nj/hoboken/gmc-terrain/217576
https://turo.com/rentals/cars/nj/mount-olive-township/chevrolet-cruze/392840
https://turo.com/rentals/cars/nj/old-bridge-township/mercedes-benz-cls-class/488471
https://turo.com/rentals/cars/nj/garfield/nissan-altima/355678
https://turo.com/rentals/suvs/nj/jersey-city/infiniti-qx60/454257
https://turo.com/rentals/cars/nj/montvale/toyota-camry/551790
https://turo.com/rentals/cars/nj/edgewater/volkswagen-jetta/519761
https://turo.com/rentals/suvs/nj/weehawken/chevrolet-tahoe/97631
https://turo.com/rentals/cars/nj/elizabeth/honda-accord/524828
https://turo.com/rentals/cars/nj/montville/volkswagen-jetta/553559