Goal: Collect all the images from a site as I browse.
I've tried:
requests and wget don't work even with cookies set and all headers changed to mimic Firefox.
Firefox cache has the images, but they all have a random string as the name. I need logical names to sort them.
selenium-wire is very close to working. When I do driver.get(), driver.requests gives me all the requests as expected which can then be saved. The problem is when I click buttons on the site, the new requests do not get added to driver.requests. I tried:
driver = webdriver.Firefox()
driver.get("url")
while True:
time.sleep(1)
# browse site
for request in driver.requests:
if request.response:
if "image/jpeg" in request.response.headers['Content-Type']:
with open(request.url, 'wb') as f:
request.response.body
