Say I had a io.BytesIO()
I wanted to write a response to sitting on a thread:
f = io.ByteIO()
with requests.Session() as s:
r = s.get(url, stream = True)
for chunk in r.iter_content(chunk_size = 1024):
f.write(chunk)
Now this is not to harddisk but rather in memory (got plenty of it for my purpose), so I don't have to worry about the needle being a bottleneck. I know for blocking I/O (file read/write) the GIL is released from the docs and this SO post by Alex Martelli, but I wonder, does the GIL just release on f.write()
and then reacquire on the __next__()
call of the loop?
So what I end up with are a bunch of fast GIL acquisitions and releases. Obviously I would have to time this to determine anything worth note, but does writing to in memory file objects on a multithreaded web scraper in general support GIL bypass?
If not, I'll just handle the large responses and dump them into a queue and process on __main__
.