I'm using the new Python Requests library to make http requests. I obtain a cookie from the server as text. How do I turn that into a CookieJar
with the cookie in it?
![](../../users/profiles/95735.webp)
- 38,049
- 34
- 218
- 341
![](../../users/profiles/76701.webp)
- 71,442
- 73
- 210
- 338
-
2I see your Python mailing list message, and I see here the reason for your desperation. – Matt Joiner Aug 03 '11 at 15:34
-
1I don't see it. What's so bad/convoluted about the top answer? – panzi Aug 03 '11 at 19:35
-
1The top answer is great, I was made desperate by the answers that came before it. – Ram Rachum Aug 06 '11 at 12:33
-
2With [`requests.Session()`](http://docs.python-requests.org/en/latest/user/advanced/#session-objects) there is *no need* to worry about cookie jars. The session object manages receiving and sending cookies for you. – Martijn Pieters Feb 12 '14 at 19:17
-
See [this answer](https://stackoverflow.com/a/47913559/274677) for a way to accomplish that without using the [Session](http://docs.python-requests.org/en/latest/user/advanced/#session-objects) object. – Marcus Junius Brutus Dec 20 '17 at 21:31
10 Answers
I'm confused by this question. The requests library will put the cookies in the jar for you.
import requests
import cookielib
URL = '...whatever...'
jar = cookielib.CookieJar()
r = requests.get(URL, cookies=jar)
r = requests.get(URL, cookies=jar)
The first request to the URL will fill the jar. The second request will send the cookies back to the server. The same goes for the standard library's urllib module cookielib. (doc currently available for 2.x Version)
![](../../users/profiles/7593876.webp)
- 110
- 14
![](../../users/profiles/877078.webp)
- 649
- 5
- 4
-
-
2A new release was just pushed (v0.6.0) that allows you to attach cookies to a request with a simple dictionary. http://docs.python-requests.org/en/latest/user/quickstart/#cookies – Kenneth Reitz Aug 18 '11 at 18:36
-
Thanks! This made me realize that in webtest, testapp.post(url) will add any received cookies to testapps cookiejar also. – robert king Jun 20 '13 at 23:57
-
3This wasn't working for me, so I asked a [similar question](http://stackoverflow.com/questions/21736970/using-requests-module-how-to-handle-set-cookie-in-request-response#21737086) to clarify. – tommy_o Feb 12 '14 at 19:15
-
3Note that this is **not needed** if you are using a `requests.Session` object; it'll handle the cookie jar for you, entirely. – Martijn Pieters Feb 12 '14 at 19:15
-
7Just wanted to point out that "import cookielib" is only valid under 2.X - in 3.X, it's "import http.cookiejar". – Doormatt Apr 14 '14 at 00:37
-
2I couldn't get this code working. Even passing a cookie jar to each request didn't persist my cookies, but the `requests.Session` listed here in the comments and further down worked perfectly. – djsumdog Dec 09 '14 at 23:04
-
1This did **not** work for me with the latest release (v2.18.4). This worked instead: https://stackoverflow.com/a/47913559/274677 (without using `Session`). But I guess `Session` is the way to go. – Marcus Junius Brutus Dec 20 '17 at 20:06
-
This doesn't work for me either, with Python 2.7 and Requests 2.21.0. I have to use jar.set_cookie to put every cookie into the Jar or it gets nothing from the request.get method. – Jing He Apr 15 '19 at 17:33
A requests Session
will also receive and send cookies.
s = requests.Session()
s.get('http://httpbin.org/cookies/set/sessioncookie/123456789')
r = s.get("http://httpbin.org/cookies")
print(r.text)
# '{"cookies": {"sessioncookie": "123456789"}}'
(Code above stolen from http://www.python-requests.org/en/latest/user/advanced/#session-objects)
If you want cookies to persist on disk between runs of your code, you can directly use a cookie jar and save/load them. More cumbersome, but still pretty easy:
import requests
import cookielib
cookie_file = '/tmp/cookies'
cj = cookielib.LWPCookieJar(cookie_file)
# Load existing cookies (file might not yet exist)
try:
cj.load()
except:
pass
s = requests.Session()
s.cookies = cj
s.get('http://httpbin.org/cookies/set/sessioncookie/123456789')
r = s.get("http://httpbin.org/cookies")
# Save cookies to disk, even session cookies
cj.save(ignore_discard=True)
Then look in the file:
$ cat /tmp/cookies
#LWP-Cookies-2.0
Set-Cookie3: sessioncookie=123456789; path="/"; domain="httpbin.org"; path_spec; discard; version=0
![](../../users/profiles/69689.webp)
- 22,610
- 3
- 63
- 68
-
2I really wish this was voted to the top. The top answer selected does not work at all with requests-2.3.0 – djsumdog Dec 09 '14 at 03:41
-
1_Thank you_ for the `ignore_discard=True` hint! I couldn't figure out why my jar saved an empty file. – Michael Oct 02 '19 at 18:39
I think many of these answers are missing the point. Sometimes that other library isn't using requests under the hood. Or doesn't expose the cookiejar it's using. Sometimes all we have is the cookie string. In my case I'm trying to borrow the auth cookie from pyVmomi.
import requests
import http.cookies
raw_cookie_line = 'foo="a secret value"; Path=/; HttpOnly; Secure; '
simple_cookie = http.cookies.SimpleCookie(raw_cookie_line)
cookie_jar = requests.cookies.RequestsCookieJar()
cookie_jar.update(simple_cookie)
Which gives us the following cookie_jar
:
In [5]: cookie_jar
Out[5]: <RequestsCookieJar[Cookie(version=0, name='foo', value='a secret value', port=None, port_specified=False, domain='', domain_specified=False, domain_initial_dot=False, path='/', path_specified=True, secure=True, expires=None, discard=False, comment='', comment_url=False, rest={'HttpOnly': True}, rfc2109=False)]>
Which we can use as normal:
requests.get(..., cookies=cookie_jar)
![](../../users/profiles/26630.webp)
- 10,956
- 3
- 27
- 34
To help you out, I wrote an entire module. I tried it with my personal webpage and google's cookies, so I'd assume it works.
I got help from How to add cookie to existing cookielib CookieJar instance in Python?
I have a lot of unpythonic code in here including a semi-kludge, so your mileage may vary. Tweak it as you wish, especially with the assumed items (such as port 80) the "request" as an argument below is of type requests.request and I realized that the "method" argument must be all capitals. Hope I could help!
Note: I haven't had time to add comments for clarification so you'll have to use the source.
import Cookie,cookielib,requests,datetime,time #had this out but realized later I needed it when I continued testing
def time_to_tuple(time_string):
wday = {'Mon':0,'Tue':1,'Wed':2,'Thu':3,'Fri':4,'Sat':5,'Sun':6}
mon = {'Jan':1,'Feb':2,'Mar':3,'Apr':4,'May':5,'Jun':6,'Jul':7,'Aug':8,'Sep':9,'Oct':10,'Nov':11,'Dec':12}
info = time_string.split(' ')
info = [i.strip() for i in info if type(i)==str]
month = None
for i in info:
if '-' in i:
tmp = i.split('-')
for m in tmp:
try:
tmp2 = int(m)
if tmp2<31:
mday = tmp2
elif tmp2 > 2000:
year = tmp2
except:
for key in mon:
if m.lower() in key.lower():
month = mon[key]
elif ':' in i:
tmp = i.split(':')
if len(tmp)==2:
hour = int(tmp[0])
minute = int(tmp[1])
if len(tmp)==3:
hour = int(tmp[0])
minute = int(tmp[1])
second = int(tmp[2])
else:
for item in wday:
if ((i.lower() in item.lower()) or (item.lower() in i.lower())):
day = wday[item]
if month is None:
for item in mon:
if ((i.lower() in item.lower()) or (item.lower() in i.lower())):
month = mon[item]
return year,month,mday,hour,minute,second
def timefrom(year,month,mday,hour,minute,second):
time_now = time.gmtime()
datetime_now = datetime.datetime(time_now.tm_year,time_now.tm_mon,
time_now.tm_mday,time_now.tm_hour,
time_now.tm_min,time_now.tm_sec)
then = datetime.datetime(year,month,mday,hour,minute,second)
return (datetime_now-then).total_seconds()
def timeto(year,month,mday,hour,minute,second):
return -1*timefrom(year,month,mday,hour,minute,second)
##['comment', 'domain', 'secure', 'expires', 'max-age', 'version', 'path', 'httponly']
def parse_request(request):
headers = request.headers
cookieinfo = headers['set-cookie'].split(';')
name = 'Undefined'
port=80
port_specified=True
c = Cookie.SmartCookie(headers['set-cookie'])
cj = cookielib.CookieJar()
for m in c.values():
value = m.coded_value
domain = m['domain']
expires = m['expires']
if type(expires) == str:
tmp = time_to_tuple(expires)
expires = timeto(tmp[0],tmp[1],tmp[2],tmp[3],tmp[4],tmp[5])
max_age=m['max-age']
version = m['version']
if version == '':
version = 0
path = m['path']
httponly = m['httponly']
if httponly == '':
if 'httponly' in headers['set-cookie'].lower():
httponly = True
else:
httponly = False
secure = m['secure']
comment=m['comment']
port = 80
port_specified=False
domain_specified=True
domain_initial_dot = domain.startswith('.')
path_specified=True
discard = True
comment_url=None
rest={'HttpOnly':httponly}
rfc2109=False
ck = cookielib.Cookie(version,name,value,port,port_specified,domain,
domain_specified,domain_initial_dot,path,path_specified,
secure,expires,discard,comment,comment_url,rest,rfc2109)
cj.set_cookie(ck)
return cj
![](../../users/profiles/-1.webp)
- 1
- 1
![](../../users/profiles/871980.webp)
- 8,255
- 2
- 35
- 60
-
I have name="undefined" due to the fact that I haven't been able to find out where the name is. If someone could point out where, I would be happy to update the code. – Snakes and Coffee Aug 03 '11 at 06:08
-
If you have too many cookies (which I doubt, given the function of cookies), feel free to use yield instead of return. – Snakes and Coffee Aug 03 '11 at 06:11
-
1A new release was just pushed (v0.6.0) that allows you to attach cookies to a request with a simple dictionary. http://docs.python-requests.org/en/latest/user/quickstart/#cookies – Kenneth Reitz Aug 18 '11 at 18:39
-
Isn't all this code equivalent to the existing `cookielib.CookieJar.extract_cookies(response, request)` ??? – MestreLion Jun 05 '14 at 06:08
Well cookielib.LWPCookieJar has load and save methods on it. Look at the format and see if it matches the native cookie format, you may well be able to load your cookie straight into a cookie jar using StringIO. Alternatively, if requests is using urllib2 under the hood couldn't you add a cookie handler to the default opener?
![](../../users/profiles/5341.webp)
- 7,810
- 2
- 28
- 32
As dstanek answered, requests will automatically put response cookies in a cookie jar for you.
However, if you manually specify a Cookie
header entry, requests will not put those cookies in a jar for you. This means any subsequent requests will be lacking your initial set of cookies, but will have any new cookies going forward.
If you do need to manually create a cookie jar for requests, use requests.cookies.RequestsCookieJar
. In case their example code changes:
jar = requests.cookies.RequestsCookieJar()
jar.set('tasty_cookie', 'yum', domain='httpbin.org', path='/cookies')
jar.set('gross_cookie', 'blech', domain='httpbin.org', path='/elsewhere')
url = 'http://httpbin.org/cookies'
r = requests.get(url, cookies=jar)
Note that if you provide a cookie jar and a Cookie
header, the header takes precedence, but the cookie jar will still be maintained for future requests.
![](../../users/profiles/477563.webp)
- 18,561
- 2
- 47
- 99
I'm trying to do the same thing. This is what I have so far, and for some reason it isn't sending the cookies along in the header. It might get you far enough along to solve your problem though.
import requests
import cookielib
import logging
log = logging.getLogger(__name__)
def auth(auth_url, cookies):
cj = cookielib.CookieJar()
for x in cookies:
if len(cookies[x]) > 0:
ck = cookielib.Cookie(version=1, name=x, value=cookies[x],
port=None, port_specified=False, domain='.example.com',
domain_specified=True,
domain_initial_dot=True, path='/',
path_specified=True, secure=False,
expires=None, discard=True,
comment=None, comment_url=None,
rest=None, rfc2109=True)
log.info(ck)
cj.set_cookie(ck)
log.info("cookies = %s " % cj)
response = requests.get(auth_url, cookies=cj)
log.info("response %s \n" % response)
log.info("response.headers %s \n" % response.headers)
log.info("response.content %s \n" % response.content)
![](../../users/profiles/356788.webp)
- 68,551
- 9
- 45
- 57
-
1A new release was just pushed (v0.6.0) that allows you to attach cookies to a request with a simple dictionary. http://docs.python-requests.org/en/latest/user/quickstart/#cookies – Kenneth Reitz Aug 18 '11 at 18:38
-
Assuming that you have requested for url
and you got headers
as response. Type type of url
is string. Type type of headers
is list.
import urllib2
import cookielib
class dummyResponse:
def __init__(self,headers):
self.headers=headers
def info(self):
return dummyInfo(self.headers)
class dummyInfo:
def __init__(self,headers):
self.headers=headers
def getheaders(self,key):
#Headers are in the form: 'Set-Cookie: key=val\r\n'. We want 'key=val'
newMatches=[]
for header in self.headers:
if header.lower().startswith(key.lower()):
clearHeader=header[len(key)+1:].strip()
newMatches.append(clearHeader)
return newMatches
req=urllib2.Request(url)
resp=dummyResponse(headers)
jar=cookielib.CookieJar()
jar.extract_cookies(resp, req)
![](../../users/profiles/514429.webp)
- 4,452
- 3
- 34
- 49
Simplified version of overthink's answer, on how to get a cookiejar and persist the cookies in Python3:
import requests
s = requests.Session()
r1 = s.get('https://stackoverflow.com')
print("r1",r1.cookies) #Have cookie
print("s",s.cookies) #Have cookie(jar)
r2 = s.get('https://stackoverflow.com') #The cookie from r1 is resend
print("r2",r2.cookies) #No cookie (could be a new one)
print("s",s.cookies) #Keep the cookie(jar) from r1
To persist the cookies between sessions you have to save and reuse the cookiejar in Session (the s variable). If you get different answers between r1/r2/s on other sites, check if there is a redirect. As example will r1/r2 get no cookie for https://www.stackoverflow.com because it is redirected to the site without www.
![](../../users/profiles/2326672.webp)
- 3,630
- 1
- 36
- 32
Try this site: Voidspace article
Over the years I have found voidspace to be extremely useful for doing this kind of stuff. Hope I helped, although I'm quite a dunce. The code is available at Voidspace Recipes as source code .py although the download file is a ".py-" file.
![](../../users/profiles/871980.webp)
- 8,255
- 2
- 35
- 60
-
Really overly complex for such a simple task. I guess the only thing I can do is wait for `requests` to implement cookie-handling. – Ram Rachum Aug 01 '11 at 18:12
-
I'm a web programmer, but I'm mostly involved with sending cookies. I'll revise when I get home since company internet disallows programmatic web-diving. – Snakes and Coffee Aug 02 '11 at 15:37
-
A new release was just pushed (v0.6.0) that allows you to attach cookies to a request with a simple dictionary. http://docs.python-requests.org/en/latest/user/quickstart/#cookies – Kenneth Reitz Aug 18 '11 at 18:38