Force requests to use IPv4 / IPv6

Question

How to force the requests library to use a specific internet protocol version for a get request? Or can this be achieved better with another method in Python? I could but I do not want to use curl…

Example to clarify purpose:

import requests
r = requests.get('https://my-dyn-dns-service.domain/?hostname=my.domain',
                 auth = ('myUserName', 'my-password'))

score 14 · Answer 1 · answered Oct 27 '17 at 09:55

14

I've found a minimalistic solution to force urrlib3 to use either ipv4 or ipv6. This method is used by urrlib3 for creating new connection both for Http and Https. You can specify in it any AF_FAMILY you want to use.

import socket
import requests.packages.urllib3.util.connection as urllib3_cn


  def allowed_gai_family():
    """
     https://github.com/shazow/urllib3/blob/master/urllib3/util/connection.py
    """
    family = socket.AF_INET
    if urllib3_cn.HAS_IPV6:
        family = socket.AF_INET6 # force ipv6 only if it is available
    return family

urllib3_cn.allowed_gai_family = allowed_gai_family

answered Oct 27 '17 at 09:55

Yuri Fedoseev

349
3
7

Nice solution! And it also can be used to force IPv4 simply by changing the function to `def allowed_gai_family(): return socket.AF_INET` – MestreLion Mar 15 '21 at 18:47
The question if for requests not urrlib3 – Pedro Lobito May 07 '21 at 17:57

score 10 · Answer 2 · answered Apr 26 '18 at 13:23

This is a hack, but you can monkey-patch getaddrinfo to filter to only IPv4 addresses:

# Monkey patch to force IPv4, since FB seems to hang on IPv6
import socket
old_getaddrinfo = socket.getaddrinfo
def new_getaddrinfo(*args, **kwargs):
    responses = old_getaddrinfo(*args, **kwargs)
    return [response
            for response in responses
            if response[0] == socket.AF_INET]
socket.getaddrinfo = new_getaddrinfo

score 4 · Answer 3 · answered Oct 03 '19 at 21:50

I've written a runtime patch for requests+urllib3+socket that allows passing the required address family optionally and on a per-request basis.

Unlike other solutions there is no monkeypatching involved, rather you replace your imports of requests with the patched file and it present a request-compatible interface with all exposed classes subclassed and patched and all “simple API” function reimplemented. The only noticeable difference should be the fact that there is an extra family parameter exposed that you can use to restrict the address family used during name resolution to socket.AF_INET or socket.AF_INET6. A somewhat complicated (but mostly just LoC intensive) series of strategic method overrides is then used to pass this value all the way down to the bottom layers of urllib3 where it will be used in an alternate implementation of the socket.create_connection function call.

TL;DR usage looks like this:

import socket

from . import requests_wrapper as requests  # Use this load the patch


# This will work (if IPv6 connectivity is available) …
requests.get("http://ip6only.me/", family=socket.AF_INET6)
# … but this won't
requests.get("http://ip6only.me/", family=socket.AF_INET)

# This one will fail as well
requests.get("http://127.0.0.1/", family=socket.AF_INET6)

# This one will work if you have IPv4 available
requests.get("http://ip6.me/", family=socket.AF_INET)

# This one will work on both IPv4 and IPv6 (the default)
requests.get("http://ip6.me/", family=socket.AF_UNSPEC)

Full link to the patch library (~350 LoC): https://gitlab.com/snippets/1900824

score 3 · Answer 4 · edited May 23 '17 at 12:10

I took a similar approach to https://stackoverflow.com/a/33046939/5059062, but instead patched out the part in socket that makes DNS requests so it only does IPv6 or IPv4, for every request, which means this can be used in urllib just as effectively as in requests.

This might be bad if your program also uses unix pipes and other such things, so I urge caution with monkeypatching.

import requests
import socket
from unittest.mock import patch
import re

orig_getaddrinfo = socket.getaddrinfo
def getaddrinfoIPv6(host, port, family=0, type=0, proto=0, flags=0):
    return orig_getaddrinfo(host=host, port=port, family=socket.AF_INET6, type=type, proto=proto, flags=flags)

def getaddrinfoIPv4(host, port, family=0, type=0, proto=0, flags=0):
    return orig_getaddrinfo(host=host, port=port, family=socket.AF_INET, type=type, proto=proto, flags=flags)

with patch('socket.getaddrinfo', side_effect=getaddrinfoIPv6):
    r = requests.get('http://ip6.me')
    print('ipv6: '+re.search(r'\+3>(.*?)</',r.content.decode('utf-8')).group(1))

with patch('socket.getaddrinfo', side_effect=getaddrinfoIPv4):
    r = requests.get('http://ip6.me')
    print('ipv4: '+re.search(r'\+3>(.*?)</',r.content.decode('utf-8')).group(1))

and without requests:

import urllib.request
import socket
from unittest.mock import patch
import re

orig_getaddrinfo = socket.getaddrinfo
def getaddrinfoIPv6(host, port, family=0, type=0, proto=0, flags=0):
    return orig_getaddrinfo(host=host, port=port, family=socket.AF_INET6, type=type, proto=proto, flags=flags)

def getaddrinfoIPv4(host, port, family=0, type=0, proto=0, flags=0):
    return orig_getaddrinfo(host=host, port=port, family=socket.AF_INET, type=type, proto=proto, flags=flags)

with patch('socket.getaddrinfo', side_effect=getaddrinfoIPv6):
    r = urllib.request.urlopen('http://ip6.me')
    print('ipv6: '+re.search(r'\+3>(.*?)</',r.read().decode('utf-8')).group(1))

with patch('socket.getaddrinfo', side_effect=getaddrinfoIPv4):
    r = urllib.request.urlopen('http://ip6.me')
    print('ipv4: '+re.search(r'\+3>(.*?)</',r.read().decode('utf-8')).group(1))

Tested in 3.5.2

score 1 · Answer 5 · edited May 23 '17 at 11:54

1

This is totally untested and will probably require some tweaks, but combining answers from Using Python “requests” with existing socket connection and how to force python httplib library to use only A requests, it looks like you should be able to create an IPv6 only socket and then have requests use that for its connection pool with something like:

try:
    from http.client import HTTPConnection
except ImportError:
    from httplib import HTTPConnection

class MyHTTPConnection(HTTPConnection):
    def connect(self):
        print("This actually called called")
        self.sock = socket.socket(socket.AF_INET6)
        self.sock.connect((self.host, self.port,0,0))
        if self._tunnel_host:
            self._tunnel()

requests.packages.urllib3.connectionpool.HTTPConnection = MyHTTPConnection

edited May 23 '17 at 11:54

Community

1
1

answered Oct 09 '15 at 20:49

Foon

5,551
11
36
39

Sorry, I still get an IPv4 answer. I tried `print(requests.get('https://icanhazip.com').text)`. – ominug Oct 09 '15 at 21:39
Unfortunately, I don't have access to anything with ipv6 at the moment; I did add a print statement and a connect statement just to make sure I didn't miss something with how one goes about monkeypatching the HTTPConnection object – Foon Oct 09 '15 at 22:25
Ok, I tried it again. The connect method is not called. – ominug Oct 11 '15 at 09:19
I removed the last 3 parameters of `self.sock.connect(self.host, self.port,0,0)` and tried `MyHTTPConnection('icanhazip.com').request('GET', '/')` directly. Then the method is invoked, but an error is returned: `TypeError: getsockaddrarg: AF_INET6 address must be tuple, not str`. – ominug Oct 11 '15 at 09:22
Yeah, I forgot to use double parens (one for the call, one for the tuple) – Foon Oct 11 '15 at 12:54
AttributeError: 'module' object has no attribute 'packages' – 1a1a11a Mar 07 '16 at 19:25
@1a1a11a Hmm... in one of the answers I cribbed off of, they did mention that you needed a fairly new version of requests (although that was a couple years old answer IIRC) to get the request.packages bit ... what version of requests are you using? I just did a pip install and got 2.9.1 and that had requests.packages – Foon Mar 07 '16 at 19:32

score 1 · Answer 6 · answered Aug 30 '16 at 17:31

After reading the previous answer, I had to modify the code to force IPv4 instead of IPv6. Notice that I used socket.AF_INET instead of socket.AF_INET6, and self.sock.connect() has 2-item tuple argument.

I also needed to override the HTTPSConnection which is much different than HTTPConnection since requests wraps the httplib.HTTPSConnection to verify the certificate if the ssl module is available.

import socket
import ssl
try:
    from http.client import HTTPConnection
except ImportError:
    from httplib import HTTPConnection
from requests.packages.urllib3.connection import VerifiedHTTPSConnection

# HTTP
class MyHTTPConnection(HTTPConnection):
    def connect(self):
        self.sock = socket.socket(socket.AF_INET)
        self.sock.connect((self.host, self.port))
        if self._tunnel_host:
            self._tunnel()

requests.packages.urllib3.connectionpool.HTTPConnection = MyHTTPConnection
requests.packages.urllib3.connectionpool.HTTPConnectionPool.ConnectionCls = MyHTTPConnection

# HTTPS
class MyHTTPSConnection(VerifiedHTTPSConnection):
    def connect(self):
        self.sock = socket.socket(socket.AF_INET)
        self.sock.connect((self.host, self.port))
        if self._tunnel_host:
            self._tunnel()
        self.sock = ssl.wrap_socket(self.sock, self.key_file, self.cert_file)

requests.packages.urllib3.connectionpool.HTTPSConnection = MyHTTPSConnection
requests.packages.urllib3.connectionpool.VerifiedHTTPSConnection = MyHTTPSConnection
requests.packages.urllib3.connectionpool.HTTPSConnectionPool.ConnectionCls = MyHTTPSConnection

This answer worked for me, with the caveat that it results in `InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning` when running over https — kyrenia, Jun 18 '17 at 17:15

Force requests to use IPv4 / IPv6

6 Answers6

Linked