Use urllib in python to fetch IPv6 URL with FQDN

Question

I want to fetch an IPv6 page with urllib. Works with square brack IPv6 notation but I have no clue how to (easily) convince python to do an IPv6 request when I give it the FQDN Like the below ip is: https://www.dslreports.com/whatismyip

from sys import version_info

PY3K = version_info >= (3, 0)

if PY3K:
    import urllib.request as urllib
else:
    import urllib2 as urllib

url = None
opener = urllib.build_opener()
opener.addheaders = [('User-agent',
     "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36")]
url = opener.open("http://[2607:fad0:3706:1::1000]/whatismyip", timeout=3)
content = url.read()

To clarify, you want to always use an IPv6 address when both IPv4 and IPv6 addresses are available? — Gavin Panella, Aug 11 '17 at 16:46
Yes, as the code should, in turn but not in that order, check IPv4 then IPv6. — Steve Clement, Aug 13 '17 at 10:58

Alexandre Fenyo · Answer 1 · 2018-02-01T02:47:52.700

[update: this line about Python 2 / Python 3 is non longer valid since the question has been updated]

First, you seem to use Python 2. This is important because the urllib module has been split into parts and renamed in Python 3.

Secondly, your code snippet seems incorrect: build_opener is not a function available with urllib. It is available with urllib2.

So, I assume that your code is in fact the following one:

import urllib2
opener = urllib2.build_opener()
opener.addheaders = [('User-agent',
 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36")]
url = opener.open("http://www.dslreports.com/whatismyip", timeout=3)

If your DNS resolver handles correctly IPv6 resource records, and if your operating system is built with dual-stack IPv4/IPv6 or single IPv6-only stack, and if you have a correct IPv6 network path to dslreports.com, this Python program will use IPv6 to connect to www.dslreports.com. So, there is no need to convince python to do an IPv6 request.

Well I was not really clear I guess. As you see with my update to my snippet the code is python2 and 3 compatible. Further on in the ethos of python, explicit is better the implicit. I do not want it to rely on a users ability to properly configure IPv6. You should have the hand on whether to use v4 or v6. — Steve Clement, Aug 13 '17 at 10:52

score 0 · Accepted Answer · answered Aug 13 '17 at 22:00

I finally solved my issue. Not in the most elegant way, but it works for me.

After reading:

Force requests to use IPv4 / IPv6 and Python urllib2 force IPv4

I decided to do an DNS lookup and just send a Host header with the FQDN to grab the content. (Host headers are needed for vhosts)

Here is the ugly snippet:

# Ugly hack to get either IPv4 or IPv6 response from server
parsed_uri = urlparse(server)
fqdn = "{uri.netloc}".format(uri=parsed_uri)
scheme = "{uri.scheme}".format(uri=parsed_uri)
path = "{uri.path}".format(uri=parsed_uri)

try:
    ipVersion = ip_kind(fqdn[1:-1])
    ip = fqdn
except ValueError:
    addrs = socket.getaddrinfo(fqdn, 80)
    if haveIPv6:
        ipv6_addrs = [addr[4][0] for addr in addrs if addr[0] == socket.AF_INET6]
        ip = "[" + ipv6_addrs[0] + "]"
    else:
        ipv4_addrs = [addr[4][0] for addr in addrs if addr[0] == socket.AF_INET]
        ip = ipv4_addrs[0]

server = "{}://{}{}".format(scheme, ip, path)

url = urllib.Request(server, None, {'User-agent' : 'Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5'})
# Next line adds the host header
url.host = fqdn
content = urllib.urlopen(url).read()

This is far from ideal and it could be much cleaner but it works for me.

It is implemented here: https://github.com/SteveClement/ipgetter/tree/IPv6 This simply goes through a list of servers that return you your border gateway ip, now in IPv6 too.

Use urllib in python to fetch IPv6 URL with FQDN

2 Answers2