Using Python “requests” with existing socket connection

Question

The Python “requests” library is currently all the rage, because of the beautiful interface that it provides for making HTTP requests — but beneath it there seems to be many layers of indirection — sessions, HTTP adapters, and finally the mechanics of urllib3.

Where in this stack of abstractions is the right place to intervene if I already hold an open socket, and want to use “requests” to send an HTTP response down that socket and receive a reply back?

Without some kind of intervention (or customization?), the stack will try to create a new TCP/IP socket for me, but in my particular application my code is not called until a connection has already been established on my behalf, so I will need to convince Requests to talk on that existing socket if I want to be able to use Requests' features.

The Requests library:

http://pypi.python.org/pypi/requests

https://github.com/kennethreitz/requests

Answer 1

The following code needs requests from git (especially requests.packages.urllib3.poolmanager.PoolManager._new_pool())

I tested it using ncat -v -l 127.0.0.1 8000

The problem is the fact, that the connection isn't opened by urllib3 but by httplib from the standard library.

import socket
import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3 import PoolManager, HTTPConnectionPool

try:
    from http.client import HTTPConnection
except ImportError:
    from httplib import HTTPConnection


class MyAdapter(HTTPAdapter):
    def init_poolmanager(self, connections, maxsize):
        self.poolmanager = MyPoolManager(num_pools=connections,
                                         maxsize=maxsize)


class MyPoolManager(PoolManager):
    def _new_pool(self, scheme, host, port):
        # Important!
        if scheme == 'http' and host == my_host and port == my_port:
            return MyHTTPConnectionPool(host, port, **self.connection_pool_kw)
        return super(PoolManager, self)._new_pool(self, scheme, host, port)


class MyHTTPConnectionPool(HTTPConnectionPool):
    def _new_conn(self):
        self.num_connections += 1
        return MyHTTPConnection(host=self.host,
                            port=self.port,
                            strict=self.strict)


class MyHTTPConnection(HTTPConnection):
    def connect(self):
        """Connect to the host and port specified in __init__."""
        # Original
        # self.sock = socket.create_connection((self.host, self.port),
        #                                    self.timeout, self.source_address)
        # Important!
        self.sock = my_socket
        if self._tunnel_host:
            self._tunnel()


if __name__ == '__main__':
    import time

    my_host = '127.0.0.1'
    my_port = 8000

    my_socket = socket.create_connection((my_host, my_port))
    time.sleep(4)
    s = requests.Session()
    s.mount('http://', MyAdapter())
    s.get('http://127.0.0.1:8000/foo')

Edit:

Or direct monkeypatching of the connectionpool:

class MyHTTPConnection(HTTPConnection):
    def connect(self):
        self.sock = my_socket
        if self._tunnel_host:
            self._tunnel()

requests.packages.urllib3.connectionpool.HTTPConnection = MyHTTPConnection

if __name__ == '__main__':
    my_host = '127.0.0.1'
    my_port = 8000

    my_socket = socket.create_connection((my_host, my_port))
    requests.get('http://127.0.0.1:8000/foo')

Answer 2

Go straight to the urllib3 library; it holds a connection pool in the urllib3.connectionpool module.

You could replace the pool or adjust it by hacking the poolmanager module perhaps.