Skip to main content Brad's PyNotes

Http.client Module and Requests Library

TL;DR

The http.client module provides low-level HTTP protocol client functionality, but Python’s documentation recommends using the third-party Requests library instead for higher-level HTTP operations. Requests offers automatic connection pooling, elegant API design, and handles the complexities that make http.client fiddly to use directly.

Interesting!

Python’s standard library documentation directly recommends the third-party Requests library for most HTTP operations. The difference is striking - here’s a GET request with JSON response:

python code snippet start

# http.client: manual connection and header handling
import http.client, json
conn = http.client.HTTPSConnection("api.example.com")
conn.request("GET", "/")
data = json.loads(conn.getresponse().read())
conn.close()

# requests: clean and readable
import requests
data = requests.get("https://api.example.com/").json()

python code snippet end

The http.client Module: Low-Level HTTP

The http.client module exposes the raw HTTP protocol through classes like HTTPConnection and HTTPSConnection. Here’s a basic GET request:

python code snippet start

import http.client

conn = http.client.HTTPSConnection("api.example.com")
conn.request("GET", "/")
response = conn.getresponse()

print(response.status, response.reason)  # 200 OK
data = response.read()
conn.close()

python code snippet end

Making a POST request requires manual header management and parameter encoding:

python code snippet start

import http.client
import urllib.parse

params = urllib.parse.urlencode({'username': 'alice', 'action': 'login'})
headers = {"Content-type": "application/x-www-form-urlencoded"}

conn = http.client.HTTPConnection("example.com")
conn.request("POST", "/api/login", params, headers)
response = conn.getresponse()
data = response.read()
conn.close()

python code snippet end

Requests: HTTP for Humans

The Requests library transforms HTTP operations into intuitive, readable code. The same GET request becomes:

python code snippet start

import requests

response = requests.get("https://api.example.com/")
print(response.status_code, response.reason)  # 200 OK
data = response.text

python code snippet end

POST requests with parameters are dramatically simpler:

python code snippet start

import requests

response = requests.post("https://example.com/api/login",
                        data={'username': 'alice', 'action': 'login'})
print(response.json())  # Automatic JSON parsing

python code snippet end

Key Requests Features

Requests handles complexity automatically:

Sessions with persistent cookies:

python code snippet start

import requests

session = requests.Session()
session.get("https://example.com/login")
# Cookies automatically maintained across requests
session.get("https://example.com/dashboard")

python code snippet end

Authentication built-in:

python code snippet start

from requests.auth import HTTPBasicAuth

response = requests.get("https://api.example.com/user",
                       auth=HTTPBasicAuth('user', 'password'))

python code snippet end

Timeout protection:

python code snippet start

try:
    response = requests.get("https://slow-site.com", timeout=5)
except requests.Timeout:
    print("Request took too long")

python code snippet end

File uploads made easy:

python code snippet start

files = {'file': open('report.pdf', 'rb')}
response = requests.post("https://example.com/upload", files=files)

python code snippet end

Installation

While http.client is built into Python, Requests requires installation:

bash code snippet start

pip install requests

bash code snippet end

The trade-off is worthwhile: Requests’ elegance, reliability, and comprehensive feature set make it the de facto standard for HTTP operations in Python. The module powers countless production applications and is maintained by the Python community with the same rigor as standard library modules.

For HTTP operations paired with asynchronous programming, see asyncio integration . The module works seamlessly with JSON parsing for API interactions. For lower-level network programming, explore socket operations and urllib utilities .

Reference: Python http.client Documentation and Requests Documentation