Using a proxy in Python: requests, httpx and aiohttp with HTTP, SOCKS5 and authentication
In this article
Python has three popular HTTP libraries (requests, httpx and aiohttp), and each declares a proxy slightly differently. This guide has working sample code for all three, with authenticated HTTP and SOCKS5 proxies, plus the common errors. Every example reads the proxy from a PROXY environment variable in the form http://USER:PASS@HOST:PORT, so the password stays out of your code.
requests
The most familiar library. Pass the proxy through proxies, and use a Session to reuse connections:
import os
import requests
PROXY = os.environ["PROXY"] # http://USER:PASS@HOST:PORT
session = requests.Session()
session.proxies = {"http": PROXY, "https": PROXY}
resp = session.get("https://api.ipify.org", timeout=30)
print("IP via proxy:", resp.text)Note: the "https" key in proxies is the kind of site you visit, while the value still starts with http:// because that's how you talk to the proxy.
SOCKS5 with requests
Add SOCKS support with pip install "requests[socks]", then use the socks5h prefix so the proxy resolves domain names:
PROXY = "socks5h://USER:PASS@HOST:PORT"
session.proxies = {"http": PROXY, "https": PROXY}httpx
httpx offers both sync and async styles. In recent versions, the proxy goes in the proxy argument:
import os
import httpx
with httpx.Client(proxy=os.environ["PROXY"], timeout=30) as client:
resp = client.get("https://api.ipify.org")
print("IP via proxy:", resp.text)The async version:
import asyncio
import os
import httpx
async def main():
async with httpx.AsyncClient(proxy=os.environ["PROXY"], timeout=30) as client:
resp = await client.get("https://api.ipify.org")
print("IP via proxy:", resp.text)
asyncio.run(main())For SOCKS5, install pip install "httpx[socks]" and use socks5://USER:PASS@HOST:PORT. If an older httpx complains about proxy=, upgrade it: older versions used a different argument name.
aiohttp
aiohttp takes the proxy per request through the proxy argument, and supports HTTP proxies only:
import asyncio
import os
import aiohttp
PROXY = os.environ["PROXY"] # http://USER:PASS@HOST:PORT
async def main():
timeout = aiohttp.ClientTimeout(total=30)
async with aiohttp.ClientSession(timeout=timeout) as session:
async with session.get("https://api.ipify.org", proxy=PROXY) as resp:
print("IP via proxy:", await resp.text())
asyncio.run(main())For SOCKS5 with aiohttp, install the aiohttp-socks library and create the session with its connector. To have aiohttp read the HTTPS_PROXY environment variable, create the session with trust_env=True.
Passwords with special characters
If the username or password contains @, :, / or #, encode it before building the proxy URL:
import os
from urllib.parse import quote
user = quote(os.environ["PROXY_USER"], safe="")
password = quote(os.environ["PROXY_PASS"], safe="")
PROXY = "http://" + user + ":" + password + "@HOST:PORT"Without encoding, the library splits the URL wrongly and you get a 407 or a puzzling connection error.
Environment variables
requests and httpx read HTTP_PROXY, HTTPS_PROXY and NO_PROXY automatically when you don't pass a proxy in code:
export HTTPS_PROXY="http://USER:PASS@HOST:PORT"
python script.pyHandy for small scripts, but easy to forget: a leftover variable can send other programs through the proxy too. For real projects, declaring the proxy explicitly in code is easier to control.
Common errors
- 407 Proxy Authentication Required: wrong username or password, or special characters left unencoded.
ProxyError/ can't connect: wrong host or port, or a SOCKS5 URL without the library's SOCKS support installed.Missing dependencies for SOCKS support: requests withoutrequests[socks].- Errors for a few seconds right after rotating: the 4G device is reconnecting; retry shortly.
- The IP is still your home IP: the proxy isn't on the session you're using, or
NO_PROXYexcludes that domain.
The full list is in common proxy errors; handling 403 and 429 when scraping is in 403, 429 and 503 errors when scraping.
Rotating from Python
With a 4G proxy, rotating means calling the rotation link (not through the proxy) and waiting until the new IP appears. Full code is in rotating proxy IPs via API. For Scrapy, see using a proxy with Scrapy.
65Proxy proxies in Python
Our 4G proxies offer HTTP and SOCKS5 with a username and password, and work with all three libraries above. Singapore mobile-carrier IPs, a dedicated proxy that isn't shared, rotation by link or API (at most once every 5 minutes), priced 1 day $4, 7 days $13, 1 month $32 (200GB each), plus monthly plans with more bandwidth (500GB $40, 1TB $65). See pricing.
Frequently asked questions
Should I use requests, httpx or aiohttp?
requests for simple scripts; httpx when you want sync and async with the same style; aiohttp when you're already in the asyncio ecosystem. All three work well with proxies.
Should I create a new session after rotating?
Yes. A new session ensures no old connections stay open over the previous IP, and cookies don't carry over from one session to the next.
Can I send many parallel requests through one 4G proxy?
Yes, but they all go out on one IP. Keep parallel connections modest to avoid being limited. See how to scrape without getting blocked.