Proxy for bots and automation: one identity per IP, a polite pace, and what to do when a CAPTCHA appears
In this article
- 01 What a proxy does for automation, and what it doesn't
- 02 Five rules that matter more than the IP
- 03 When a challenge appears, stop and read it
- 04 Example: a polite fetcher that stops on a challenge
- 05 Rotating the IP responsibly
- 06 How many proxies, and how much bandwidth
- 07 What not to automate
- 08 65Proxy
- 09 Frequently asked questions
Scripts, scheduled jobs, test suites and browser automation all send traffic that a person didn't click by hand, and that is the point of them. This guide is about running that kind of legitimate automation through a proxy so it stays reliable: what a proxy adds, five rules that matter more than the IP, what to do when a site answers with a CAPTCHA, and a Python example that detects a challenge and stops cleanly instead of pushing on. It isn't about getting round a site's rules, and it won't help with that.
What a proxy does for automation, and what it doesn't
It helps with:
- Location. Seeing the page a Singapore visitor sees, or testing from a Singapore mobile network.
- Separation. Giving each real account or job its own address, so one session's activity isn't mixed with another's.
- A better starting point. A mobile IP is less likely to be flagged on sight than a datacenter IP, which means fewer challenges before your own behaviour has counted. See clean proxies and CAPTCHAs.
It doesn't help with:
- Permission. If a site forbids automation, a proxy doesn't make your script allowed.
- Behaviour. Speed, repetition and a browser that looks automated are read independently of the IP.
- CAPTCHAs and challenges. They check the visit, not only the address. We don't sell proxies as a way to beat them.
The overview of what bots are and which kinds are welcome is in what is botting.
Five rules that matter more than the IP
- One identity per IP and session. Keep cookies, user agent and account together with the IP they started on. Don't run many accounts through one address at once, and don't move one account between addresses at random.
- A human pace, with jitter. A few seconds between pages, with random variation, and few parallel connections. A fixed interval is itself a signal. More detail is in how to scrape without getting blocked.
- Back off when told to. On 429 wait for the
Retry-Aftertime, then slow down. Don't retry in a tight loop. The status codes are explained in 403, 429 and 503 errors when scraping. - Identify yourself and cache. Use a clear User-Agent with a contact address where appropriate, and don't re-download pages that haven't changed.
- Stop when the site says no. A repeating challenge, a block page or a request to stop is the site's decision. Respect it. See the next section.
When a challenge appears, stop and read it
| What you got | What it usually means | What to do |
|---|---|---|
| 429 Too Many Requests | You're going too fast | Wait for Retry-After, then run slower |
| 403 on the first request, while a browser on the same network gets in | Your program lacks what a browser sends, or the IP type is refused | Check headers and test the IP by hand; see common proxy errors |
| A "verifying you're human" page | The site wants a browser check | Use a real browser at a human pace, if the site allows automation at all |
| One CAPTCHA now and then | Your visit sits in the grey zone | Slow down, keep the session consistent, check the IP |
| CAPTCHA after CAPTCHA | The site doesn't want this traffic | Stop the run. Ask for an API or permission instead |
Two things that look like fixes and aren't:
- Rotating the IP and carrying on at the same speed. The site sees the same pace on the new address. Rotating is for an IP that is limited, not a way to ignore a challenge.
- Using a CAPTCHA-solving service. It defeats the purpose of the check, breaks most sites' terms and raises your legal risk. We advise against it; see the lines not to cross.
If you're a person hitting loops rather than a script, the step-by-step is in CAPTCHA keeps appearing or loops: 8 fixes.
Example: a polite fetcher that stops on a challenge
This Python sketch loads pages through your proxy at a human pace, honours Retry-After, and stops the whole run the moment a challenge appears. The proxy details come from an environment variable; never put them in the code.
import os
import random
import time
import requests
PROXY = os.environ["PROXY"] # http://USER:PASS@HOST:PORT
HEADERS = {"User-Agent": "my-price-checker/1.0 (contact: [email protected])"}
MAX_TRIES = 3
class Challenged(Exception):
"""The site asked for a human. Stop the run; don't try to solve it."""
def looks_like_challenge(resp):
# Cloudflare marks challenge responses with this header; other vendors differ.
if resp.headers.get("cf-mitigated") == "challenge":
return True
body = resp.text[:20000].lower()
return any(word in body for word in ("captcha", "verify you are human", "unusual traffic"))
def retry_after(resp, default=60):
value = resp.headers.get("Retry-After", "")
return int(value) if value.isdigit() else default
def polite_get(session, url):
for attempt in range(1, MAX_TRIES + 1):
resp = session.get(url, timeout=30)
if looks_like_challenge(resp):
raise Challenged(url)
if resp.status_code == 429:
time.sleep(retry_after(resp) * attempt)
continue
resp.raise_for_status()
return resp
raise RuntimeError(f"still rate limited after {MAX_TRIES} tries: {url}")
def run(urls):
session = requests.Session()
session.headers.update(HEADERS)
session.proxies = {"http": PROXY, "https": PROXY}
for url in urls:
try:
page = polite_get(session, url)
except Challenged:
print("Challenge page. Stopping the run:", url)
break
print(url, len(page.text))
time.sleep(random.uniform(3, 8)) # a human pace, with jitterNotes on the sketch: the word check also matches an ordinary page that talks about CAPTCHAs, so tune the markers for your target. For JavaScript-heavy pages use a real browser instead of requests; see scraping JavaScript sites. The general setup for Python is in using a proxy in Python.
Rotating the IP responsibly
A 4G proxy rotates by calling its rotation link, and each 65Proxy proxy allows that at most once every 5 minutes. Rotate when an IP is limited, not before every request, and start a fresh session (new cookies) after a rotation, because the site has just seen a new visitor. The code for calling the link and waiting for the new IP is in rotate a proxy IP with an API.
How many proxies, and how much bandwidth
- One proxy per concurrent identity. Count the accounts or jobs that run at the same time, not the ones that exist. How many proxies do I need walks through it.
- Budget the bandwidth from your page size. Multiply the weight of a typical page by the pages you load. Challenged and repeated loads use allowance too. How far does 500GB go shows how to measure.
What not to automate
- Actions on accounts you don't own, or fake accounts of any kind.
- Fake engagement, reviews and followers. See proxy for seeding for why.
- Beating purchase limits and queues, which is what proxies for flash sales explains.
- Game bots that play for you.
- Collecting personal data without a legal basis.
65Proxy
65Proxy sells Singapore 4G mobile proxies for legitimate work: one proxy per order with its own login and rotation link, HTTP and SOCKS5, priced 1 day $4, 7 days $13, 1 month $32 (200GB each), plus monthly plans with more bandwidth (500GB $40, 1TB $65). Try one for an hour on your real target first with the free trial. We don't sell them as a way to beat CAPTCHAs, bans or limits.
Frequently asked questions
Can a proxy make my bot undetectable?
No. A proxy changes the address a site sees. Pace, browser signals and behaviour are read separately, and a site can challenge any visitor whenever it wants.
How many requests per minute are safe?
There is no universal number. Start with one request every few seconds, watch the share of 429s and challenges, and slow down when they appear. Some sites state a limit in their documentation or API terms; follow it.
Should I rotate the IP on every request?
No. It breaks sessions, looks unnatural and, with a 5-minute rotation limit on each proxy, isn't possible anyway. Rotate when an IP is limited.
What should my script do when it meets a CAPTCHA?
Stop, log it and alert you. Then check your pace, look for an API or ask for access. The example above shows the shape.
Can I use Playwright or Selenium with the proxy?
Yes. Both can take a proxy with a username and password; the setup for Playwright and Puppeteer is in rotate a proxy IP with an API. The pace and challenge rules apply the same way.