How to scrape without getting blocked: pace, sessions and clean IPs
In this article
The most common question in scraping is "how do I stop getting blocked?" People reach for more proxies, then better proxies, and still hit walls. The uncomfortable truth is that a clean IP is necessary but not sufficient: most blocks come from how you request, not where from. This guide covers what actually triggers blocks and how to behave like a normal visitor, with a proxy doing its part, not carrying the whole load.
Blocks come from behaviour, not just the IP
A site decides you're a bot from a mix of signals:
- Speed and volume: bursts of requests no human could make.
- Patterns: the same path, same interval, no variation.
- Many identities on one IP: ten accounts logging in from a single address.
- A mismatched fingerprint: headers, time zone or browser that don't add up.
A clean IP (see clean proxies and blacklists) sets a good starting point. It can't save behaviour that screams "automation". Why that holds for CAPTCHAs in particular is in clean proxy and CAPTCHA, and the rules for running bots politely are in proxy for bots and automation.
Slow down and back off
Pace is the single biggest lever:
- Pause between requests: a few seconds, with some randomness, not a fixed metronome.
- Cap parallel connections: a handful at once, not hundreds.
- Honour
Retry-Afteron 429 and 503, and back off progressively (5, 10, 20 seconds…) when there's no header. - Run off-peak for the site where you can.
Going slower almost always beats going wider.
One identity per IP and session
Treat an IP like a person:
- Keep cookies with their IP: one session stays on one IP; a new IP means a new session.
- Don't run many accounts through one IP at the same time.
- Rotate only when you're actually limited, not for every request (each 65Proxy proxy rotates at most once every 5 minutes). How per-request, timed and on-demand rotation differ is in rotating proxy types.
- Don't rotate mid-session: it breaks logins and looks suspicious.
Look like a normal visitor
- Send a sensible User-Agent and headers, including an
Accept-Languagethat matches the region you're appearing from. - Don't over-fake. A random fingerprint on every request is itself a flag; be consistently normal.
- Use a real browser when the site needs one: JavaScript-heavy pages are covered in scraping JavaScript sites.
- Read the error to know what to change: common proxy errors tells 407, 429 and connection failures apart.
Respect robots.txt, terms and personal data
Staying unblocked and staying fair overlap more than people expect:
- Read
robots.txtand the site's terms, and respect what they disallow. - Collect only public data; don't log in to reach it, and don't bypass CAPTCHAs.
- Be careful with personal data: names, phones and emails are protected by laws such as Singapore's PDPA.
- Don't overload sites: if you slow someone's site down, expect a block, and know you're harming them.
Where 65Proxy fits
Our Singapore 4G proxies give you two of the things on this list: a clean mobile IP to start from, and a fresh one on demand when a site limits you (by link, the 5-minute rule applies). Priced 1 day $4, 7 days $13, 1 month $32 (200GB each), plus monthly plans with more bandwidth (500GB $40, 1TB $65). The pacing, sessions and fingerprint are yours to get right: this guide is how. See proxies for web scraping and pricing.
Frequently asked questions
I rotated the IP and still get blocked. Why?
Because the block is about behaviour, not the IP. Slow down, keep one identity per session, fix any fingerprint mismatch, and respect the site's limits. A new IP alone won't help if the pattern stays the same.
Do more IPs mean fewer blocks?
Usually not. A reasonable pace on a few good IPs beats a flood of requests across many. More IPs just spread the same bad behaviour around; see clean proxies and blacklists.
Should I randomise everything to look human?
No. Inconsistency (a new browser, time zone and language every request) is itself suspicious. Aim to look like one ordinary, consistent visitor, not a different one each time.