Proxies for large-scale, multi-country scraping: choosing proxy types, system design and real costs
In this article
Once data collection grows past a few thousand pages a day from one site and spreads to hundreds of thousands of pages across countries, the question is no longer "which proxy should I buy" but "how do I design the system so cost and success rate stay sensible". This guide is for that scale. Up front: 65Proxy's Singapore 4G proxies fit only a small slice of this problem. We'll say exactly which.
First: can you avoid scraping?
At scale, this question is worth the most money:
- An official or paid API from the data source.
- Licensed datasets from data providers.
- A direct agreement with the site owner: many sell data exports.
Proxies, servers and the effort of maintaining scrapers often cost far more than buying clean data. See also is web scraping legal.
Mix proxy types
At scale, no single proxy type suits every site. The common approach is to split by how hard the target is:
| Site group | Traits | Suitable proxy |
|---|---|---|
| Easy (doesn't block servers) | Most information sites, public APIs | Datacenter (cheap, fast) |
| Medium (blocks server ranges) | Marketplaces, price-comparison sites | Per-request rotating residential |
| Hard, IP-sensitive | Public social media, strong anti-bot sites | Mobile, in small numbers |
| Country-specific content | Prices, search results per country | A proxy in that country |
Each type in detail: datacenter proxies, residential proxies, 4G proxies.
System design
What a large scraping system needs:
- A URL queue: separating "what to fetch" from "how to fetch it", allowing retries and priorities.
- Rate limits per domain, not per proxy. More proxies don't entitle you to hit one site harder.
- A proxy router: picking the proxy type per site group and tracking each group's success rate.
- Retries with backoff that honour
Retry-After. See 403, 429 and 503 errors when scraping. - Caching and change-based fetching:
ETag,If-Modified-Sinceand sitemaps so you only fetch what's new. - Monitoring: success rate, error codes, latency and cost per domain.
Cost it properly
Don't compare proxies on price per GB or per IP. Work out cost per successfully fetched page:
- total cost of proxies + servers + effort,
- divided by pages fetched successfully (after errors, retries and blocked pages).
Cheap proxies with a low success rate are often dearer than expensive ones with a high rate. And with per-GB proxies, blocking images, fonts and video is the fastest saving. See scraping JavaScript sites.
Lines that don't change with scale
- Respect robots.txt and each site's terms.
- Don't collect personal data without a lawful basis.
- Don't beat CAPTCHAs or get round blocks: scale makes the legal risk bigger, not smaller. See scraping anti-bot websites.
- Don't overload sites: the combined speed of all your proxies against one site is still your speed.
Where 65Proxy fits
Plainly: a large, multi-country scraping system shouldn't rely on 65Proxy's 4G proxies. We only have Singapore IPs, one IP per proxy at a time, rotated at most every 5 minutes. Where we fit is the small group of IP-sensitive Singapore sites (where datacenter and many residential IPs get blocked) at a moderate pace. Priced 1 day $4, 7 days $13, 1 month $32 (200GB each), plus monthly plans with more bandwidth (500GB $40, 1TB $65); 4G proxy with 500GB covers how to size one. For the rest, use rotating residential and datacenter proxies from providers built for scale.
Frequently asked questions
How many IPs for a million pages a day?
It depends on how many domains and the polite pace for each. A million pages across thousands of domains differs from a million pages on one domain. The latter almost certainly needs an API or an agreement with the owner. The basic maths is in how many proxies do I need.
Is per-request rotating residential always better?
No. For flows that need a session (carts, cookie-based pagination), a new IP per request breaks the session. Use sticky sessions for those.
Should I build my own proxies on cloud servers?
For sites that don't block servers, yes: cheap and under your control. But cloud IPs are datacenter IPs, so hard sites still need residential or mobile proxies.