Skip to content
Esc

Articles

What is a proxy? A plain-English guide for beginners: how it works, the types, and what it's forWhat is a 4G proxy? How mobile proxies work, pros, cons and when to use oneRotating proxies explained: rotation types and how to change a proxy IP with a linkSOCKS5 vs HTTP proxies: what's the difference and which should you use?Proxy vs VPN: what's the difference, and when should you use each?Static vs rotating proxies: how they differ, pros and cons, and when to use eachDedicated vs shared proxies: how they differ, and what "dedicated" means for a 4G proxyIPv4 vs IPv6 proxies: how they differ, why IPv6 proxies are cheap, and when they don't workHow do websites detect proxies? 8 signals anti-bot systems look atProxy glossary A to Z: 35 terms you'll meet when buying and using proxies, explained simplyProxy vs antidetect browser: how they differ, and do you need a proxy if you have antidetect?What is a residential proxy? How it works, pros and cons, and when to use oneWhat is a datacenter proxy? Pros and cons, why it's cheap, and when it's still the right choiceHow to find your IP address on a computer or phone: public IP, local IP, and your IP behind a proxyWhy do you keep getting CAPTCHAs? 9 common triggers, and where your IP fits inreCAPTCHA vs hCaptcha vs Cloudflare Turnstile: how each one decides who gets challengedWhat is botting? Good bots, bad bots, and where your own automation fitsHow to get a Singapore IP address: VPN, proxy, cloud server or SIM, and how to check itProxies for Facebook: managing several accounts more reliably with 4G IPsProxies for TikTok and TikTok Shop: when you need a Singapore IP and how to use it properlyProxies for advertising: checking and managing ads aimed at SingaporeProxies for e-commerce sellers: see marketplaces like a buyer, track competitors, keep store logins steadyProxies for web scraping: which type to choose, how to rotate, and how to play fairDo you need a proxy to buy tickets on ThaiTicketMajor, Ticketmelon or SISTIC?Scraping Google and Bing search results with a proxy, and when to use the API insteadHow to scrape without getting blocked: pace, sessions and clean IPsProxies for price monitoring: tracking competitor and retail prices across regionsScraping anti-bot and anti-proxy websites (Cloudflare, Akamai, DataDome): what works, and the limitsIs web scraping legal? Robots.txt, terms, personal data and copyright (Vietnam, Singapore and beyond)What are 4G proxies used for? 10 jobs mobile proxies do well, and the ones to avoid4G proxies for testing apps and websites as a Singapore mobile user: QA, ads, payments, bot rulesProxies for Instagram: managing brand and client accounts steadily with 4G IPsUsing a proxy to check flight and hotel prices as a Singapore user: when it helps and what to knowResearching the Singapore market with a proxy: see search, ads, marketplaces and competitors as a local userProxies for LinkedIn: managing B2B accounts and reaching the Singapore market the right wayProxies for X (Twitter): managing brand and client accounts steadily and within the rulesProxies for airdrops: why farming wallets through proxies gets disqualified, and how to join airdrops safelyProxies for seeding, bought likes and fake reviews: why they get caught, the risks, and lasting ways to growProxies for flash sales and limited drops: what a proxy can and can't do, and how to shop sales properlyProxies for large-scale, multi-country scraping: choosing proxy types, system design and real costsProxies for Netflix, YouTube and foreign streaming: why they usually fail, cost a lot, and what to do insteadSelling to Singapore from Vietnam: choosing channels, GST, shipping, payments and the tools you needSelling on TikTok Shop Singapore from abroad: seller requirements, workable routes and what a proxy can doThe Singapore online shopping market: who shops, where, how they pay and the big sale seasonsA workflow for agencies managing client accounts with proxies and antidetect: inventory, access, offboardingLocal SEO in Singapore: checking Google and Google Maps rankings as a Singapore user, the right wayProxies for Google Ads: when you need one, the official tools to use first, and the policy lines to knowRoblox lagging, high ping: does a proxy help, and the fixes that really reduce Roblox lagMultiple Roblox accounts (alts): what the rules allow and forbid, whether you need a proxy, and IP bansLag and high ping on Singapore game servers: do proxies or VPNs lower ping, and what actually helpsDiscord proxy: what it can and can't do, how to set it up, and managing communities the right wayShopee and Lazada Singapore proxies: see what Singapore buyers see, run your shop, track competitorsProxies for online business (MMO): which jobs need one, which type to pick and what to avoidProxy for bots and automation: one identity per IP, a polite pace, and what to do when a CAPTCHA appearsHow to set up a proxy on Windows, macOS, iPhone and Android (with a username and password)How to add a proxy to GoLogin, AdsPower, Hidemyacc, GenLogin and MultiloginRotate a proxy IP automatically with an API: curl, Python and Node.js examplesHow to check a proxy: IP, country, carrier, speed and DNS and WebRTC leaksCommon proxy errors and how to fix them: 407, timeouts, ERR_PROXY_CONNECTION_FAILEDScraping JavaScript-rendered sites with a headless browser and a proxy403, 429 and 503 errors when scraping: causes and how to handle them properly (with Python code)Using a proxy with Scrapy: authenticated setup, polite throttling, error fixes and rotating between runsUsing a proxy with curl and wget: HTTP, SOCKS5, authentication, environment variables and debuggingUsing a proxy in Python: requests, httpx and aiohttp with HTTP, SOCKS5 and authenticationSetting up a proxy in Chrome and Edge: a browser-only proxy without changing the whole computerSingapore proxy for Telegram: set up SOCKS5 on desktop, iPhone and Android, and how it differs from MTProtoSlow 4G proxy? How to measure speed and latency properly, and what really makes a proxy fasterUsing proxies safely: protect your proxy password, rotation link, accounts and dataUsing Proxifier for software without proxy support: SOCKS5 setup, per-app rules and DNSRoblox proxies for developers: calling Roblox APIs from HttpService, Open Cloud, and a safe self-hosted relayProtecting your Roblox account: cookie theft, "free Robux", malicious extensions, and how to stay safeHow to set up a proxy in AdsPower: HTTP or SOCKS5, where to paste it, checking the Singapore IP and rotatingHow to add a proxy to GoLogin: HTTP or SOCKS5, where to enter it, checking the Singapore IP and rotatingHow to add a proxy to GenLogin: HTTP or SOCKS5, where to paste it, checking the Singapore IP and rotatingHow to add a proxy to Hidemium: HTTP or SOCKS5, where to enter it, checking the Singapore IP and rotatingHow to add a proxy to BitBrowser: HTTP or SOCKS5, where to enter it, checking the Singapore IP and rotatingCAPTCHA keeps appearing or loops forever: 8 fixes, in the order to try themMobile vs residential vs datacenter proxies: comparing 4G, home and server IPs, and how to chooseSingapore proxies: when you need a Singapore IP and which type to chooseClean proxies and blacklists: what a clean IP means, and why "never blocked" is a myth4G rotating proxies: how mobile IP rotation works, what clean rotation means and how to choose oneHow to tell a real 4G proxy from a fake: check the carrier, IP type, latency and rotationFree vs paid proxies: the real risks of free proxies, and when paying is worth itDedicated 4G proxies, not shared: why a proxy only you use matters, and how to check it yourselfHow many proxies do I need? Working out the number for accounts, teams and scrapingStatic IP proxies for allowlists, APIs and remote access: 5 options and how to choose the right oneVietnam or Singapore proxy? Choosing which country's IP fits your workSingtel, M1 and StarHub proxies: which Singapore carrier a 4G proxy runs on, and how to check itClean proxy and CAPTCHA: how IP reputation decides how many challenges you seeThe real cost of a dirty proxy: CAPTCHAs, retries, wasted bandwidth and blocked accountsSingapore SOCKS5 proxy: when to use it, how to set it up, and how to check it5G proxy vs 4G proxy: what changes, what doesn't, and whether 5G is worth paying more forCheap 4G proxies: $4 gets you 200GB, and how to compare mobile proxy prices properly before you buyRefer a friend and earn 10% on every proxy order: how the 65Proxy referral program worksWhy choose 65Proxy? 8 reasons you can verify, and when you should go elsewhereHow to buy a 4G proxy at 65Proxy: sign up, top up, pay, connect and rotate, step by stepWhere to buy a reliable proxy? 12 things to check before paying any providerTesting a 4G proxy in a 1-hour trial: a minute-by-minute checklistBuy proxies with USDT: paying for a Singapore 4G proxy over TRC20 or BEP20, step by step4G proxy with 500GB: what the plan includes, how far 500GB goes, and how bandwidth is countedGuides to 4G mobile proxies

Proxies for web scraping: which type to choose, how to rotate, and how to play fair

By use case 4 min read

In this article
  1. 01 First: do you need a proxy at all?
  2. 02 Choosing a proxy for the job
  3. 03 Rotating the IP sensibly
  4. 04 Rate limits and retries
  5. 05 Playing fair
  6. 06 Example: Python with a proxy and a rate limit
  7. 07 Are 65Proxy's Singapore 4G proxies a fit?
  8. 08 Frequently asked questions

Web scraping means using a program to read public data from websites: product prices, listings, search results. As your request volume grows, sites start limiting or blocking your IP. That's when people think of proxies. But a proxy isn't always needed, and the wrong type costs money and still gets blocked. This guide helps you decide whether you need a proxy, which type, how to rotate, and how to collect data responsibly.

First: do you need a proxy at all?

Often not:

  • The site has an API or open data: an official API is always more stable and legitimate than parsing HTML.
  • Small volumes: a few hundred pages a day at a gentle pace usually work fine from your own IP.
  • Other sources exist: RSS, sitemaps, ready-made exports.

A proxy is needed when you want country-specific data (prices or search results in Singapore), or when your reasonable volume still exceeds what one IP is allowed.

Choosing a proxy for the job

Need Proxy type Why
High volume, sites that don't block server IPs Datacenter Cheap, fast, many IPs
Many pages and countries, varied IPs Residential, rotating per request Very large IP pools
Singapore data, sites that check IPs closely, moderate volume 4G mobile Trusted IPs on a real mobile network

A 4G proxy's strength is trust, not quantity: at any moment one proxy is one IP. The full comparison is in 4G vs residential vs datacenter proxies.

Rotating the IP sensibly

A rotating proxy changes the IP per request, on a timer or when you call a link. Whichever kind you use:

  • Don't rotate for no reason: a new IP for every request doesn't make you "invisible"; it just fragments your sessions.
  • Rotate when limited: when a site starts returning 429s or captchas, slow down first, then rotate. A fetcher that backs off and stops cleanly is in proxy for bots and automation.
  • Keep cookies with their IP: one session (cookies) should stay on one IP; a new IP means a new session.
  • With 65Proxy's 4G proxies: rotate with each proxy's own link, at most once every 5 minutes. How to call the link and wait for the new IP from code is in rotating the IP with an API.

Rate limits and retries

Sustainable scraping rests on a reasonable pace more than on the number of IPs:

  • Pause between requests (a few seconds, with some randomness) and cap parallel connections.
  • Honour Retry-After when a site returns 429 or 503; without it, back off progressively (5, 10, 20 seconds…).
  • Cache: don't re-download pages that haven't changed; use If-Modified-Since / ETag where the site supports them.
  • Run off-peak for the site, where you can.

Playing fair

  • Read the site's terms and robots.txt; respect what they don't allow.
  • Only collect public data: don't log in to scrape behind a login, don't bypass captchas.
  • Be careful with personal data: names, phone numbers and emails are protected by personal-data laws such as Singapore's PDPA; don't collect them without a lawful basis.
  • Don't overload sites: if your activity slows someone else's site down, you're harming them.

The legal lines are covered in detail in is web scraping legal; for sites behind Cloudflare or other anti-bot services, see scraping anti-bot websites. Using Scrapy? See using a proxy with Scrapy.

Example: Python with a proxy and a rate limit

The code below checks robots.txt, goes through the proxy, pauses between requests and backs off when limited. Install with pip install requests and set the PROXY environment variable to http://USER:PASS@HOST:PORT:

Python
import os
import random
import time
from urllib import robotparser

import requests

AGENT = "price-monitor/1.0 (contact: [email protected])"
PROXY = os.environ["PROXY"]

session = requests.Session()
session.proxies = {"http": PROXY, "https": PROXY}
session.headers["User-Agent"] = AGENT

robots = robotparser.RobotFileParser("https://example.com/robots.txt")
robots.read()


def wait_after(response, attempt):
    try:
        return int(response.headers["Retry-After"])
    except (KeyError, ValueError):
        return 5 * 2 ** attempt


def fetch(url, tries=4):
    for attempt in range(tries):
        try:
            r = session.get(url, timeout=30)
        except requests.RequestException:
            time.sleep(5 * 2 ** attempt)  # a slow network, or the proxy is rotating
            continue
        if r.status_code in (429, 503):
            time.sleep(wait_after(r, attempt))
            continue
        r.raise_for_status()
        return r.text
    raise RuntimeError("giving up after several tries: " + url)


for url in ["https://example.com/p/1", "https://example.com/p/2"]:
    if not robots.can_fetch(AGENT, url):
        continue
    html = fetch(url)
    # ... read the data you need from html ...
    time.sleep(random.uniform(2, 5))  # one request every few seconds

Are 65Proxy's Singapore 4G proxies a fit?

They fit when you need data as Singapore users see it, from sites that check IPs closely, at moderate volume (say, tracking a few hundred products on a Singapore marketplace each day). The plans by duration are priced 1 day $4, 7 days $13, 1 month $32 with 200GB each, and there are monthly plans with more bandwidth (500GB $40, 1TB $65).

If you need millions of pages a day, datacenter or per-request residential proxies are usually more economical (how to combine them is in proxies for large-scale, multi-country scraping). We say so plainly so you pick the right tool.

Frequently asked questions

It depends on the data, how you collect it and the site's terms. Public data, collected at a reasonable pace and free of personal data, usually carries less risk; for large projects or anything involving personal data, get legal advice.

How much bandwidth is enough?

It depends on page size. Measure a typical page (your browser's Network tab, or the total downloaded in your script), multiply by pages per day, then compare with the plan's bandwidth (200GB on the plans by duration, 500GB $40, 1TB $65 on the monthly plans with more). 4G proxy with 500GB walks through the measurement.

Should I use a headless browser (Playwright)?

Only when the page needs JavaScript to show the data. A browser uses far more bandwidth and resources than plain requests. Using a proxy with Playwright is covered in rotating the IP with an API.