Scraping Google and Bing search results with a proxy — and when to use the API instead
In this article
Reading search results with a program — for rank tracking, keyword research or market research — is one of the hardest scraping jobs there is. Search engines block aggressively, and the results they show depend heavily on where the request comes from. A proxy changes where you appear, which matters for local results, but it won't turn Google into an open data source. This guide is honest about when to scrape SERPs, when to use an official API instead, and how to do it without hammering anyone.
Why search engines are hard to scrape
Search results aren't meant to be read by bots, and it shows:
- Strong anti-bot systems throw CAPTCHAs and blocks the moment traffic looks automated.
- Results are localised and personalised — by country, language, and sometimes history — so there is no single "the ranking".
- The HTML changes often, so a parser that worked last month can break.
- It's against the search engines' terms to scrape results, which carries its own risk.
Changing your IP helps with the "where" but does nothing about pace, parsing or terms. It's one piece, not a solution.
Try the official API first
Before scraping, check whether an API gives you what you need — it's more stable and more legitimate:
- Google Programmable Search / Custom Search JSON API returns structured results for a configured scope.
- Bing Web Search API returns results and metadata directly.
- Google Ads Keyword Planner covers search volume and keyword ideas.
- Google Search Console gives you your own site's real positions and clicks — the best source for your own rankings.
If one of these fits, you avoid the whole blocking problem.
When a proxy genuinely helps
A proxy earns its place when you need results as a searcher in Singapore sees them:
- Local rank tracking for your own keywords in the Singapore market.
- Checking the local pack and local competitors that only show to Singapore users.
- Ad verification — seeing which ads run on your keywords in Singapore (see proxies for ads).
A Singapore IP with a Singapore locale shows you the local SERP instead of your own country's.
Doing it without getting blocked
Pace and consistency matter far more than the number of IPs:
- One query at a time, at a human pace — a few seconds apart, not hundreds a minute.
- Set the locale consistently (country and language) so results are stable and comparable.
- Keep a session on one IP, and rotate only when you actually get limited (each 65Proxy proxy rotates at most once every 5 minutes).
- Parse defensively and expect layout changes.
The general playbook is in how to scrape without getting blocked and proxies for web scraping.
65Proxy Singapore 4G for SERP checks
We sell real Singapore mobile IPs — right for seeing local Singapore results and tracking your own keywords at a moderate pace. HTTP and SOCKS5, rotation by link, 200GB per plan, priced 1 day $4, 7 days $13, 1 month $40. For large-scale SERP data across many countries, a dedicated SERP API is usually more economical and reliable — we'll say so rather than sell you the wrong tool. See pricing.
Frequently asked questions
Is scraping Google results legal?
It's against Google's terms of service, and the legal picture varies by country. Public results read at a reasonable pace carry less risk, but the official API is both safer and more stable — prefer it where it fits.
Why do I see different results than my client?
Results are localised and personalised. A fresh session on a Singapore IP, with a Singapore locale, gets you closer to a neutral local view than your own logged-in browser.
How many queries can I run?
There's no fixed number — pace matters more than IP count. Spread queries out and back off on the first sign of limiting; see how to scrape without getting blocked.