Browser7

Web Scraping / Avoid Getting Blocked

How to Scrape Without Getting Blocked

You write a scraper that works perfectly - then after a few hundred requests, you start getting 403 errors, empty responses, or CAPTCHA pages. The site has identified your traffic as automated and blocked it. Here is how to avoid that.

Why websites block scrapers

IP reputation

Datacenter IPs are the most common sign of automated traffic. If your requests come from AWS, Google Cloud, or any hosting provider, websites can identify them immediately. Many sites block entire datacenter IP ranges outright, regardless of how your requests are structured.

Browser fingerprinting

Websites collect dozens of signals from your browser - screen resolution, installed fonts, WebGL renderer, canvas hash, audio context, and more. A default headless Chrome installation has a recognizable fingerprint that anti-bot systems flag instantly. Even with stealth plugins, maintaining a consistent, realistic fingerprint across requests is difficult.

Rate and pattern analysis

Real users do not request 50 pages in 10 seconds from the same IP. Anti-bot systems track request rates, navigation patterns, and session behavior. Requesting pages in a predictable sequence (every product in a category, every page of search results) is a strong signal that the traffic is automated.

TLS fingerprinting

Before your HTTP request even arrives, the TLS handshake reveals information about your client. Libraries like Python requests and Node.js fetch have TLS fingerprints that differ from real browsers. Some anti-bot systems reject connections based on TLS alone, before looking at any HTTP headers.

How Browser7 prevents blocks

Browser7 handles every layer of anti-detection automatically. Every request uses a residential IP, a real Chrome browser with a realistic fingerprint, and proper TLS negotiation. You do not need to configure any of this - it is the default behavior.

from browser7 import Browser7

client = Browser7(api_key="b7_your_api_key")

# Residential proxy, real browser fingerprint, automatic rotation
result = client.render(
    "https://www.amazon.com/dp/B0DDZJS3SB",
    country_code="US",
)

print(result.html)

That code is complete. There is no proxy configuration, no user agent string, no stealth plugin, and no fingerprint spoofing. Browser7 handles all of it:

  • Residential IPs only - every request routes through a real residential IP address, not a datacenter
  • Automatic IP rotation - IPs rotate between requests so no single address makes too many calls to one site
  • Real browser fingerprint - a full Chrome browser with consistent, realistic fingerprinting signals
  • Native TLS - the TLS handshake matches a real Chrome browser because it is a real Chrome browser

Force a fresh IP when you need one

If a particular request fails or returns unexpected content, you can force Browser7 to use a completely new residential IP for the next attempt. This is useful for retries or when scraping sites that are particularly aggressive about rate limiting individual IPs.

from browser7 import Browser7

client = Browser7(api_key="b7_your_api_key")

# Force a new residential IP for this request
result = client.render(
    "https://www.amazon.com/dp/B0DDZJS3SB",
    country_code="US",
    force_new_proxy=True,
)

print(result.html)

Practical tips for staying unblocked

Space out your requests

Even with residential proxies, hitting a site with hundreds of concurrent requests can trigger rate limits. Add reasonable delays between requests. Browser7 handles the proxy and fingerprint layer - you handle the pacing.

Use geo-targeting for local sites

If you are scraping a site that serves a specific country, use the country_code parameter to route your request through an IP in that country. A US IP hitting a Japanese site looks unusual. A Japanese IP hitting a Japanese site looks normal.

Do not scrape pages you do not need

The simplest way to avoid blocks is to make fewer requests. If you only need product prices, do not scrape every product page, review page, and image page. Fewer requests means less chance of triggering any rate limit or pattern detection.

What this costs

Residential proxies, fingerprint rotation, and anti-detection are included in every Browser7 request at $0.01 per page. There is no premium proxy tier, no stealth mode surcharge, and no credit multiplier for protected sites. The price is the same whether the target site has anti-bot protection or not.

See it in practice

These guides scrape sites with strong anti-bot protection:

Try it yourself

100 free renders with residential proxies and anti-detection included. No payment required.