Browser7

Guides / Scrape Twitter/X

How to Scrape Twitter/X in 2026

X (formerly Twitter) uses aggressive anti-bot detection, a JavaScript-heavy single-page application architecture, login prompts, and strict rate limiting. Scraping it yourself means battling constantly changing DOM structures and sophisticated fingerprinting. With Browser7, you get fully rendered profile data and tweets in a single API call.

What makes X hard to scrape

Aggressive anti-bot detection

X employs advanced browser fingerprinting, behavioral analysis, and request validation. Datacenter IPs are blocked immediately. Even residential proxies face challenges as X detects automated browsing patterns and headless browser signatures.

JavaScript-heavy SPA

X is built as a React single-page application. All content - profiles, tweets, replies, and engagement metrics - is loaded via JavaScript after the initial page shell. A simple HTTP request returns virtually no useful data. You need a real browser with full JavaScript execution.

Rate limiting

X enforces aggressive rate limits at both the IP and account level. Exceeding these limits results in temporary blocks, CAPTCHA challenges, or permanent IP bans. Even viewing profiles without an account has strict view limits.

Login prompts

X frequently shows login modals and "sign in to continue" prompts that block content for unauthenticated visitors. The amount of content visible without logging in varies and can change without notice.

Scrape an X profile

Browser7 handles proxy rotation, browser fingerprinting, CAPTCHA solving, and JavaScript rendering automatically. This example renders a public X profile page and returns the fully rendered HTML with profile data and recent tweets.

from browser7 import Browser7

client = Browser7(
    api_key="b7_your_api_key",
    base_url="https://ca-api.browser7.com/v1"
)

result = client.render(
    "https://x.com/elonmusk",
    country_code="US",
)

print(result.html)

That is the complete code. No proxy configuration, no browser setup, no login handling. The response contains the fully rendered HTML of the X profile page, including the display name, bio, follower counts, and recent tweets.

Data you can extract

The rendered HTML contains all the data X shows to a real visitor. X uses data-testid attributes on key elements, making extraction more reliable than class-based selectors. Common data points:

Profile info

  • Display name and username
  • Bio/description text
  • Profile and banner image URLs
  • Location and website link
  • Account creation date

Tweets

  • Tweet text content
  • Media attachments (images, videos)
  • Quote tweets and retweets
  • Tweet timestamps
  • Reply context and threads

Engagement

  • Like counts per tweet
  • Retweet and quote counts
  • Reply counts
  • View/impression counts
  • Bookmark counts

Account metadata

  • Follower count
  • Following count
  • Verified/premium badge status
  • Professional category
  • Pinned tweet

Complete example: render and parse profile data

Here is a complete example that renders an X profile page and extracts structured data. X uses data-testid attributes on key elements like UserName, UserDescription, and tweet, which makes parsing more reliable than class-based selectors.

from browser7 import Browser7
from bs4 import BeautifulSoup
import re
import json

client = Browser7(
    api_key="b7_your_api_key",
    base_url="https://ca-api.browser7.com/v1"
)

result = client.render(
    "https://x.com/elonmusk",
    country_code="US",
)

soup = BeautifulSoup(result.html, "html.parser")

profile = {
    "displayName": None,
    "username": None,
    "bio": None,
    "followers": None,
    "following": None,
}

# Display name and username from UserName test ID
name_el = soup.select_one('div[data-testid="UserName"]')
if name_el:
    spans = name_el.find_all("span")
    for span in spans:
        text = span.get_text(strip=True)
        if text.startswith("@"):
            profile["username"] = text
        elif text and not profile["displayName"]:
            profile["displayName"] = text

# Bio from UserDescription test ID
bio_el = soup.select_one('div[data-testid="UserDescription"]')
if bio_el:
    profile["bio"] = bio_el.get_text(strip=True)

# Following count
following_link = soup.select_one('a[href*="/following"]')
if following_link:
    text = following_link.get_text(strip=True)
    match = re.search(r"([\d,]+)", text)
    if match:
        profile["following"] = match.group(1)

# Followers count
followers_link = soup.select_one('a[href*="/followers"]')
if followers_link:
    text = followers_link.get_text(strip=True)
    match = re.search(r"([\d,.]+[MKB]?)", text)
    if match:
        profile["followers"] = match.group(1)

print(json.dumps(profile, indent=2))

CSS selectors may change if X updates their page structure. The data-testid attributes tend to be more stable than class names but can still change. Inspect the current page if any fields return null.

Sample output:

{
  "displayName": "Elon Musk",
  "username": "@elonmusk",
  "bio": "Read @America",
  "followers": "214.7M",
  "following": "1,013"
}

Extract recent tweets

The profile page contains the user's recent tweets. Each tweet is wrapped in an article[data-testid="tweet"] element, with the text content inside a div[data-testid="tweetText"] child.

from browser7 import Browser7
from bs4 import BeautifulSoup

client = Browser7(
    api_key="b7_your_api_key",
    base_url="https://ca-api.browser7.com/v1"
)

result = client.render(
    "https://x.com/elonmusk",
    country_code="US",
)

soup = BeautifulSoup(result.html, "html.parser")

# Extract recent tweets
tweets = []
for article in soup.select('article[data-testid="tweet"]'):
    text_el = article.select_one('div[data-testid="tweetText"]')
    if text_el:
        tweets.append(text_el.get_text(strip=True))

for i, tweet in enumerate(tweets[:5]):
    print(f"{i + 1}. {tweet[:120]}...")

Take a screenshot of the profile

Capture X profiles as images for social media monitoring, brand analysis dashboards, or archiving public statements and tweets.

import base64
from browser7 import Browser7

client = Browser7(
    api_key="b7_your_api_key",
    base_url="https://ca-api.browser7.com/v1"
)

result = client.render(
    "https://x.com/elonmusk",
    country_code="US",
    block_images=False,
    include_screenshot=True,
    screenshot_full_page=True,
    screenshot_format="png"
)

# Save the screenshot
with open("x-profile.png", "wb") as f:
    f.write(base64.b64decode(result.screenshot))

print("Screenshot saved")

What this costs

Every X page render costs $0.01 - the same as any other website. Residential proxies, JavaScript rendering, CAPTCHA solving, and screenshots are all included. There are no per-domain surcharges, no credit multipliers, and no bandwidth fees.

10,000 X profile pages costs $100. You know this before you start, not after.

Try it yourself

100 free renders - enough to test X scraping with no payment required.