Browser7

Use Cases / Market Research

Market Research

Good market research is built on data, and much of the data you need is publicly available on the web - competitor product catalogs, pricing strategies, customer reviews, market trends. Web scraping turns this scattered public information into structured datasets you can actually analyze.

What you can research

Competitor product catalogs

What products competitors offer, how they categorize them, what features they highlight, and how their catalog changes over time. Useful for identifying gaps in the market.

Pricing strategies

How competitors price across regions, how often they run promotions, what their pricing tiers look like, and how they position against each other. Data-driven pricing decisions start here.

Consumer sentiment

What customers say in reviews, forum discussions, and social media. Common praise and complaints reveal what the market values and what unmet needs exist.

Market sizing

Count listings, sellers, or products in a category to estimate market size. Track how these numbers change over time to understand market growth or contraction.

Scrape a competitor's product catalog

This example scrapes a Shopify store's product collection page to extract product names, prices, and URLs. The same approach works across any Shopify-powered store - they all share the same underlying structure.

from browser7 import Browser7
from bs4 import BeautifulSoup
import json

client = Browser7(api_key="b7_your_api_key")

result = client.render(
    "https://www.allbirds.com/collections/mens",
    country_code="US",
)

soup = BeautifulSoup(result.html, "html.parser")

products = []
for card in soup.select("div[data-product-card]"):
    title = card.select_one("p, h3")
    link = card.select_one('a[href*="/products/"]')

    price = None
    for el in card.find_all(["span", "p"]):
        text = el.get_text(strip=True)
        if "$" in text and len(text) < 20:
            price = text
            break

    products.append({
        "title": title.get_text(strip=True) if title else None,
        "price": price,
        "url": link.get("href", "") if link else None,
    })

print(json.dumps(products[:10], indent=2))

See our Shopify scraping guide for complete examples and reliable selectors.

A typical market research workflow

1. Identify data sources

Start with the websites that contain the data you need. For competitive analysis, that is competitor websites and the marketplaces where you both sell. For consumer sentiment, that is review sites, forums, and social platforms.

2. Extract and structure the data

Scrape each source and parse the HTML into structured fields - product names, prices, ratings, review text, dates. Browser7 gives you the rendered HTML; you choose the parsing approach (BeautifulSoup, Cheerio, DOMDocument, or any other parser).

3. Combine and analyze

Merge data from multiple sources into a single dataset. Compare prices across competitors, identify trends in customer feedback, spot gaps in product coverage. The value is in the analysis, not the individual data points.

4. Monitor changes over time

Market research is not a one-time activity. Schedule regular scrapes to track how competitor offerings, pricing, and customer sentiment change. Weekly or monthly snapshots build a timeline that reveals strategic shifts.

Why Browser7 works well for market research

Scrape any site with the same tool

Market research requires data from many different sources - each with different tech stacks, anti-bot measures, and rendering requirements. Browser7 handles all of them with the same API call. No per-site configuration, no maintaining different scrapers for different targets.

International market comparison

Research across markets means scraping from different countries. Browser7's geo-targeting with 195 countries lets you see what customers in any market see - local pricing, regional product availability, country-specific promotions.

Pay only when you research

Market research often happens in bursts - a competitive analysis before a product launch, a pricing study for a new market entry. Pay-as-you-go pricing means you pay during active research and nothing in between.

What this costs

Every page costs $0.01. A thorough competitive analysis - scraping 10 competitors across 5 product categories with 50 pages each - costs $25. Add customer review data from 3 review platforms and the total might be $50-75. That is the cost of a single market research report from a traditional agency, but the data is yours to analyze however you want.

Related guides

Try it yourself

100 free renders - enough to start your competitive analysis, no payment required.