How to Scrape Dynamic Websites: JavaScript Pages, Hidden APIs and Headless Browsers

To scrape a dynamic website, first find out where its data really lives. A JavaScript page often ships the data inside the HTML as JSON, or loads it from the site's own API, and reading that directly is faster and cheaper than rendering anything. Only when neither works do you need a headless browser that runs the page's JavaScript and waits for the content to appear.

This guide shows how to tell a static page from a dynamic one in a minute, the three ways to get the data (embedded JSON, the hidden API and a headless browser), tested Python for the first two, and three real runs from 27 October 2026 on the same JavaScript-rendered page: plain HTTP with CSS selectors (no data), the same selectors in Chrome (ten quotes), and plain HTTP reading the embedded JSON (ten quotes, in less time).

What makes a website dynamic

A static page arrives from the server with its content already in the HTML. A dynamic page arrives as a shell, and JavaScript running in the browser fills it in: it reads data from a script tag, calls an API, or both. Single-page apps built with React, Vue or Angular work this way, and so do infinite-scroll feeds, product grids with filters and most dashboards.

A plain HTTP request, which is what requests, curl or BeautifulSoup see, gets only the shell. Your selectors match nothing, not because they're wrong, but because the elements don't exist yet.

How to tell in a minute

1. Open the page in your browser, right-click and choose View page source (not Inspect). Search for a piece of text you can see on the page.
2. If the text is in the source, the page is static enough for plain HTTP.
3. If it isn't, search the source for a distinctive value (a price, a name) inside a <script> tag. If it's there, the data is embedded as JSON.
4. If it's nowhere in the source, open DevTools → Network → Fetch/XHR, reload, and look for a request whose response holds the data. That's the site's own API.

We used quotes.toscrape.com/js, a test site built for scraping practice. In a browser it shows ten quotes. Its HTML source contains zero elements with the class quote, and one script that starts with var data = [.

Method 1: read the embedded JSON

Many JavaScript pages carry their initial data in the HTML: a var data = [...] assignment, a Next.js __NEXT_DATA__ script, a window.__INITIAL_STATE__ object or JSON-LD for search engines. Read it with one regular expression and json.loads, no browser needed. Tested on 27 October 2026 with Python's standard library:

import json
import re
import urllib.request

html = urllib.request.urlopen("https://quotes.toscrape.com/js/").read().decode()
data = json.loads(re.search(r"var data = (\[.*?\]);", html, re.S).group(1))
for q in data[:3]:
    print(q["author"]["name"], "|", q["text"][:60])
print(len(data), "quotes")

Output:

Albert Einstein | “The world as we have created it is a process of our thinkin
J.K. Rowling | “It is our choices, Harry, that show what we truly are, far 
Albert Einstein | “There are only two ways to live your life. One is as though
10 quotes

Embedded JSON is usually richer than the rendered page: here each quote also carries the author's slug and Goodreads link, which the page never displays.

Method 2: call the site's own API

When the data isn't in the source, the page fetched it. The same test site has an infinite-scroll version whose JavaScript calls /api/quotes?page=N. Find such calls in the Network tab, right-click one and copy it as cURL to see the URL, parameters and headers. Then page through it directly:

import json
import urllib.request

page, quotes = 1, []
while True:
    url = f"https://quotes.toscrape.com/api/quotes?page={page}"
    data = json.load(urllib.request.urlopen(url))
    quotes += data["quotes"]
    if not data["has_next"]:
        break
    page += 1
print(len(quotes), "quotes from", page, "pages;", quotes[0]["author"]["name"])

Output, on 27 October 2026:

100 quotes from 10 pages; Albert Einstein

Ten small JSON responses gave every quote on the site. That's the fastest path, and it's how most ready-made scrapers for big sites work. On real sites, internal APIs often need headers the page sends (a token, a cookie, a client version), change without notice, and are protected by the same anti-bot systems as the pages. Copy only what the page itself sends.

Method 3: render the page in a headless browser

When the data is assembled in the browser from many requests, encrypted or signed, a browser is the reliable route: it runs the page's JavaScript, and you read the finished DOM. Locally that's Playwright, Puppeteer or Selenium; our guides to headless browsers and Playwright vs Puppeteer compare them. You'll also need to wait for the right element, scroll for lazy content, block images to save bandwidth, and run it all somewhere that isn't your laptop.

On Scrape.ly, the Web Scraper runs your page function on each page, over plain HTTP or, with useChrome, in a hosted Chrome after the page's scripts have run. The page function below uses jQuery, which the Actor injects into every page, and returns one row per quote.

Three real runs on the same page

We ran the Web Scraper three times on 27 October 2026, one page each.

Run 1: CSS selectors over plain HTTP. This is the request most tutorials start with:

async function pageFunction(context) {
    const $ = context.jQuery;
    return $('.quote').map((i, el) => ({
        text: $(el).find('.text').text(),
        author: $(el).find('.author').text(),
        tags: $(el).find('.tag').map((j, t) => $(t).text()).get(),
    })).get();
}

2026-10-27T17:26:39.461Z [scrapely.actor] INFO  Done: 1 pages fetched, 0 records saved in 2 s
2026-10-27T17:26:39.461Z [scrapely.actor] ERROR [Scrapely] Actor failed: No results: the page function returned no data for 1 page.

Zero records, because the HTML had no .quote elements. The run cost nothing: no results, no charge.

Run 2: the same page function with "useChrome": true. Chrome ran the page's script before the function looked for quotes:

2026-10-27T17:26:47.516Z [scrapely.actor] INFO  Done: 1 pages fetched, 10 records saved, Chrome avg per page: open 0.6s goto 1.8s idle 0.5s scroll 0.8s snapshot 0.3s in 7 s

Ten quotes, the first by Albert Einstein, with text and tags. The run took 17 seconds end to end.

Run 3: plain HTTP, reading the embedded JSON instead:

async function pageFunction(context) {
    const $ = context.jQuery;
    const script = $('script').map((i, el) => $(el).html()).get().find((s) => s.includes('var data ='));
    const json = script.slice(script.indexOf('['), script.indexOf('];') + 1);
    return JSON.parse(json).map((q) => ({ text: q.text, author: q.author.name, tags: q.tags }));
}

2026-10-27T17:26:42.151Z [scrapely.actor] INFO  Done: 1 pages fetched, 10 records saved in 4 s

The same ten quotes, in the same order, in 7 seconds end to end instead of 17, with no browser at all.

The lesson generalizes: try the source first, then the API, and render only what you must. Browsers are slower and heavier per page, which adds up at a million pages.

Running dynamic scrapes at scale

The code above works for one page. Production adds the parts that take the time:

• Waiting correctly. Wait for the element you need, not a fixed number of seconds. In the Web Scraper, waitUntil (networkidle2 by default) and your page function's own checks do that.
• Infinite scroll and "load more". Scroll until nothing new loads, or call the API the button calls. maxScrollHeightPixels sets how far Chrome scrolls.
• Blocking. Sites that render with JavaScript often also fingerprint browsers. Scrape.ly runs pages through datacenter or residential proxies and a hosted anti-blocking browser; our guides to Playwright stealth and getting past CAPTCHAs explain what's going on.
• Schedules and outputs. Save the input as a task, schedule it, and export JSON, CSV or Excel, or send results to a webhook.

If the site is a popular one, check the Store first. Ready-made Actors for Amazon, Zillow or Booking.com already know where each site keeps its data, which is method 1 or 2 done for you and kept working when the site changes.

What it costs

The Web Scraper charges one event per record your page function returns, platform usage (compute, proxy and browser) included. Price checked on the Store page on 27 October 2026: $0.000715 per result, which is $0.715 per 1,000 records, whether a page loads over HTTP or in Chrome.

Our runs: run 1 returned nothing and cost nothing; runs 2 and 3 each returned 10 records, 10 × $0.000715 = $0.00715 at list price ($0.00572 charged, because our plan has a Store discount).

The Free plan's $5 of monthly usage covers about 7,000 records.

What to watch out for

• Rendered isn't always complete. A page can show a skeleton for a second before data arrives. Wait for a real element, and check that the count of rows matches what you see.
• Embedded JSON drifts too. Key names change with front-end releases. Fail loudly when a key is missing instead of saving empty rows.
• Internal APIs aren't public APIs. Read the site's terms and robots.txt before you call them at volume; our robots.txt guide explains the rules.
• One browser per page is expensive locally. At scale, memory and proxies cost more than code. That's the case for running it in the cloud.

FAQ

How do I know if a website is dynamic?

View the page source and search for text you can see on the page. If it's missing, the page is rendered by JavaScript. Then check the source for JSON in a script tag, and the Network tab for API calls.

Can BeautifulSoup scrape dynamic websites?

Not on its own. BeautifulSoup parses the HTML it's given, and a dynamic page's HTML doesn't contain the rendered content. Pair it with the embedded JSON or the site's API, or with a headless browser that hands it the rendered HTML.

Do I always need Selenium or Playwright for JavaScript pages?

No. In our test, the embedded JSON gave the same ten quotes over plain HTTP in less than half the time. Use a browser only when the data isn't in the source or reachable through an API.

How do I scrape infinite scroll pages?

Either find the API request the page makes as you scroll and page through it directly, as in method 2, or render the page in a browser and scroll until no new content loads.

Can Scrape.ly scrape JavaScript-rendered pages?

Yes. The Web Scraper's useChrome option loads each page in a hosted Chrome and runs your page function after the scripts, and most Store Actors for big sites read the site's data directly.