---
title: "How to Scrape Dynamic Websites (JavaScript) | Scrape.ly"
description: "Scrape dynamic, JavaScript-rendered websites: find embedded JSON and hidden APIs, or render pages in a headless browser. Tested code and three real runs."
url: https://scrape.ly/blog/scrape-dynamic-websites
markdown: https://scrape.ly/blog/scrape-dynamic-websites.md
---

[Guides](https://scrape.ly/blog/category/guides.md)

# How to Scrape Dynamic Websites: JavaScript Pages, Hidden APIs and Headless Browsers

How to scrape dynamic, JavaScript-rendered websites: tell whether a page needs a browser, read the data from embedded JSON or the site's own API, and render the rest in a headless browser. Tested Python code and three real runs from 27 October 2026 with their exact cost.

[Scrape.ly Team The team behind Scrape.ly](https://scrape.ly/blog/author/scrapely-team.md)

- Published

  Oct 27, 2026

- Reading time

  7 min read

![Pixel-art robot with an orange >\_ on its terminal head pulls a data card out of a browser window whose page is still rendering, its top drawn and the rest grey placeholder blocks around an orange loading spinner, and drops it into a crate](https://scrape.ly/media/scrape-dynamic-websites.png?w=1080)

To scrape a dynamic website, first find out where its data really lives. A JavaScript page often ships the data inside the HTML as JSON, or loads it from the site's own API, and reading that directly is faster and cheaper than rendering anything. Only when neither works do you need a headless browser that runs the page's JavaScript and waits for the content to appear.

This guide shows how to tell a static page from a dynamic one in a minute, the three ways to get the data (embedded JSON, the hidden API and a headless browser), tested Python for the first two, and three real runs from 27 October 2026 on the same JavaScript-rendered page: plain HTTP with CSS selectors (no data), the same selectors in Chrome (ten quotes), and plain HTTP reading the embedded JSON (ten quotes, in less time).

## What makes a website dynamic

A static page arrives from the server with its content already in the HTML. A dynamic page arrives as a shell, and JavaScript running in the browser fills it in: it reads data from a script tag, calls an API, or both. Single-page apps built with React, Vue or Angular work this way, and so do infinite-scroll feeds, product grids with filters and most dashboards.

A plain HTTP request, which is what `requests`, `curl` or BeautifulSoup see, gets only the shell. Your selectors match nothing, not because they're wrong, but because the elements don't exist yet.

### How to tell in a minute

1. Open the page in your browser, right-click and choose **View page source** (not Inspect). Search for a piece of text you can see on the page.
2. If the text is in the source, the page is static enough for plain HTTP.
3. If it isn't, search the source for a distinctive value (a price, a name) inside a `<script>` tag. If it's there, the data is embedded as JSON.
4. If it's nowhere in the source, open DevTools → **Network** → **Fetch/XHR**, reload, and look for a request whose response holds the data. That's the site's own API.

We used [quotes.toscrape.com/js](https://quotes.toscrape.com/js/), a test site built for scraping practice. In a browser it shows ten quotes. Its HTML source contains zero elements with the class `quote`, and one script that starts with `var data = [`.

## Method 1: read the embedded JSON

Many JavaScript pages carry their initial data in the HTML: a `var data = [...]` assignment, a Next.js `__NEXT_DATA__` script, a `window.__INITIAL_STATE__` object or JSON-LD for search engines. Read it with one regular expression and `json.loads`, no browser needed. Tested on 27 October 2026 with Python's standard library:

```python
import json
import re
import urllib.request

html = urllib.request.urlopen("https://quotes.toscrape.com/js/").read().decode()
data = json.loads(re.search(r"var data = (\[.*?\]);", html, re.S).group(1))
for q in data[:3]:
    print(q["author"]["name"], "|", q["text"][:60])
print(len(data), "quotes")
```

Output:

```plaintext
Albert Einstein  “The world as we have created it is a process of our thinkin
J.K. Rowling  “It is our choices, Harry, that show what we truly are, far
Albert Einstein  “There are only two ways to live your life. One is as though
10 quotes
```

Embedded JSON is usually richer than the rendered page: here each quote also carries the author's slug and Goodreads link, which the page never displays.

## Method 2: call the site's own API

When the data isn't in the source, the page fetched it. The same test site has an infinite-scroll version whose JavaScript calls `/api/quotes?page=N`. Find such calls in the Network tab, right-click one and copy it as cURL to see the URL, parameters and headers. Then page through it directly:

```python
import json
import urllib.request

page, quotes = 1, []
while True:
    url = f"https://quotes.toscrape.com/api/quotes?page={page}"
    data = json.load(urllib.request.urlopen(url))
    quotes += data["quotes"]
    if not data["has_next"]:
        break
    page += 1
print(len(quotes), "quotes from", page, "pages;", quotes[0]["author"]["name"])
```

Output, on 27 October 2026:

```plaintext
100 quotes from 10 pages; Albert Einstein
```

Ten small JSON responses gave every quote on the site. That's the fastest path, and it's how most ready-made scrapers for big sites work. On real sites, internal APIs often need headers the page sends (a token, a cookie, a client version), change without notice, and are protected by the same anti-bot systems as the pages. Copy only what the page itself sends.

## Method 3: render the page in a headless browser

When the data is assembled in the browser from many requests, encrypted or signed, a browser is the reliable route: it runs the page's JavaScript, and you read the finished DOM. Locally that's Playwright, Puppeteer or Selenium; our guides to [headless browsers](https://scrape.ly/blog/what-is-a-headless-browser.md) and [Playwright vs Puppeteer](https://scrape.ly/blog/playwright-vs-puppeteer.md) compare them. You'll also need to wait for the right element, scroll for lazy content, block images to save bandwidth, and run it all somewhere that isn't your laptop.

On Scrape.ly, the [Web Scraper](https://scrape.ly/store/querystone/web-scraper.md) runs your page function on each page, over plain HTTP or, with `useChrome`, in a hosted Chrome after the page's scripts have run. The page function below uses jQuery, which the Actor injects into every page, and returns one row per quote.

## Three real runs on the same page

We ran the Web Scraper three times on 27 October 2026, one page each.

**Run 1: CSS selectors over plain HTTP.** This is the request most tutorials start with:

```javascript
async function pageFunction(context) {
    const $ = context.jQuery;
    return $('.quote').map((i, el) => ({
        text: $(el).find('.text').text(),
        author: $(el).find('.author').text(),
        tags: $(el).find('.tag').map((j, t) => $(t).text()).get(),
    })).get();
}
```

```plaintext
20261027T17:26:39.461Z [scrapely.actor] INFO  Done: 1 pages fetched, 0 records saved in 2 s
20261027T17:26:39.461Z [scrapely.actor] ERROR [Scrapely] Actor failed: No results: the page function returned no data for 1 page.
```

Zero records, because the HTML had no `.quote` elements. The run cost nothing: no results, no charge.

**Run 2: the same page function with&#x20;**__`"useChrome": true`__ **.** Chrome ran the page's script before the function looked for quotes:

```plaintext
20261027T17:26:47.516Z [scrapely.actor] INFO  Done: 1 pages fetched, 10 records saved, Chrome avg per page: open 0.6s goto 1.8s idle 0.5s scroll 0.8s snapshot 0.3s in 7 s
```

Ten quotes, the first by Albert Einstein, with text and tags. The run took 17 seconds end to end.

**Run 3: plain HTTP, reading the embedded JSON instead:**

```javascript
async function pageFunction(context) {
    const $ = context.jQuery;
    const script = $('script').map((i, el) => $(el).html()).get().find((s) => s.includes('var data ='));
    const json = script.slice(script.indexOf('['), script.indexOf('];') + 1);
    return JSON.parse(json).map((q) => ({ text: q.text, author: q.author.name, tags: q.tags }));
}
```

```plaintext
20261027T17:26:42.151Z [scrapely.actor] INFO  Done: 1 pages fetched, 10 records saved in 4 s
```

The same ten quotes, in the same order, in 7 seconds end to end instead of 17, with no browser at all.

| Run              | Transport | Records | Run time | Charged  |
| ---------------- | --------- | ------- | -------- | -------- |
| 1. Selectors     | HTTP      | 0       | 12 s     | $0       |
| 2) Selectors     | Chrome    | 10      | 17 s     | $0.00572 |
| 3. Embedded JSON | HTTP      | 10      | 7 s      | $0.00572 |

The lesson generalizes: try the source first, then the API, and render only what you must. Browsers are slower and heavier per page, which adds up at a million pages.

## Running dynamic scrapes at scale

The code above works for one page. Production adds the parts that take the time:

- **Waiting correctly.** Wait for the element you need, not a fixed number of seconds. In the Web Scraper, `waitUntil` (`networkidle2` by default) and your page function's own checks do that.
- **Infinite scroll and "load more".** Scroll until nothing new loads, or call the API the button calls. `maxScrollHeightPixels` sets how far Chrome scrolls.
- **Blocking.** Sites that render with JavaScript often also fingerprint browsers. Scrape.ly runs pages through datacenter or residential proxies and a hosted [anti-blocking](https://scrape.ly/anti-blocking.md) browser; our guides to [Playwright stealth](https://scrape.ly/blog/playwright-stealth.md) and [getting past CAPTCHAs](https://scrape.ly/blog/captcha-solving-for-scraping.md) explain what's going on.
- **Schedules and outputs.** Save the input as a task, schedule it, and export JSON, CSV or Excel, or send results to a webhook.

If the site is a popular one, check the [Store](https://scrape.ly/store.md) first. Ready-made Actors for [Amazon](https://scrape.ly/blog/how-to-scrape-amazon.md), [Zillow](https://scrape.ly/blog/scrape-zillow-data.md) or [Booking.com](https://scrape.ly/blog/scrape-booking-com.md) already know where each site keeps its data, which is method 1 or 2 done for you and kept working when the site changes.

Scrape.ly Store Web Scraper Run your own page function on any site, over HTTP or in a hosted Chrome after the page's scripts run. Follow links, set limits and export JSON, CSV or Excel.

[Try it free](https://scrape.ly/store/querystone/web-scraper.md) [Browse the Store](https://scrape.ly/store.md)

## What it costs

The Web Scraper charges one event per record your page function returns, platform usage (compute, proxy and browser) included. Price checked on the Store page on 27 October 2026: **$0.000715 per result**, which is $0.715 per 1,000 records, whether a page loads over HTTP or in Chrome.

**Our runs:** run 1 returned nothing and cost nothing; runs 2 and 3 each returned 10 records, 10 × $0.000715 = $0.00715 at list price ($0.00572 charged, because our plan has a Store discount).

The Free plan's $5 of monthly usage covers about 7,000 records.

## What to watch out for

- **Rendered isn't always complete.** A page can show a skeleton for a second before data arrives. Wait for a real element, and check that the count of rows matches what you see.
- **Embedded JSON drifts too.** Key names change with front-end releases. Fail loudly when a key is missing instead of saving empty rows.
- **Internal APIs aren't public APIs.** Read the site's terms and robots.txt before you call them at volume; our [robots.txt guide](https://scrape.ly/blog/robots-txt-web-scraping.md) explains the rules.
- **One browser per page is expensive locally.** At scale, memory and proxies cost more than code. That's the case for running it in the cloud.

## FAQ

### How do I know if a website is dynamic?

View the page source and search for text you can see on the page. If it's missing, the page is rendered by JavaScript. Then check the source for JSON in a script tag, and the Network tab for API calls.

### Can BeautifulSoup scrape dynamic websites?

Not on its own. BeautifulSoup parses the HTML it's given, and a dynamic page's HTML doesn't contain the rendered content. Pair it with the embedded JSON or the site's API, or with a headless browser that hands it the rendered HTML.

### Do I always need Selenium or Playwright for JavaScript pages?

No. In our test, the embedded JSON gave the same ten quotes over plain HTTP in less than half the time. Use a browser only when the data isn't in the source or reachable through an API.

### How do I scrape infinite scroll pages?

Either find the API request the page makes as you scroll and page through it directly, as in method 2, or render the page in a browser and scroll until no new content loads.

### Can Scrape.ly scrape JavaScript-rendered pages?

Yes. The Web Scraper's `useChrome` option loads each page in a hosted Chrome and runs your page function after the scripts, and most Store Actors for big sites read the site's data directly.

Scrape.ly Store Scrape a JavaScript page Paste a URL, write a page function, and switch on Chrome when the page needs JavaScript. $0.715 per 1,000 records, platform usage included.

[Try it free](https://scrape.ly/store/querystone/web-scraper.md) [Browse the Store](https://scrape.ly/store.md)

About the author

[Scrape.ly Team](https://scrape.ly/blog/author/scrapely-team.md) The team behind Scrape.ly

Scrape.ly is the web data platform from TeraShift GmbH in Switzerland, powered by Evomi: ready-made scrapers in the Scrape.ly Store, a cloud runtime, an API with JavaScript and Python SDKs, proxies, a hosted anti-bot browser and an MCP server for AI agents.

- [Website](https://scrape.ly/index.md)

Keep reading

## More in Guides

- [![Pixel-art robot with an orange >\_ on its terminal head carries a data card towards a crate at the end of a cable from a server, above which a grey cloud holds a browser window showing a web page](https://scrape.ly/media/what-is-a-cloud-browser.png?w=750)](https://scrape.ly/blog/what-is-a-cloud-browser.md)
  ### [What Is a Cloud Browser? How Hosted Browsers Work and Why Scrapers Use Them](https://scrape.ly/blog/what-is-a-cloud-browser.md)
  [A cloud browser is a web browser that runs on a provider's servers while you control it remotely. The two kinds, how a hosted browser works over CDP (with tested code), local headless vs cloud browser, use cases, real run costs and how providers compare.](https://scrape.ly/blog/what-is-a-cloud-browser.md)
- [![Pixel-art robot with an orange >\_ on its terminal head walks away from a solved CAPTCHA panel, its grid of sky-and-road picture tiles with traffic lights, three tiles picked in orange and its checkbox ticked green, carrying a data card towards a crate](https://scrape.ly/media/captcha-solving-for-scraping.png?w=750)](https://scrape.ly/blog/captcha-solving-for-scraping.md)
  ### [CAPTCHA Bypass for Web Scraping: Why CAPTCHAs Appear and How Scrapers Handle Them](https://scrape.ly/blog/captcha-solving-for-scraping.md)
  [CAPTCHA bypass for scrapers, done responsibly: the CAPTCHA types you'll meet, why they appear, how proxies, fingerprints and pacing prevent most of them, tested detection code, what solving services cost, and a hosted browser that solves Cloudflare Turnstile for you.](https://scrape.ly/blog/captcha-solving-for-scraping.md)
- [![Pixel-art robot with an orange >\_ on its terminal head pulls a data card out of a monitor showing the Java coffee cup logo with orange steam above lines of code, and drops it into a crate](https://scrape.ly/media/java-web-scraping.png?w=750)](https://scrape.ly/blog/java-web-scraping.md)
  ### [Java Web Scraping: Jsoup, HttpClient, Playwright and Selenium (Tested Code)](https://scrape.ly/blog/java-web-scraping.md)
  [A practical guide to Java web scraping: Jsoup for HTML, java.net.http for polite crawling with retries, virtual threads for speed, Playwright and Selenium for JavaScript pages, and calling cloud scrapers from Java. Every snippet tested on JDK 25 on 26 October 2026.](https://scrape.ly/blog/java-web-scraping.md)

## The internet is full of data. Start owning yours.

Scrape.ly gives you the infrastructure, the tooling, and the team to extract, process, and operationalize web data at any scale. Setup takes minutes.

[Get Started Free](https://my.scrape.ly/register) [Chat with Support](mailto:hello@scrape.ly?subject=Question%20about%20Scrape.ly\&body=Hi%2C%20I%20have%20a%20question%20about%20Scrape.ly.)

No credit card. No setup calls.