--- title: "Selenium Web Scraping in Python: Step by Step | Scrape.ly" description: "Selenium web scraping in Python: headless Chrome, locators, explicit waits, pagination, infinite scroll, screenshots and CSV export, tested code for 2026." url: https://scrape.ly/blog/selenium-web-scraping markdown: https://scrape.ly/blog/selenium-web-scraping.md --- [Guides](https://scrape.ly/blog/category/guides.md) # Selenium Web Scraping in Python: Step-by-Step Guide Selenium web scraping in Python, step by step: install, headless Chrome, locators, explicit waits, clicking through pages, infinite scroll, screenshots and CSV/JSON export, tested with Selenium 4.51 on 23 October 2026. Plus when Playwright fits better and how to run it at scale. [Scrape.ly Team The team behind Scrape.ly](https://scrape.ly/blog/author/scrapely-team.md) - Published Oct 23, 2026 - Reading time 8 min read ![Pixel-art robot with an orange >\_ on its terminal head pulls a data card out of a browser window showing the green Selenium logo, which is cabled to a monitor showing the blue and yellow Python logo](https://scrape.ly/media/selenium-web-scraping.png?w=1080) Selenium web scraping means driving a real browser from your code with Selenium's WebDriver API, so you can collect data from pages that only show it after JavaScript runs. In Python that takes one `pip install selenium`, a headless Chrome, and explicit waits in the right places. This guide builds a working scraper step by step on two sandbox sites made for practice, [books.toscrape.com](https://books.toscrape.com/) and [quotes.toscrape.com](https://quotes.toscrape.com/): locators, waits, clicking through pages, infinite scroll, screenshots and export. Every snippet ran on 23 October 2026 with Python 3.14, Selenium 4.51.0 and Chrome 155 on macOS. ## Selenium, requests or Playwright? | | requests + BeautifulSoup | Selenium | Playwright | | --------------- | ------------------------- | ------------------------------------------------- | ----------------------------------------- | | Runs JavaScript | No | Yes, in a real browser | Yes, in a real browser | | Speed per page | Fastest: one HTTP request | Slower: full page load | Slower: full page load | | Waiting | Not needed | Explicit waits you write | Auto-waits before actions | | Languages | Python | Java, Python, C#, Ruby, JavaScript, Kotlin | TypeScript/JavaScript, Python, .NET, Java | | Best for | Static HTML and JSON APIs | Existing Selenium code, Ruby teams, Selenium Grid | New browser scrapers | If the data is already in the page source, you don't need a browser at all: our [Python web scraping guide](https://scrape.ly/blog/web-scraping-python.md) covers requests, BeautifulSoup and Scrapy. Use Selenium when the page is built by JavaScript or you need to click, scroll and type like a user. ## Step 1: Install Selenium ```shell python3 -m venv .venv source .venv/bin/activate pip install selenium ``` You don't download chromedriver anymore. Selenium Manager, which ships with Selenium, "automatically discovers, downloads, and caches the drivers required by Selenium when these drivers are unavailable" ([Selenium Manager docs](https://www.selenium.dev/documentation/selenium_manager/)). Since Selenium 4.11 it can also fetch Chrome itself, as a Chrome for Testing build, if Chrome isn't installed. In our test it found the installed Google Chrome and its driver on the first run, with no extra setup. ## Step 2: Start headless Chrome ```python from selenium import webdriver from selenium.webdriver.common.by import By def make_driver() -> webdriver.Chrome: options = webdriver.ChromeOptions() options.add_argument("--headless=new") options.add_argument("--window-size=1366,900") options.add_experimental_option( "prefs", {"profile.managed_default_content_settings.images": 2} ) return webdriver.Chrome(options=options) driver = make_driver() try: driver.get("https://books.toscrape.com/") print(driver.title) finally: driver.quit() ``` It prints `All products | Books to Scrape - Sandbox`. Three options matter for scraping: - **`--headless=new`** runs Chrome without a window, using the same flag as Selenium's own Chrome examples. - **`--window-size`** gives the page a predictable desktop layout. Responsive sites hide or move elements at smaller sizes, and your selectors should see the same page every run. - **The images preference** stops Chrome from downloading images. The data is in the HTML, so you save bandwidth and time on every page. Always call `driver.quit()` in a `finally` block. A crashed script otherwise leaves Chrome processes running in the background. ## Step 3: Find elements with locators Selenium finds elements with `find_element` (the first match, or a `NoSuchElementException`) and `find_elements` (a list, empty if nothing matches). You say how to search with a `By` strategy: | Locator | Example | When to use it | | ----------------- | ---------------------------- | ------------------------------------------ | | `By.CSS_SELECTOR` | `"article.product_pod h3 a"` | Your default: short, readable, fast | | `By.XPATH` | `"//li[@class='next']/a"` | Matching by text or walking up to a parent | | `By.CLASS_NAME` | `"price_color"` | A single class name | | `By.ID` | `"search"` | Elements with a unique `id` | | `By.LINK_TEXT` | `"next"` | Links by their exact visible text | | `By.TAG_NAME` | `"h1"` | Simple pages | Here are the first three on the books catalogue: ```python driver.get("https://books.toscrape.com/") books = driver.find_elements(By.CSS_SELECTOR, "article.product_pod") print(len(books), "books") first_book = books[0] print(first_book.find_element(By.CSS_SELECTOR, "h3 a").get_attribute("title")) print(first_book.find_element(By.CLASS_NAME, "price_color").text) print(driver.find_element(By.XPATH, "//li[@class='next']/a").get_attribute("href")) ``` ```plaintext 20 books A Light in the Attic £51.77 https://books.toscrape.com/catalogue/page-2.html ``` Two things to notice. Calling `find_element` on an element searches inside it, which keeps each book's fields together. And `get_attribute("href")` returns the absolute URL, so you don't need `urljoin` as you do with BeautifulSoup. `.text` returns only the visible text; use `get_attribute("textContent")` for hidden text. ## Step 4: Wait for JavaScript content with explicit waits Open [quotes.toscrape.com/js](https://quotes.toscrape.com/js/): the quotes are written into the page by a script after it loads. If you look for them right after `driver.get()`, they may not exist yet. Selenium's docs call this race between browser and script "one of the primary causes of flaky tests" ([waits docs](https://www.selenium.dev/documentation/webdriver/waits/)). The fix is an explicit wait: poll until a condition is true, or fail after a timeout. ```python from selenium.webdriver.support import expected_conditions as EC from selenium.webdriver.support.ui import WebDriverWait wait = WebDriverWait(driver, timeout=10) driver.get("https://quotes.toscrape.com/js/") cards = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "div.quote"))) ``` `wait.until` returns as soon as the elements exist, so a fast page costs you nothing. Useful conditions include `presence_of_element_located`, `visibility_of_element_located`, `element_to_be_clickable` and `staleness_of`. You can also pass any function that takes the driver, as the infinite-scroll step shows. Stick to explicit waits: the docs warn "Do not mix implicit and explicit waits. Doing so can cause unpredictable wait times." Avoid `time.sleep()` as your only wait. It's either too short on a slow day or wastes seconds on every page. ## Step 5: Scrape several pages by clicking Next Now put it together: read the quotes, click **Next**, wait for the new page, repeat. ```python from selenium.common.exceptions import NoSuchElementException, TimeoutException def scrape_js_pages(driver, max_pages=3): wait = WebDriverWait(driver, timeout=10) driver.get("https://quotes.toscrape.com/js/") quotes = [] for _ in range(max_pages): cards = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "div.quote"))) for card in cards: quotes.append({ "text": card.find_element(By.CSS_SELECTOR, "span.text").text, "author": card.find_element(By.CSS_SELECTOR, "small.author").text, "tags": [t.text for t in card.find_elements(By.CSS_SELECTOR, "a.tag")], }) try: next_link = driver.find_element(By.CSS_SELECTOR, "li.next > a") except NoSuchElementException: break next_link.click() wait.until(EC.staleness_of(cards[0])) # old page is gone return quotes ``` It returned 30 quotes from 3 pages, the first one Albert Einstein's "The world as we have created it is a process of our thinking," tagged change, deep-thoughts, thinking and world. The key line is `EC.staleness_of(cards[0])`. After the click, the old elements still exist for a moment. Without that wait, the next loop could read page 1 again, or crash with a `StaleElementReferenceException` when the old elements disappear mid-read. Waiting for the old card to go stale guarantees you read the new page. When the next link is a plain URL, as here, you can also read its `href` and call `driver.get()` instead of clicking. Clicking is what you need for buttons that trigger JavaScript. ## Step 6: Handle infinite scroll [quotes.toscrape.com/scroll](https://quotes.toscrape.com/scroll) loads 10 more quotes each time you reach the bottom. Scroll with JavaScript, wait until the count grows, and stop when it doesn't: ```python import time def scrape_infinite_scroll(driver, pause=1.0, max_rounds=20): driver.get("https://quotes.toscrape.com/scroll") wait = WebDriverWait(driver, timeout=10) wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "div.quote"))) seen = 0 for _ in range(max_rounds): driver.execute_script("window.scrollTo(0, document.body.scrollHeight);") try: WebDriverWait(driver, timeout=5).until( lambda d: len(d.find_elements(By.CSS_SELECTOR, "div.quote")) > seen ) except TimeoutException: break # no new quotes: we reached the end seen = len(driver.find_elements(By.CSS_SELECTOR, "div.quote")) time.sleep(pause) return [ { "text": card.find_element(By.CSS_SELECTOR, "span.text").text, "author": card.find_element(By.CSS_SELECTOR, "small.author").text, } for card in driver.find_elements(By.CSS_SELECTOR, "div.quote") ] ``` It collected all 100 quotes; the last was George R.R. Martin's "... a mind needs books as a sword needs a whetstone." The `max_rounds` cap protects you from feeds that never end, and the short pause gives the site's script time to settle between loads. Before you scroll, check DevTools' **Network** tab. This page fetches its quotes from `/api/quotes?page=N`, a JSON endpoint you can call directly with requests, no browser needed. Many infinite-scroll sites work the same way. ## Step 7: Take screenshots Screenshots are the fastest way to debug a headless scraper: you see exactly what Chrome saw. ```python driver.save_screenshot("page.png") # the visible viewport driver.find_element(By.CSS_SELECTOR, "div.quote").screenshot("quote.png") # one element ``` Both write PNG files. Save one automatically whenever a wait times out, and most "the selector stopped working" mysteries solve themselves: you'll see a cookie banner, a login wall or a challenge page. ## Step 8: Export to CSV and JSON ```python import csv import json quotes = scrape_js_pages(driver) with open("quotes.csv", "w", newline="", encoding="utf-8") as f: writer = csv.DictWriter(f, fieldnames=["text", "author", "tags"]) writer.writeheader() for row in quotes: writer.writerow({**row, "tags": ", ".join(row["tags"])}) with open("quotes.json", "w", encoding="utf-8") as f: json.dump(quotes, f, ensure_ascii=False, indent=2) ``` CSV has no lists, so the tags are joined into one cell; JSON keeps them as a list. The first CSV row came out as `“The world as we have created it…”,Albert Einstein,"change, deep-thoughts, thinking, world"`. ### Tip: parse with BeautifulSoup once the page is rendered On pages with hundreds of elements, every `find_element` call is a round trip to the browser. Let Selenium render the page, then hand the HTML to BeautifulSoup (`pip install beautifulsoup4`): ```python from bs4 import BeautifulSoup driver.get("https://quotes.toscrape.com/js/") WebDriverWait(driver, 10).until( EC.presence_of_element_located((By.CSS_SELECTOR, "div.quote")) ) soup = BeautifulSoup(driver.page_source, "html.parser") authors = [a.text for a in soup.select("div.quote small.author")] print(len(authors), authors[:3]) ``` It printed `10 ['Albert Einstein', 'J.K. Rowling', 'Albert Einstein']`. ## When to choose Playwright instead Selenium works, as every step above shows. For a new scraper, though, Playwright usually needs less code: it waits for elements on its own before every action, blocks or captures network requests with one `page.route()` call, and runs many isolated sessions in one browser. Stay with Selenium if your team writes Ruby, you need real Safari, or you already run Selenium Grid. Our [Playwright vs Selenium comparison](https://scrape.ly/blog/playwright-vs-selenium.md) runs the same scraper in both. One thing neither library changes: in our test, `navigator.webdriver` returned `true` in headless Selenium Chrome, as it does in headless Playwright. Sites with serious bot protection check signals like that, plus fingerprints and IP reputation. ## Selenium web scraping at scale with Scrape.ly A Selenium script on your laptop is one browser, one IP and one machine. Scaling it means many browsers, rotating proxies, retries, scheduling and storage. Scrape.ly gives you three ways to get there: - **Skip the browser with a ready Actor.** The [Scrape.ly Store](https://scrape.ly/store.md) has 568 ready-made scrapers for sites like Google Maps, Amazon and LinkedIn, priced per result on each Store page. Many don't need a browser at all: the [Google Maps Scraper](https://scrape.ly/store/cornerledger/google-maps-scraper.md) saved 10 places with 11 plain HTTP requests in our [Google Maps tutorial](https://scrape.ly/blog/how-to-scrape-google-maps.md). For any other public site, [Web Scraper](https://scrape.ly/store/querystone/web-scraper.md) runs your own page function on every page for $0.50 per 1,000 results. - **Host your Selenium code as an Actor.** Scrape.ly runs Python Actors on its own base images, and one family, `scrapely/actor-python-selenium`, comes with Selenium for Python 3.11 to 3.14. Start a project with `scrapely create my-actor --template crawlee-bs4-py`, base its Dockerfile on the Selenium image, put your scraper in the main function, and deploy with `scrapely push`. You get the platform's datasets, schedules, webhooks and API, plus a maximum charge per run. The [Python SDK deployment guide](https://scrape.ly/docs/sdk-python-deploy.md) covers the Dockerfile and project layout. - **Move hard sites to the Scrape.ly browser.** For sites that block automated Chrome, port the scraper to Playwright (the steps map one to one) and turn on the [Scrape.ly browser](https://scrape.ly/docs/browser.md) for your Actor. Every Playwright launch then connects to a hosted Chromium with anti-bot fingerprints, Cloudflare Turnstile solving and its own residential proxy, with no code changes. You pick the exit country, state or city and the operating system. It costs $0.45 per browser hour at list price, billed in 30-second blocks plus the browser's proxy traffic. ## What to watch out for - **Resource use.** Each Selenium session is a full Chrome. Ten parallel scrapers are ten browsers' worth of memory. Block images, reuse one driver for many pages, and always `quit()`. - **Flaky waits.** Most broken Selenium scrapers are timing bugs. Wait for a specific condition, never a fixed sleep, and screenshot on timeouts. - **Layout changes.** Selectors break when sites redesign. Validate every record and fail loudly on empty fields instead of saving blanks. - **Blocking and rate limits.** Pause between pages and keep volumes reasonable. For protected sites, use the Scrape.ly browser or a Store Actor that already handles proxies and retries. - **Personal data and terms.** Scrape public data only, read the site's terms, and treat names and profiles under privacy laws such as the GDPR. Scrape.ly Store Actors only collect publicly available data shown to any visitor. ## FAQ ### Is Selenium good for web scraping? Yes, for pages that need a real browser: JavaScript-rendered content, clicks, forms and infinite scroll. It's slower than plain HTTP requests, so use requests and BeautifulSoup when the data is already in the page source, and consider Playwright for new browser scrapers. ### How do I run Selenium headless in Python? Add `options.add_argument("--headless=new")` to `webdriver.ChromeOptions()` and pass the options to `webdriver.Chrome(options=options)`. Set a window size too, so the page renders its desktop layout. ### Do I still need to download chromedriver? No. Current Selenium 4 releases include Selenium Manager, which finds or downloads the matching driver automatically and caches it, and it can fetch Chrome for Testing if no Chrome is installed. ### How do I scroll a page with Selenium? Run `driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")`, then wait until new elements appear, and repeat until the element count stops growing. Check the Network tab first: infinite-scroll pages often load from a JSON API you can call directly. ### Is Selenium better than BeautifulSoup? They do different jobs. BeautifulSoup parses HTML you already have; Selenium drives a browser to produce it. They combine well: render with Selenium, then parse `driver.page_source` with BeautifulSoup. ### Can websites detect Selenium? Yes. Automated Chrome reports `navigator.webdriver` as `true`, which our headless test confirmed, and bot protection also checks fingerprints and IP reputation. A hosted browser with real fingerprints and residential IPs, such as the Scrape.ly browser, is the dependable way through. Scrape.ly Store Web Scraper Scrape any public site without running browsers yourself. Your page function, link following, proxies and retries included. $0.50 per 1,000 results. [Try it free](https://scrape.ly/store/querystone/web-scraper.md) [Browse the Store](https://scrape.ly/store.md) About the author [Scrape.ly Team](https://scrape.ly/blog/author/scrapely-team.md) The team behind Scrape.ly Scrape.ly is the web data platform from TeraShift GmbH in Switzerland, powered by Evomi: ready-made scrapers in the Scrape.ly Store, a cloud runtime, an API with JavaScript and Python SDKs, proxies, a hosted anti-bot browser and an MCP server for AI agents. - [Website](https://scrape.ly/index.md) Keep reading ## More in Guides - [![Pixel-art robot with an orange >\_ on its terminal head lifts a lit strip of text off the screen of an old grey terminal monitor, where one highlighted line is being read, and drops it into a crate](https://scrape.ly/media/screen-scraping.png?w=750)](https://scrape.ly/blog/screen-scraping.md) ### [What Is Screen Scraping (and How It Differs From Web Scraping)](https://scrape.ly/blog/screen-scraping.md) [Screen scraping reads data from what a program displays: terminal screens, desktop windows, rendered pages or images. See its three kinds, how RPA and banks use it, where regulators drew lines, and why web scraping from structured data beats reading pixels.](https://scrape.ly/blog/screen-scraping.md) - [![Pixel-art robot with an orange >\_ on its terminal head carries a row of data from a browser page on the left to a spreadsheet marked with the green Microsoft Excel logo on the right](https://scrape.ly/media/scrape-data-from-website-to-excel.png?w=750)](https://scrape.ly/blog/scrape-data-from-website-to-excel.md) ### [How to Scrape Data From a Website Into Excel (3 Ways)](https://scrape.ly/blog/scrape-data-from-website-to-excel.md) [Three ways to get website data into Excel: copy and paste, Excel's built-in From Web (Power Query), or a ready-made scraper that exports XLSX and refreshes straight into your workbook. Steps, limits and working Power Query code.](https://scrape.ly/blog/scrape-data-from-website-to-excel.md) - [![Pixel-art robot with an orange >\_ on its terminal head carries a row of data from a web page on the left to a spreadsheet marked with the green Google Sheets logo on the right](https://scrape.ly/media/scrape-website-to-google-sheets.png?w=750)](https://scrape.ly/blog/scrape-website-to-google-sheets.md) ### [IMPORTXML + Google Sheets: How to Scrape a Website Into a Spreadsheet (and Beyond)](https://scrape.ly/blog/scrape-website-to-google-sheets.md) [How to scrape a website into Google Sheets: IMPORTXML, IMPORTHTML, IMPORTDATA and IMPORTFEED with exact syntax and tested XPath on books.toscrape.com, Google's refresh rules, an Apps Script fetcher, and scheduled scrapes that write straight into your sheet.](https://scrape.ly/blog/scrape-website-to-google-sheets.md) ## The internet is full of data. Start owning yours. Scrape.ly gives you the infrastructure, the tooling, and the team to extract, process, and operationalize web data at any scale. Setup takes minutes. The advantage lasts forever. [Get Started Free](https://my.scrape.ly/register) [Chat with Support](mailto:support@scrape.ly?subject=Question%20about%20Scrape.ly\&body=Hi%2C%20I%20have%20a%20question%20about%20Scrape.ly.) No credit card. No setup calls.