`, the full title is in the `title` attribute of the link inside ``, and the price is in `
`:
```plaintext
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
public class FirstScraper {
public static void main(String[] args) throws Exception {
Document doc = Jsoup.connect("https://books.toscrape.com/")
.userAgent("books-tutorial/1.0 (you@example.com)")
.timeout(15_000)
.get();
for (Element card : doc.select("article.product_pod")) {
String title = card.selectFirst("h3 a").attr("title");
String price = card.selectFirst("p.price_color").text();
String url = card.selectFirst("h3 a").absUrl("href");
System.out.println(title + " | " + price + " | " + url);
}
System.out.println(doc.select("article.product_pod").size() + " books on this page");
}
}
```
The first lines of output:
```plaintext
A Light in the Attic £51.77 https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html
Tipping the Velvet £53.74 https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html
Soumission £50.10 https://books.toscrape.com/catalogue/soumission_998/index.html
...
20 books on this page
```
Three things Jsoup does for you here. `absUrl("href")` resolves the relative link against the page's URL, so you never glue URLs together by hand. The pound sign comes out right, because Jsoup detects the encoding from the page's meta tag even though this server sends no charset in its headers. And `select` throws nothing when a selector matches nothing: `selectFirst` returns `null`, which is your cue to validate records before you save them.
### Jsoup selectors worth knowing
Jsoup speaks CSS plus a few extras that save you from writing loops. On a book's detail page, `doc.selectFirst("th:containsOwn(UPC) + td").text()` finds the table cell next to the "UPC" header and returned `a897fe39b1053632`. `:has(...)` filters by descendants (`article.product_pod:has(p.star-rating.Five)`), `eachAttr("title")` collects one attribute from every match into a list, and `className()` returns the full class string, which matters on this site because the star rating lives in a class name (`star-rating Three`).
## Crawl every page: HttpClient, retries and politeness
`Jsoup.connect()` is fine for a few pages. For a real crawl, use the JDK's `HttpClient` for downloading and Jsoup only for parsing. You get one connection pool for the whole run, explicit timeouts, and full control over retries. This crawler follows the **next** link through the catalogue, retries 429 and 5xx responses with backoff, waits a second between requests and writes a CSV:
```plaintext
import java.io.IOException;
import java.io.PrintWriter;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Duration;
import java.util.ArrayList;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
public class BooksCrawler {
record Book(String title, double price, int rating, boolean inStock, String url) {}
static final Map RATINGS = Map.of("One", 1, "Two", 2, "Three", 3, "Four", 4, "Five", 5);
static final HttpClient CLIENT = HttpClient.newBuilder()
.connectTimeout(Duration.ofSeconds(10))
.followRedirects(HttpClient.Redirect.NORMAL)
.build();
static String fetch(String url) throws IOException, InterruptedException {
HttpRequest request = HttpRequest.newBuilder(URI.create(url))
.header("User-Agent", "books-tutorial/1.0 (you@example.com)")
.timeout(Duration.ofSeconds(15))
.GET()
.build();
for (int attempt = 1; ; attempt++) {
HttpResponse response =
CLIENT.send(request, HttpResponse.BodyHandlers.ofString(StandardCharsets.UTF_8));
int status = response.statusCode();
if (status == 200) return response.body();
boolean retryable = status == 429 status >= 500;
if (!retryable attempt == 3) throw new IOException("HTTP " + status + " for " + url);
Thread.sleep(1000L << attempt); // back off: 2 s, then 4 s
}
}
static List parse(Document doc) {
List books = new ArrayList<>();
for (Element card : doc.select("article.product_pod")) {
Element link = card.selectFirst("h3 a");
books.add(new Book(
link.attr("title"),
Double.parseDouble(card.selectFirst("p.price_color").text().replace("£", "")),
RATINGS.get(card.selectFirst("p.star-rating").classNames().stream()
.filter(RATINGS::containsKey).findFirst().orElse("One")),
card.selectFirst("p.availability").text().contains("In stock"),
link.absUrl("href")));
}
return books;
}
public static void main(String[] args) throws Exception {
int maxPages = args.length > 0 ? Integer.parseInt(args[0]) : 50;
String url = "https://books.toscrape.com/catalogue/page-1.html";
List all = new ArrayList<>();
for (int page = 1; page <= maxPages && url != null; page++) {
Document doc = Jsoup.parse(fetch(url), url); // base URI makes absUrl() work
all.addAll(parse(doc));
Element next = doc.selectFirst("li.next a");
url = next == null ? null : next.absUrl("href");
Thread.sleep(1000); // one request per second
}
try (PrintWriter out = new PrintWriter(Files.newBufferedWriter(Path.of("books.csv")))) {
out.println("title,price,rating,in_stock,url");
for (Book b : all) {
out.printf(Locale.ROOT, "\"%s\",%.2f,%d,%b,%s%n",
b.title().replace("\"", "\"\""), b.price(), b.rating(), b.inStock(), b.url());
}
}
System.out.println(all.size() + " books saved to books.csv");
System.out.println(all.get(0));
}
}
```
Run without arguments, it fetched all 50 pages and printed:
```plaintext
1000 books saved to books.csv
Book[title=A Light in the Attic, price=51.77, rating=3, inStock=true, url=https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html]
```
The whole run took 107 seconds on our connection, 50 of them the deliberate one-second pauses. Pass a number (`java ... BooksCrawler 5`) to stop early while you test. What each piece is for:
- **`Jsoup.parse(html, url)`** parses a string you downloaded yourself. The second argument is the base URI, without which `absUrl()` returns an empty string.
- **The retry loop** repeats only what is worth repeating: 429 (slow down) and 5xx (server trouble). A 404 fails at once. The wait doubles each time, so you back off instead of hammering a struggling server.
- **`BodyHandlers.ofString(UTF_8)`** decodes the body explicitly. `HttpClient` falls back to UTF-8 when no charset is sent, but being explicit documents the assumption.
- **`Locale.ROOT`** **in **__`printf`__ **.** Our first run wrote `51,77` instead of `51.77` into the CSV, because `%.2f` follows the machine's locale and the test Mac was set to German. Any number you format for a file needs a fixed locale.
- **`Thread.sleep(1000)`** keeps you at about one request per second. On a sandbox it's courtesy; on a real site it's what keeps your IP off a block list.
## Faster: fetch pages in parallel with virtual threads
When you know the URLs up front, fetch several at once. Virtual threads (Java 21+) make each request cheap to run in its own thread, and a `Semaphore` caps how many hit the server at the same time:
```plaintext
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.time.Duration;
import java.util.ArrayList;
import java.util.List;
import java.util.Locale;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.Semaphore;
import org.jsoup.Jsoup;
public class ParallelPages {
static final HttpClient CLIENT = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(10)).build();
static final Semaphore LIMIT = new Semaphore(4); // at most 4 requests in flight
static List titles(int page) throws Exception {
String url = "https://books.toscrape.com/catalogue/page-" + page + ".html";
HttpRequest request = HttpRequest.newBuilder(URI.create(url)).timeout(Duration.ofSeconds(15)).build();
LIMIT.acquire();
try {
String html = CLIENT.send(request, HttpResponse.BodyHandlers.ofString(StandardCharsets.UTF_8)).body();
return Jsoup.parse(html, url).select("article.product_pod h3 a").eachAttr("title");
} finally {
LIMIT.release();
}
}
public static void main(String[] args) throws Exception {
long start = System.nanoTime();
List all = new ArrayList<>();
try (ExecutorService pool = Executors.newVirtualThreadPerTaskExecutor()) {
List>> pages = new ArrayList<>();
for (int page = 1; page <= 10; page++) {
int p = page;
pages.add(pool.submit(() -> titles(p)));
}
for (Future> f : pages) all.addAll(f.get());
}
System.out.printf(Locale.ROOT, "%d titles in %.1f s, first: %s%n",
all.size(), (System.nanoTime() start) 1e9, all.get(0));
}
}
```
It printed `200 titles in 1.8 s, first: A Light in the Attic`. Keep the semaphore small on real sites: four parallel requests to one host is already more than many small sites expect.
## Scrape JavaScript pages: Playwright and Selenium for Java
Open [quotes.toscrape.com/js](https://quotes.toscrape.com/js/) and view the source: the quotes aren't in it, a script writes them into the page. Jsoup only ever sees that empty source, so you need a real browser.
Before you reach for one, check the browser's **Network** tab for a JSON endpoint. Many JavaScript pages load their data from an API you can call with `HttpClient` directly; the infinite-scroll version of the quotes site, for example, reads `https://quotes.toscrape.com/api/quotes?page=1`.
### Playwright for Java
Add `com.microsoft.playwright:playwright:1.64.0` to your build. On first run, Playwright downloads the browsers it needs:
```plaintext
import com.microsoft.playwright.Browser;
import com.microsoft.playwright.BrowserType;
import com.microsoft.playwright.Locator;
import com.microsoft.playwright.Page;
import com.microsoft.playwright.Playwright;
public class QuotesPlaywright {
public static void main(String[] args) {
try (Playwright playwright = Playwright.create()) {
Browser browser = playwright.chromium().launch(new BrowserType.LaunchOptions().setHeadless(true));
Page page = browser.newPage();
page.navigate("https://quotes.toscrape.com/js/");
page.waitForSelector("div.quote"); // rendered by JavaScript
for (Locator quote : page.locator("div.quote").all()) {
String text = quote.locator("span.text").innerText();
String author = quote.locator("small.author").innerText();
System.out.println(author + ": " + text);
}
System.out.println(page.locator("div.quote").count() + " quotes");
}
}
}
```
It printed all 10 quotes on the page, starting with Albert Einstein, and `10 quotes`. `waitForSelector` is the important line: it waits until the script has rendered the element, instead of guessing with a fixed sleep.
### Selenium
With `org.seleniumhq.selenium:selenium-java:4.51.0`, Selenium Manager finds or downloads a matching ChromeDriver for you, so there's no driver to install by hand:
```plaintext
import java.time.Duration;
import java.util.List;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;
public class QuotesSelenium {
public static void main(String[] args) {
ChromeOptions options = new ChromeOptions().addArguments("--headless=new");
WebDriver driver = new ChromeDriver(options);
try {
driver.get("https://quotes.toscrape.com/js/");
new WebDriverWait(driver, Duration.ofSeconds(10))
.until(ExpectedConditions.presenceOfElementLocated(By.cssSelector("div.quote")));
List quotes = driver.findElements(By.cssSelector("div.quote"));
for (WebElement quote : quotes) {
System.out.println(quote.findElement(By.cssSelector("small.author")).getText()
+ ": " + quote.findElement(By.cssSelector("span.text")).getText());
}
System.out.println(quotes.size() + " quotes");
} finally {
driver.quit();
}
}
}
```
Same 10 quotes. If you're choosing between the two, our [Playwright vs Selenium comparison](https://scrape.ly/blog/playwright-vs-selenium.md) runs the same scraper in both; the short version is that Playwright's auto-waiting and browser management make scraping code shorter. And if you want to know what "headless" actually changes, read [what a headless browser is](https://scrape.ly/blog/what-is-a-headless-browser.md).
## Run Java web scraping jobs in the cloud with Scrape.ly
The code above runs on your machine. Running it every day, from IPs that don't get blocked, with storage, retries and an alert when a redesign breaks a selector, is a different job. [Scrape.ly](https://scrape.ly/store.md) runs that layer for you, and your Java code calls it over a plain REST API with `java.net.http`. No SDK needed.
For big platforms, the Store has 570+ ready-made scrapers (Actors), maintained by their publishers. For any other public site, the [Web Scraper](https://scrape.ly/store/querystone/web-scraper.md) Actor fetches pages over HTTP through Scrape.ly's residential proxy and runs a small JavaScript page function you write. This Java program runs it on the Travel category of books.toscrape.com and gets the rows back in the same request:
```plaintext
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;
public class RunActor {
public static void main(String[] args) throws Exception {
String token = System.getenv("SCRAPELY_TOKEN");
String input = """
{
"startUrls": [{"url": "https://books.toscrape.com/catalogue/category/books/travel_2/index.html"}],
"pageFunction": "async function pageFunction(context) { const $ = context.jQuery; return $('article.product_pod').map((i, el) => ({ title: $(el).find('h3 a').attr('title'), price: $(el).find('p.price_color').text(), inStock: $(el).find('p.availability').text().includes('In stock') })).get(); }",
"proxyConfiguration": {"useScrapelyProxy": true}
}
""";
HttpRequest request = HttpRequest.newBuilder(URI.create(
"https://api.scrape.ly/v2/actors/querystone~web-scraper/run-sync-get-dataset-items"))
.header("Authorization", "Bearer " + token)
.header("Content-Type", "application/json")
.timeout(Duration.ofSeconds(60))
.POST(HttpRequest.BodyPublishers.ofString(input))
.build();
HttpResponse response = HttpClient.newHttpClient()
.send(request, HttpResponse.BodyHandlers.ofString());
System.out.println(response.statusCode());
System.out.println(response.body()); // a JSON array, one object per book
}
}
```
Create a token under **Settings → API & Integrations** and export it as `SCRAPELY_TOKEN`. The page function returns an array, and each element becomes one row. Parse the JSON with Jackson or Gson, or add `?format=csv` to the URL to get CSV instead.
### What the run returned
We ran this exact input on 26 October 2026. The run took 7.6 seconds, fetched 1 page and saved 11 books. The first five rows:
| title | price | inStock |
| ------------------------------------------------------------------- | ------ | ------- |
| It's Only the Himalayas | £45.17 | true |
| Full Moon over Noah’s Ark: An Odyssey to Mount Ararat and Beyond | £49.43 | true |
| See America: A Celebration of Our National Parks & Treasured Sites | £48.87 | true |
| Vagabonding: An Uncommon Guide to the Art of Long-Term World Travel | £36.94 | true |
| Under the Tuscan Sun | £37.33 | true |
From the run log:
```plaintext
20261026T14:33:56.579Z [Scrapely] Container started (setup 0.9 s)
20261026T14:33:58.377Z [scrapely.proxy_configuration] INFO Platform proxy access check succeeded.
20261026T14:34:02.922Z [scrapely.actor] INFO Done: 1 pages fetched, 11 records saved in 5 s
20261026T14:34:02.922Z [scrapely.actor] INFO [Status message]: Saved 11 results from 1 page.
```
Web Scraper costs $0.000715 per result, with platform usage (compute, proxy) included. So this run was 11 × $0.000715 = $0.0079 at list price (we were charged a little less, which reflects a plan discount). The free plan's $5 of monthly credit covers about 7,000 such rows.
For jobs that take longer than about 30 seconds, don't hold a request open. Start the run with `POST https://api.scrape.ly/v2/actors/querystone~web-scraper/runs`, read the run ID from `data.id`, poll `https://api.scrape.ly/v2/actor-runs/RUN_ID?waitForFinish=60` until the status is `SUCCEEDED`, then fetch `https://api.scrape.ly/v2/actor-runs/RUN_ID/dataset/items`. Three plain HTTP calls, the same `HttpClient` code.
Why route a Java scraper through Scrape.ly instead of running it yourself:
- **No blocking work on your side.** Requests go through a residential proxy, and the [Scrape.ly browser](https://scrape.ly/docs/browser.md) handles anti-bot fingerprints and Cloudflare Turnstile when a site needs a real browser.
- **Maintained scrapers for big sites.** Amazon, Google Maps, LinkedIn and hundreds more have their own Actors, fixed by their publishers when the site changes. Browse the [Store](https://scrape.ly/store.md) before you write a parser.
- **Schedules and deliveries without code.** Run the same input daily and send the rows to Google Sheets, Slack, email or a webhook; see [how to schedule and automate web scraping](https://scrape.ly/blog/schedule-web-scraping.md).
- **Costs you can cap.** Every run takes a maximum charge and stops when it reaches it.
If you also write Python or Node.js, the same Actors are a client call away: see [web scraping with Python](https://scrape.ly/blog/web-scraping-python.md) and [Node.js web scraping](https://scrape.ly/blog/node-js-web-scraping.md).
Scrape.ly Store Web Scraper Run your own page function on any public site and call it from Java over plain HTTP. Residential proxy included. $0.71 per 1,000 results.
[Try it free](https://scrape.ly/store/querystone/web-scraper.md) [Browse the Store](https://scrape.ly/store.md)
## What to watch out for
- **Blocking.** Fast requests from one IP get throttled or blocked. Keep delays, back off on 429, and move to rotating residential proxies when a site insists; our guide to [rotating proxies for web scraping](https://scrape.ly/blog/rotating-proxies-for-web-scraping.md) explains the options. On Scrape.ly the proxy is one input field.
- **Selectors break.** Sites redesign. Check that every record has a title and a parsable price, and fail loudly instead of writing blank rows.
- **Locale and encoding bugs.** Format numbers with `Locale.ROOT`, and decode bodies with the right charset. Both bugs produce files that look fine until someone imports them.
- **Personal data and terms.** Scrape public pages, read the site's terms and `robots.txt`, and don't collect personal data you don't need; privacy laws such as the GDPR apply to it. Scrape.ly Store Actors only collect publicly available data shown to any visitor.
## FAQ
### Is Java good for web scraping?
Yes. Jsoup is one of the most pleasant HTML parsers in any language, the JDK ships a modern HTTP client, and virtual threads make concurrent fetching cheap. Java fits especially well when the scraper feeds an existing Java or Spring backend. Python has more scraping tutorials, but nothing a scraper needs is missing in Java.
### What is the best Java library for web scraping?
Jsoup for static HTML: it fetches, parses and queries with CSS selectors. Add `java.net.http.HttpClient` for crawls that need retries and connection reuse, and Playwright for Java when pages render their content with JavaScript.
### How do I scrape a JavaScript-rendered page in Java?
First look for a JSON API in the browser's Network tab and call it with `HttpClient`. If there isn't one, load the page in Playwright or Selenium, wait for the element you need (`waitForSelector` or `WebDriverWait`), then read it.
### Java vs Python for web scraping: which is better?
Python has the bigger scraping ecosystem (BeautifulSoup, Scrapy) and shorter scripts. Java gives you static types, strong concurrency and an easy fit into JVM services. Choose the language your team already ships; for the heavy parts, such as proxies, browsers and scheduling, a platform like Scrape.ly works the same from both.
### Is web scraping with Java legal?
The language doesn't matter; what you scrape and how does. Collecting publicly available data is generally lawful in many places, but respect site terms, avoid overloading servers and handle personal data under privacy law. This isn't legal advice; our guide on [whether web scraping is legal](https://scrape.ly/blog/is-web-scraping-legal.md) covers the details.
About the author
[Scrape.ly Team](https://scrape.ly/blog/author/scrapely-team.md) The team behind Scrape.ly
Scrape.ly is the web data platform from TeraShift GmbH in Switzerland, powered by Evomi: ready-made scrapers in the Scrape.ly Store, a cloud runtime, an API with JavaScript and Python SDKs, proxies, a hosted anti-bot browser and an MCP server for AI agents.
- [Website](https://scrape.ly/index.md)
Keep reading
## More in Guides
- [](https://scrape.ly/blog/how-to-build-a-web-scraper.md)
### [How to Build a Web Scraper: 8 Steps From First Request to the Cloud](https://scrape.ly/blog/how-to-build-a-web-scraper.md)
[How to build a web scraper that keeps working: define the data, inspect the site, fetch politely, parse and validate, follow pagination, store in SQLite with price history, schedule it and scale it. Tested Python code, plus when a ready-made cloud scraper beats building your own.](https://scrape.ly/blog/how-to-build-a-web-scraper.md)
- [](https://scrape.ly/blog/what-is-a-headless-browser.md)
### [What Is a Headless Browser? Uses, Tools and Detection](https://scrape.ly/blog/what-is-a-headless-browser.md)
[A headless browser is a real browser without a visible window, driven from code. Headless vs headed, how Chrome's headless mode works today, tested Playwright, Puppeteer and Selenium examples, how sites detect headless browsers, and when you need one.](https://scrape.ly/blog/what-is-a-headless-browser.md)
- [](https://scrape.ly/blog/schedule-web-scraping.md)
### [How to Automate Web Scraping: Schedules, Cron, Filters and Deliveries](https://scrape.ly/blog/schedule-web-scraping.md)
[How to automate web scraping end to end: run scrapers on a schedule with cron expressions, keep only new or changed items, add an AI summary, and deliver results to Google Sheets, Slack, email or a webhook. Compared with cron and GitHub Actions, with real run costs.](https://scrape.ly/blog/schedule-web-scraping.md)
## The internet is full of data. Start owning yours.
Scrape.ly gives you the infrastructure, the tooling, and the team to extract, process, and operationalize web data at any scale. Setup takes minutes.
[Get Started Free](https://my.scrape.ly/register) [Chat with Support](mailto:hello@scrape.ly?subject=Question%20about%20Scrape.ly\&body=Hi%2C%20I%20have%20a%20question%20about%20Scrape.ly.)
No credit card. No setup calls.