Java web scraping comes down to two jobs: download a page, then pull structured data out of its HTML. For most sites, Jsoup does both in a few lines, Java's built-in HttpClient gives you control over headers, timeouts and retries, and Playwright or Selenium take over when the data only appears after JavaScript runs.
This guide builds a scraper step by step, from a 15-line Jsoup script to a crawler that walks all 50 pages of a catalogue and writes a CSV. Then it covers concurrency with virtual threads, headless browsers, and how to call ready-made cloud scrapers from Java when you'd rather not run the infrastructure yourself. Every snippet was run on 26 October 2026 with Eclipse Temurin JDK 25.0.5 against books.toscrape.com and quotes.toscrape.com, two sandbox sites built for practice.
Java web scraping libraries compared
| Library | What it does | Use it when | Version tested |
|---|---|---|---|
| Jsoup | Fetches pages and parses HTML with CSS selectors | The data is in the page source | 1.23.2 |
java.net.http.HttpClient | HTTP client in the JDK (Java 11+) | You want retries, custom headers and HTTP/2 without a dependency | JDK 25.0.5 |
| Playwright for Java | Drives Chromium, Firefox or WebKit | The page renders its data with JavaScript | 1.64.0 |
| Selenium | Drives a real browser over WebDriver | You already use Selenium or Selenium Grid | 4.51.0 |
Start with Jsoup. Pair it with HttpClient once you crawl more than a handful of pages, and add a browser only when a site forces you to.
Set up a Java scraping project
You need a JDK 17 or newer for the code below (it uses records and text blocks; the virtual-thread example needs 21). With Maven, add Jsoup to pom.xml:
<dependency>
<groupId>org.jsoup</groupId>
<artifactId>jsoup</artifactId>
<version>1.23.2</version>
</dependency>With Gradle it's implementation("org.jsoup:jsoup:1.23.2"). For a quick experiment you don't need a build tool at all: download the jar from Maven Central and run a single source file directly, which is how we ran the first two examples:
curl -LO https://repo1.maven.org/maven2/org/jsoup/jsoup/1.23.2/jsoup-1.23.2.jar
java -cp jsoup-1.23.2.jar FirstScraper.javaYour first scraper with Jsoup
Open books.toscrape.com, right-click a book and choose Inspect. Each book sits in an <article class="product_pod">, the full title is in the title attribute of the link inside <h3>, and the price is in <p class="price_color">:
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
public class FirstScraper {
public static void main(String[] args) throws Exception {
Document doc = Jsoup.connect("https://books.toscrape.com/")
.userAgent("books-tutorial/1.0 (you@example.com)")
.timeout(15_000)
.get();
for (Element card : doc.select("article.product_pod")) {
String title = card.selectFirst("h3 a").attr("title");
String price = card.selectFirst("p.price_color").text();
String url = card.selectFirst("h3 a").absUrl("href");
System.out.println(title + " | " + price + " | " + url);
}
System.out.println(doc.select("article.product_pod").size() + " books on this page");
}
}The first lines of output:
A Light in the Attic | £51.77 | https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html
Tipping the Velvet | £53.74 | https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html
Soumission | £50.10 | https://books.toscrape.com/catalogue/soumission_998/index.html
...
20 books on this pageThree things Jsoup does for you here. absUrl("href") resolves the relative link against the page's URL, so you never glue URLs together by hand. The pound sign comes out right, because Jsoup detects the encoding from the page's meta tag even though this server sends no charset in its headers. And select throws nothing when a selector matches nothing: selectFirst returns null, which is your cue to validate records before you save them.
Jsoup selectors worth knowing
Jsoup speaks CSS plus a few extras that save you from writing loops. On a book's detail page, doc.selectFirst("th:containsOwn(UPC) + td").text() finds the table cell next to the "UPC" header and returned a897fe39b1053632. :has(...) filters by descendants (article.product_pod:has(p.star-rating.Five)), eachAttr("title") collects one attribute from every match into a list, and className() returns the full class string, which matters on this site because the star rating lives in a class name (star-rating Three).
Crawl every page: HttpClient, retries and politeness
Jsoup.connect() is fine for a few pages. For a real crawl, use the JDK's HttpClient for downloading and Jsoup only for parsing. You get one connection pool for the whole run, explicit timeouts, and full control over retries. This crawler follows the next link through the catalogue, retries 429 and 5xx responses with backoff, waits a second between requests and writes a CSV:
import java.io.IOException;
import java.io.PrintWriter;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Duration;
import java.util.ArrayList;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
public class BooksCrawler {
record Book(String title, double price, int rating, boolean inStock, String url) {}
static final Map<String, Integer> RATINGS = Map.of("One", 1, "Two", 2, "Three", 3, "Four", 4, "Five", 5);
static final HttpClient CLIENT = HttpClient.newBuilder()
.connectTimeout(Duration.ofSeconds(10))
.followRedirects(HttpClient.Redirect.NORMAL)
.build();
static String fetch(String url) throws IOException, InterruptedException {
HttpRequest request = HttpRequest.newBuilder(URI.create(url))
.header("User-Agent", "books-tutorial/1.0 (you@example.com)")
.timeout(Duration.ofSeconds(15))
.GET()
.build();
for (int attempt = 1; ; attempt++) {
HttpResponse<String> response =
CLIENT.send(request, HttpResponse.BodyHandlers.ofString(StandardCharsets.UTF_8));
int status = response.statusCode();
if (status == 200) return response.body();
boolean retryable = status == 429 || status >= 500;
if (!retryable || attempt == 3) throw new IOException("HTTP " + status + " for " + url);
Thread.sleep(1000L << attempt); // back off: 2 s, then 4 s
}
}
static List<Book> parse(Document doc) {
List<Book> books = new ArrayList<>();
for (Element card : doc.select("article.product_pod")) {
Element link = card.selectFirst("h3 a");
books.add(new Book(
link.attr("title"),
Double.parseDouble(card.selectFirst("p.price_color").text().replace("£", "")),
RATINGS.get(card.selectFirst("p.star-rating").classNames().stream()
.filter(RATINGS::containsKey).findFirst().orElse("One")),
card.selectFirst("p.availability").text().contains("In stock"),
link.absUrl("href")));
}
return books;
}
public static void main(String[] args) throws Exception {
int maxPages = args.length > 0 ? Integer.parseInt(args[0]) : 50;
String url = "https://books.toscrape.com/catalogue/page-1.html";
List<Book> all = new ArrayList<>();
for (int page = 1; page <= maxPages && url != null; page++) {
Document doc = Jsoup.parse(fetch(url), url); // base URI makes absUrl() work
all.addAll(parse(doc));
Element next = doc.selectFirst("li.next a");
url = next == null ? null : next.absUrl("href");
Thread.sleep(1000); // one request per second
}
try (PrintWriter out = new PrintWriter(Files.newBufferedWriter(Path.of("books.csv")))) {
out.println("title,price,rating,in_stock,url");
for (Book b : all) {
out.printf(Locale.ROOT, "\"%s\",%.2f,%d,%b,%s%n",
b.title().replace("\"", "\"\""), b.price(), b.rating(), b.inStock(), b.url());
}
}
System.out.println(all.size() + " books saved to books.csv");
System.out.println(all.get(0));
}
}Run without arguments, it fetched all 50 pages and printed:
1000 books saved to books.csv
Book[title=A Light in the Attic, price=51.77, rating=3, inStock=true, url=https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html]The whole run took 107 seconds on our connection, 50 of them the deliberate one-second pauses. Pass a number (java ... BooksCrawler 5) to stop early while you test. What each piece is for:
Jsoup.parse(html, url)parses a string you downloaded yourself. The second argument is the base URI, without whichabsUrl()returns an empty string.- The retry loop repeats only what is worth repeating: 429 (slow down) and 5xx (server trouble). A 404 fails at once. The wait doubles each time, so you back off instead of hammering a struggling server.
BodyHandlers.ofString(UTF_8)decodes the body explicitly.HttpClientfalls back to UTF-8 when no charset is sent, but being explicit documents the assumption.Locale.ROOTinprintf. Our first run wrote51,77instead of51.77into the CSV, because%.2ffollows the machine's locale and the test Mac was set to German. Any number you format for a file needs a fixed locale.Thread.sleep(1000)keeps you at about one request per second. On a sandbox it's courtesy; on a real site it's what keeps your IP off a block list.
Faster: fetch pages in parallel with virtual threads
When you know the URLs up front, fetch several at once. Virtual threads (Java 21+) make each request cheap to run in its own thread, and a Semaphore caps how many hit the server at the same time:
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.time.Duration;
import java.util.ArrayList;
import java.util.List;
import java.util.Locale;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.Semaphore;
import org.jsoup.Jsoup;
public class ParallelPages {
static final HttpClient CLIENT = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(10)).build();
static final Semaphore LIMIT = new Semaphore(4); // at most 4 requests in flight
static List<String> titles(int page) throws Exception {
String url = "https://books.toscrape.com/catalogue/page-" + page + ".html";
HttpRequest request = HttpRequest.newBuilder(URI.create(url)).timeout(Duration.ofSeconds(15)).build();
LIMIT.acquire();
try {
String html = CLIENT.send(request, HttpResponse.BodyHandlers.ofString(StandardCharsets.UTF_8)).body();
return Jsoup.parse(html, url).select("article.product_pod h3 a").eachAttr("title");
} finally {
LIMIT.release();
}
}
public static void main(String[] args) throws Exception {
long start = System.nanoTime();
List<String> all = new ArrayList<>();
try (ExecutorService pool = Executors.newVirtualThreadPerTaskExecutor()) {
List<Future<List<String>>> pages = new ArrayList<>();
for (int page = 1; page <= 10; page++) {
int p = page;
pages.add(pool.submit(() -> titles(p)));
}
for (Future<List<String>> f : pages) all.addAll(f.get());
}
System.out.printf(Locale.ROOT, "%d titles in %.1f s, first: %s%n",
all.size(), (System.nanoTime() - start) / 1e9, all.get(0));
}
}It printed 200 titles in 1.8 s, first: A Light in the Attic. Keep the semaphore small on real sites: four parallel requests to one host is already more than many small sites expect.
Scrape JavaScript pages: Playwright and Selenium for Java
Open quotes.toscrape.com/js and view the source: the quotes aren't in it, a script writes them into the page. Jsoup only ever sees that empty source, so you need a real browser.
Before you reach for one, check the browser's Network tab for a JSON endpoint. Many JavaScript pages load their data from an API you can call with HttpClient directly; the infinite-scroll version of the quotes site, for example, reads https://quotes.toscrape.com/api/quotes?page=1.
Playwright for Java
Add com.microsoft.playwright:playwright:1.64.0 to your build. On first run, Playwright downloads the browsers it needs:
import com.microsoft.playwright.Browser;
import com.microsoft.playwright.BrowserType;
import com.microsoft.playwright.Locator;
import com.microsoft.playwright.Page;
import com.microsoft.playwright.Playwright;
public class QuotesPlaywright {
public static void main(String[] args) {
try (Playwright playwright = Playwright.create()) {
Browser browser = playwright.chromium().launch(new BrowserType.LaunchOptions().setHeadless(true));
Page page = browser.newPage();
page.navigate("https://quotes.toscrape.com/js/");
page.waitForSelector("div.quote"); // rendered by JavaScript
for (Locator quote : page.locator("div.quote").all()) {
String text = quote.locator("span.text").innerText();
String author = quote.locator("small.author").innerText();
System.out.println(author + ": " + text);
}
System.out.println(page.locator("div.quote").count() + " quotes");
}
}
}It printed all 10 quotes on the page, starting with Albert Einstein, and 10 quotes. waitForSelector is the important line: it waits until the script has rendered the element, instead of guessing with a fixed sleep.
Selenium
With org.seleniumhq.selenium:selenium-java:4.51.0, Selenium Manager finds or downloads a matching ChromeDriver for you, so there's no driver to install by hand:
import java.time.Duration;
import java.util.List;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;
public class QuotesSelenium {
public static void main(String[] args) {
ChromeOptions options = new ChromeOptions().addArguments("--headless=new");
WebDriver driver = new ChromeDriver(options);
try {
driver.get("https://quotes.toscrape.com/js/");
new WebDriverWait(driver, Duration.ofSeconds(10))
.until(ExpectedConditions.presenceOfElementLocated(By.cssSelector("div.quote")));
List<WebElement> quotes = driver.findElements(By.cssSelector("div.quote"));
for (WebElement quote : quotes) {
System.out.println(quote.findElement(By.cssSelector("small.author")).getText()
+ ": " + quote.findElement(By.cssSelector("span.text")).getText());
}
System.out.println(quotes.size() + " quotes");
} finally {
driver.quit();
}
}
}Same 10 quotes. If you're choosing between the two, our Playwright vs Selenium comparison runs the same scraper in both; the short version is that Playwright's auto-waiting and browser management make scraping code shorter. And if you want to know what "headless" actually changes, read what a headless browser is.
Run Java web scraping jobs in the cloud with Scrape.ly
The code above runs on your machine. Running it every day, from IPs that don't get blocked, with storage, retries and an alert when a redesign breaks a selector, is a different job. Scrape.ly runs that layer for you, and your Java code calls it over a plain REST API with java.net.http. No SDK needed.
For big platforms, the Store has 570+ ready-made scrapers (Actors), maintained by their publishers. For any other public site, the Web Scraper Actor fetches pages over HTTP through Scrape.ly's residential proxy and runs a small JavaScript page function you write. This Java program runs it on the Travel category of books.toscrape.com and gets the rows back in the same request:
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;
public class RunActor {
public static void main(String[] args) throws Exception {
String token = System.getenv("SCRAPELY_TOKEN");
String input = """
{
"startUrls": [{"url": "https://books.toscrape.com/catalogue/category/books/travel_2/index.html"}],
"pageFunction": "async function pageFunction(context) { const $ = context.jQuery; return $('article.product_pod').map((i, el) => ({ title: $(el).find('h3 a').attr('title'), price: $(el).find('p.price_color').text(), inStock: $(el).find('p.availability').text().includes('In stock') })).get(); }",
"proxyConfiguration": {"useScrapelyProxy": true}
}
""";
HttpRequest request = HttpRequest.newBuilder(URI.create(
"https://api.scrape.ly/v2/actors/querystone~web-scraper/run-sync-get-dataset-items"))
.header("Authorization", "Bearer " + token)
.header("Content-Type", "application/json")
.timeout(Duration.ofSeconds(60))
.POST(HttpRequest.BodyPublishers.ofString(input))
.build();
HttpResponse<String> response = HttpClient.newHttpClient()
.send(request, HttpResponse.BodyHandlers.ofString());
System.out.println(response.statusCode());
System.out.println(response.body()); // a JSON array, one object per book
}
}Create a token under Settings → API & Integrations and export it as SCRAPELY_TOKEN. The page function returns an array, and each element becomes one row. Parse the JSON with Jackson or Gson, or add ?format=csv to the URL to get CSV instead.
What the run returned
We ran this exact input on 26 October 2026. The run took 7.6 seconds, fetched 1 page and saved 11 books. The first five rows:
| title | price | inStock |
|---|---|---|
| It's Only the Himalayas | £45.17 | true |
| Full Moon over Noah’s Ark: An Odyssey to Mount Ararat and Beyond | £49.43 | true |
| See America: A Celebration of Our National Parks & Treasured Sites | £48.87 | true |
| Vagabonding: An Uncommon Guide to the Art of Long-Term World Travel | £36.94 | true |
| Under the Tuscan Sun | £37.33 | true |
From the run log:
2026-10-26T14:33:56.579Z [Scrapely] Container started (setup 0.9 s)
2026-10-26T14:33:58.377Z [scrapely.proxy_configuration] INFO Platform proxy access check succeeded.
2026-10-26T14:34:02.922Z [scrapely.actor] INFO Done: 1 pages fetched, 11 records saved in 5 s
2026-10-26T14:34:02.922Z [scrapely.actor] INFO [Status message]: Saved 11 results from 1 page.Web Scraper costs $0.000715 per result, with platform usage (compute, proxy) included. So this run was 11 × $0.000715 = $0.0079 at list price (we were charged a little less, which reflects a plan discount). The free plan's $5 of monthly credit covers about 7,000 such rows.
For jobs that take longer than about 30 seconds, don't hold a request open. Start the run with POST https://api.scrape.ly/v2/actors/querystone~web-scraper/runs, read the run ID from data.id, poll https://api.scrape.ly/v2/actor-runs/RUN_ID?waitForFinish=60 until the status is SUCCEEDED, then fetch https://api.scrape.ly/v2/actor-runs/RUN_ID/dataset/items. Three plain HTTP calls, the same HttpClient code.
Why route a Java scraper through Scrape.ly instead of running it yourself:
- No blocking work on your side. Requests go through a residential proxy, and the Scrape.ly browser handles anti-bot fingerprints and Cloudflare Turnstile when a site needs a real browser.
- Maintained scrapers for big sites. Amazon, Google Maps, LinkedIn and hundreds more have their own Actors, fixed by their publishers when the site changes. Browse the Store before you write a parser.
- Schedules and deliveries without code. Run the same input daily and send the rows to Google Sheets, Slack, email or a webhook; see how to schedule and automate web scraping.
- Costs you can cap. Every run takes a maximum charge and stops when it reaches it.
If you also write Python or Node.js, the same Actors are a client call away: see web scraping with Python and Node.js web scraping.
What to watch out for
- Blocking. Fast requests from one IP get throttled or blocked. Keep delays, back off on 429, and move to rotating residential proxies when a site insists; our guide to rotating proxies for web scraping explains the options. On Scrape.ly the proxy is one input field.
- Selectors break. Sites redesign. Check that every record has a title and a parsable price, and fail loudly instead of writing blank rows.
- Locale and encoding bugs. Format numbers with
Locale.ROOT, and decode bodies with the right charset. Both bugs produce files that look fine until someone imports them. - Personal data and terms. Scrape public pages, read the site's terms and
robots.txt, and don't collect personal data you don't need; privacy laws such as the GDPR apply to it. Scrape.ly Store Actors only collect publicly available data shown to any visitor.
FAQ
Is Java good for web scraping?
Yes. Jsoup is one of the most pleasant HTML parsers in any language, the JDK ships a modern HTTP client, and virtual threads make concurrent fetching cheap. Java fits especially well when the scraper feeds an existing Java or Spring backend. Python has more scraping tutorials, but nothing a scraper needs is missing in Java.
What is the best Java library for web scraping?
Jsoup for static HTML: it fetches, parses and queries with CSS selectors. Add java.net.http.HttpClient for crawls that need retries and connection reuse, and Playwright for Java when pages render their content with JavaScript.
How do I scrape a JavaScript-rendered page in Java?
First look for a JSON API in the browser's Network tab and call it with HttpClient. If there isn't one, load the page in Playwright or Selenium, wait for the element you need (waitForSelector or WebDriverWait), then read it.
Java vs Python for web scraping: which is better?
Python has the bigger scraping ecosystem (BeautifulSoup, Scrapy) and shorter scripts. Java gives you static types, strong concurrency and an easy fit into JVM services. Choose the language your team already ships; for the heavy parts, such as proxies, browsers and scheduling, a platform like Scrape.ly works the same from both.
Is web scraping with Java legal?
The language doesn't matter; what you scrape and how does. Collecting publicly available data is generally lawful in many places, but respect site terms, avoid overloading servers and handle personal data under privacy law. This isn't legal advice; our guide on whether web scraping is legal covers the details.
