--- url: https://scrape.ly/blog/category/guides markdown: https://scrape.ly/blog/category/guides.md --- Blog category # Guides How web scraping works: crawling, APIs, browsers, exports and the legal side, explained with working code. [RSS feed](https://scrape.ly/blog/rss.xml) - [![Pixel-art robot with an orange >\_ on its terminal head carries a row of data from a web page on the left into a table on the right that is filling up row by row](https://scrape.ly/media/what-is-web-scraping.png?w=750)](https://scrape.ly/blog/what-is-web-scraping.md) ### [What Is Web Scraping? How It Works and What It Is Used For](https://scrape.ly/blog/what-is-web-scraping.md) [What is web scraping? The automated collection of data from websites into structured records. See how it works step by step with tested Python code, what teams use it for, how it differs from crawling, APIs and screen scraping, and how to do it without code.](https://scrape.ly/blog/what-is-web-scraping.md) - [![Pixel-art robot with an orange >\_ on its terminal head pulls a data card out of a monitor showing the blue and yellow Python logo beside lines of code, and drops it into a crate](https://scrape.ly/media/web-scraping-python.png?w=750)](https://scrape.ly/blog/web-scraping-python.md) ### [Web Scraping With Python: A Complete 2026 Guide](https://scrape.ly/blog/web-scraping-python.md) [A complete guide to web scraping with Python: requests and BeautifulSoup, CSS selectors, pagination, CSV and JSON export, polite delays and retries, async httpx, Playwright for JavaScript pages, Scrapy, and how to run it all in the cloud. Every script tested on 23 October 2026.](https://scrape.ly/blog/web-scraping-python.md) - [![Pixel-art robot with an orange >\_ on its terminal head stands before a pair of balanced golden scales of justice, holding out a data card](https://scrape.ly/media/is-web-scraping-legal.png?w=750)](https://scrape.ly/blog/is-web-scraping-legal.md) ### [Is Web Scraping Legal? US, EU and UK Law, Court Cases and a Checklist (2026)](https://scrape.ly/blog/is-web-scraping-legal.md) [Is web scraping legal? What US, EU and UK law actually says in 2026, from Van Buren and hiQ v. LinkedIn to Meta and X v. Bright Data, GDPR, database rights, robots.txt and the newest SerpApi rulings, plus a practical checklist.](https://scrape.ly/blog/is-web-scraping-legal.md) - [![Pixel-art robot with an orange >\_ on its terminal head pulls a data card out of a browser window showing a web page and drops it into a crate of collected data](https://scrape.ly/media/how-to-scrape-a-website.png?w=750)](https://scrape.ly/blog/how-to-scrape-a-website.md) ### [How to Scrape Data From a Website: 4 Methods, No-Code to Python](https://scrape.ly/blog/how-to-scrape-a-website.md) [How to scrape data from a website four ways: a ready-made scraper with no code, a generic cloud scraper with one extraction function, a Python script you run yourself, and an AI agent over MCP. Each with real runs, exact costs and tested code.](https://scrape.ly/blog/how-to-scrape-a-website.md) - [![Pixel-art robot with an orange >\_ on its terminal head walks past a raised red-and-white boom barrier, carrying a data card from the web page behind it towards a crate](https://scrape.ly/media/web-scraping-without-getting-blocked.png?w=750)](https://scrape.ly/blog/web-scraping-without-getting-blocked.md) ### [Web Scraping Without Getting Blocked: 10 Techniques That Work](https://scrape.ly/blog/web-scraping-without-getting-blocked.md) [Web scraping without getting blocked comes down to ten habits: real headers, pacing, sessions, good IPs, real browsers, clean fingerprints, challenge handling, honeypot checks, robots.txt and caching. Tested Python for each, and how Scrape.ly handles them for you.](https://scrape.ly/blog/web-scraping-without-getting-blocked.md) - [![Pixel-art robot with an orange >\_ on its terminal head reaches for the last empty box on a clipboard checklist whose first four items carry orange ticks, with a web page standing behind it](https://scrape.ly/media/web-scraping-best-practices.png?w=750)](https://scrape.ly/blog/web-scraping-best-practices.md) ### [12 Web Scraping Best Practices for Reliable, Responsible Scrapers](https://scrape.ly/blog/web-scraping-best-practices.md) [Twelve web scraping best practices that keep scrapers polite, unblocked and accurate: robots.txt, rate limits, caching, retries, validation, monitoring, personal data and more. Each with tested Python code and how to do it on Scrape.ly.](https://scrape.ly/blog/web-scraping-best-practices.md) - [![Pixel-art scene in two halves: on the left a small robot walks along a cable linking web pages, on the right a robot with an orange >\_ on its terminal head pulls a data card out of a single web page](https://scrape.ly/media/web-crawling-vs-web-scraping.png?w=750)](https://scrape.ly/blog/web-crawling-vs-web-scraping.md) ### [Web Crawling vs Web Scraping: The Difference, With Code and Real Runs](https://scrape.ly/blog/web-crawling-vs-web-scraping.md) [Web crawling finds pages; web scraping pulls data out of them. See the difference in one Python script, in a real crawl of crawlee.dev that cost $0.0031, and learn which one your project needs.](https://scrape.ly/blog/web-crawling-vs-web-scraping.md) - [![Pixel-art robot with an orange >\_ on its terminal head carries a data card out between two routes: a cable from a browser window on the left and a cable from an API socket with curly braces and a Scrape.ly-branded plug on the right](https://scrape.ly/media/web-scraping-vs-api.png?w=750)](https://scrape.ly/blog/web-scraping-vs-api.md) ### [Web Scraping vs API: Which to Use, What Each Costs, and How to Get Both](https://scrape.ly/blog/web-scraping-vs-api.md) [Web scraping vs API: an official API gives you what the platform chooses to expose, on its terms and prices; scraping gets you what any visitor sees. Real 2026 prices for X, Google Places and Reddit, and how to call a scraper like an API.](https://scrape.ly/blog/web-scraping-vs-api.md) - [![Pixel-art robot with an orange >\_ on its terminal head stands thinking between two matching arcade cabinets, one with the Playwright logo and one with the green Selenium logo, next to a crate bearing the Scrape.ly logo](https://scrape.ly/media/playwright-vs-selenium.png?w=750)](https://scrape.ly/blog/playwright-vs-selenium.md) ### [Playwright vs Selenium for Web Scraping: The Same Scraper in Both (Python)](https://scrape.ly/blog/playwright-vs-selenium.md) [Playwright vs Selenium for web scraping: languages, browsers, waiting, network interception, parallel sessions and detection, from the official docs, plus the same Python scraper written in both and tested on 22 October 2026.](https://scrape.ly/blog/playwright-vs-selenium.md) - [![Pixel-art robot with an orange >\_ on its terminal head stands thinking between two matching browser windows on stands, one with the Playwright logo and one with the green Puppeteer logo, next to a crate bearing the Scrape.ly logo](https://scrape.ly/media/playwright-vs-puppeteer.png?w=750)](https://scrape.ly/blog/playwright-vs-puppeteer.md) ### [Playwright vs Puppeteer for Web Scraping: The Same Scraper in Both (Node.js)](https://scrape.ly/blog/playwright-vs-puppeteer.md) [Playwright vs Puppeteer for web scraping: languages, browsers, auto-waiting, locators, network interception, tracing and stealth plugins, checked on the official docs on 23 October 2026, plus the same Node.js scraper written in both and tested.](https://scrape.ly/blog/playwright-vs-puppeteer.md) - [![Pixel-art robot with an orange >\_ on its terminal head walks out through an open iron gate whose arch carries the orange Cloudflare cloud logo, carrying a data card from the web page inside towards a crate](https://scrape.ly/media/bypass-cloudflare-scraping.png?w=750)](https://scrape.ly/blog/bypass-cloudflare-scraping.md) ### [How to Bypass Cloudflare When Scraping (Responsibly)](https://scrape.ly/blog/bypass-cloudflare-scraping.md) [What Cloudflare actually checks (bot scores, TLS fingerprints, challenge pages, Turnstile), from its own docs, why a plain HTTP request fails, and the responsible ways to scrape public pages behind it, up to a hosted browser that solves Turnstile.](https://scrape.ly/blog/bypass-cloudflare-scraping.md) - [![Pixel-art robot with an orange >\_ on its terminal head, disguised in a dark fedora and a raised collar, pulls a data card out of a browser window showing the Playwright logo and drops it into a crate](https://scrape.ly/media/playwright-stealth.png?w=750)](https://scrape.ly/blog/playwright-stealth.md) ### [Playwright Stealth: How It Works, Tested Setup, and When It Is Not Enough](https://scrape.ly/blog/playwright-stealth.md) [Playwright stealth means patching a headless browser so it hides the signs of automation. The Node and Python packages, who maintains them, what they patch, tested setup code with real bot.sannysoft.com results, and where stealth stops working.](https://scrape.ly/blog/playwright-stealth.md) ## The internet is full of data. Start owning yours. Scrape.ly gives you the infrastructure, the tooling, and the team to extract, process, and operationalize web data at any scale. Setup takes minutes. The advantage lasts forever. [Get Started Free](https://my.scrape.ly/register) [Chat with Support](mailto:support@scrape.ly?subject=Question%20about%20Scrape.ly\&body=Hi%2C%20I%20have%20a%20question%20about%20Scrape.ly.) No credit card. No setup calls.