--- title: "AI agents articles | Scrape.ly" description: "Web data for AI agents and LLM apps: MCP servers, agent workflows and clean data for models." url: https://scrape.ly/blog/category/ai-agents markdown: https://scrape.ly/blog/category/ai-agents.md --- Blog category # AI agents Web data for AI agents and LLM apps: MCP servers, agent workflows and clean data for models. [RSS feed](https://scrape.ly/blog/rss.xml) - [![Pixel-art robot with an orange >\_ on its terminal head plugs a cable into a server rack labelled MCP with the Scrape.ly logo on its front, while two more cables run into it from web pages on the left](https://scrape.ly/media/best-mcp-servers-for-web-scraping.png?w=750)](https://scrape.ly/blog/best-mcp-servers-for-web-scraping.md) ### [6 Best Web Scraping MCP Servers in 2026 (Setup for Claude Code and Cursor)](https://scrape.ly/blog/best-mcp-servers-for-web-scraping.md) [The 6 best MCP servers for web scraping in 2026, compared on hosting, sign-in, tools, free tiers and spending control, with copy-paste setup for Claude Code, Cursor and VS Code and a real agent run.](https://scrape.ly/blog/best-mcp-servers-for-web-scraping.md) - [![Pixel-art robot with an orange >\_ on its terminal head, an AI sparkle in a speech bubble above it, pulls a data card out of a web page and drops it into a crate](https://scrape.ly/media/ai-web-scraping.png?w=750)](https://scrape.ly/blog/ai-web-scraping.md) ### [AI Web Scraping: How LLMs and Agents Extract Web Data](https://scrape.ly/blog/ai-web-scraping.md) [AI web scraping uses language models to read pages and fill structured fields, and AI agents to decide what to fetch. Learn how LLM extraction, agents and MCP work, when plain rules still win, and run both with working code and real runs.](https://scrape.ly/blog/ai-web-scraping.md) - [![Pixel-art robot with an orange >\_ on its terminal head carries a cable from a browser window showing the Playwright logo into a server rack labelled MCP](https://scrape.ly/media/playwright-mcp.png?w=750)](https://scrape.ly/blog/playwright-mcp.md) ### [Playwright MCP: Setup for Cursor, Claude Code and VS Code, and When to Use It for Scraping](https://scrape.ly/blog/playwright-mcp.md) [Playwright MCP lets an AI agent drive a real browser through accessibility snapshots. Set it up in Cursor, Claude Code or VS Code in a minute, learn the options and tools that matter, and see where it shines for scraping and where to hand off.](https://scrape.ly/blog/playwright-mcp.md) - [![Pixel-art robot with an orange >\_ on its terminal head, an AI sparkle in a speech bubble above it, reaches out to a standing text file with a folded corner whose dark plate reads LLMS, next to a web page](https://scrape.ly/media/what-is-llms-txt.png?w=750)](https://scrape.ly/blog/what-is-llms-txt.md) ### [What Is llms.txt? Format, Real Examples and How to Create One](https://scrape.ly/blog/what-is-llms-txt.md) [llms.txt is a Markdown file at the root of a website that gives AI agents a short, curated map of its content. The format rules from the v2 proposal, real files from Anthropic, OpenAI, Stripe and Cloudflare, llms-full.txt, Markdown twins, and how to create one in 11 seconds.](https://scrape.ly/blog/what-is-llms-txt.md) - [![Pixel-art robot with an orange >\_ on its terminal head pulls a page of text out of a monitor showing the cyan Crawl4AI logo above lines of code, and drops it into a crate](https://scrape.ly/media/crawl4ai-guide.png?w=750)](https://scrape.ly/blog/crawl4ai-guide.md) ### [Crawl4AI Tutorial: Setup, Examples and a Hosted Alternative](https://scrape.ly/blog/crawl4ai-guide.md) [A tested Crawl4AI tutorial for v0.9.5: install it, crawl a page to Markdown, extract JSON with CSS selectors or an LLM, deep crawl a site, run it in Docker or over MCP, and see when a hosted crawler saves you the infrastructure.](https://scrape.ly/blog/crawl4ai-guide.md) - [![Pixel-art robot with an orange >\_ on its terminal head carries a cable from an office tower with the blue LinkedIn logo on its sign into a server rack labelled MCP](https://scrape.ly/media/linkedin-mcp.png?w=750)](https://scrape.ly/blog/linkedin-mcp.md) ### [LinkedIn MCP: Give Claude, Cursor and ChatGPT LinkedIn Data Without Your Login](https://scrape.ly/blog/linkedin-mcp.md) [A LinkedIn MCP server lets your AI agent pull company pages, job listings, posts and ads into the chat. Set one up in Claude, Cursor or ChatGPT in a minute, without handing over your LinkedIn session, and see the exact tool calls and what came back.](https://scrape.ly/blog/linkedin-mcp.md) - [![Pixel-art robot with an orange >\_ on its terminal head carries a cable from a feed of posts headed by the orange Reddit logo into a server rack labelled MCP](https://scrape.ly/media/reddit-mcp.png?w=750)](https://scrape.ly/blog/reddit-mcp.md) ### [Reddit MCP: Connect Claude or Cursor to Reddit Data Without API Keys](https://scrape.ly/blog/reddit-mcp.md) [A Reddit MCP server lets Claude, Cursor or ChatGPT search Reddit, read subreddits and pull comment threads mid-conversation. Set one up in a minute without Reddit API credentials, and see the exact tool calls, a real run and what it cost.](https://scrape.ly/blog/reddit-mcp.md) - [![Pixel-art robot with an orange >\_ on its terminal head hands a data card from a web page on the left into a chat window whose header shows the ChatGPT logo and the orange Claude spark](https://scrape.ly/media/web-scraping-with-chatgpt.png?w=750)](https://scrape.ly/blog/web-scraping-with-chatgpt.md) ### [ChatGPT Web Scraping (and Claude): What Actually Works in 2026](https://scrape.ly/blog/web-scraping-with-chatgpt.md) [What ChatGPT and Claude can and can't do for web scraping today: search, code generation, code execution and MCP connectors. Plus a tested scraper you can ask either one to write, and how to connect them to Scrape.ly so they run real scrapers for you.](https://scrape.ly/blog/web-scraping-with-chatgpt.md) ## The internet is full of data. Start owning yours. Scrape.ly gives you the infrastructure, the tooling, and the team to extract, process, and operationalize web data at any scale. Setup takes minutes. The advantage lasts forever. [Get Started Free](https://my.scrape.ly/register) [Chat with Support](mailto:support@scrape.ly?subject=Question%20about%20Scrape.ly\&body=Hi%2C%20I%20have%20a%20question%20about%20Scrape.ly.) No credit card. No setup calls.