Find bots and AI crawlers in an access log
Paste access-log lines; only bot rows are kept, each with bot name, category, producer and an AI-crawler flag.
Updated November 13, 2026
What this example returns
The dataset’s “Bots” view shows 7 columns: User-Agent, Count, Bot name, Bot category, Bot producer, AI bot, and Bot info. See every output field.
What each setting does
The input this example runs User-Agent Parser with. “Try this input” opens it in the console, where you can run it as-is or change any field first.
- Log text
logText 66.249.66.1 - - [12/Nov/2026:10:00:00 +0000] "GET /pricing HTTP/1.1" 200 5120 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.66.1 - - [12/Nov/2026:10:00:04 +0000] "GET /docs HTTP/1.1" 200 2048 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com…
Optional. Paste access-log lines (Apache/Nginx combined format, IIS/W3C, JSON lines) or one User-Agent per line: the User-Agent of every line is picked out and parsed together with the list above. Lines without one are skipped for free.
- Merge repeated strings
deduplicate Yes
On: one row per distinct User-Agent, with how often it appeared in `count` (you pay once per distinct string). Off: one row per entry, in input order.
- Bots
botFilter Only bots, crawlers and HTTP clients
Keep only bot rows (e.g. to spot AI crawlers in your logs) or only real browsers and apps. Rows left out are not charged.
Default: Keep all rows → this example uses Only bots, crawlers and HTTP clients
{
"userAgents": [],
"logText": "66.249.66.1 - - [12/Nov/2026:10:00:00 +0000] \"GET /pricing HTTP/1.1\" 200 5120 \"-\" \"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)\"\n66.249.66.1 - - [12/Nov/2026:10:00:04 +0000] \"GET /docs HTTP/1.1\" 200 2048 \"-\" \"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)\"\n20.171.207.1 - - [12/Nov/2026:10:01:00 +0000] \"GET /blog/post-1 HTTP/1.1\" 200 8192 \"-\" \"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)\"\n3.224.1.1 - - [12/Nov/2026:10:01:30 +0000] \"GET /blog/post-2 HTTP/1.1\" 200 8192 \"-\" \"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)\"\n44.1.1.1 - - [12/Nov/2026:10:02:00 +0000] \"GET /blog/post-3 HTTP/1.1\" 200 8192 \"-\" \"Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)\"\n52.1.1.1 - - [12/Nov/2026:10:02:10 +0000] \"GET /faq HTTP/1.1\" 200 1024 \"-\" \"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot\"\n157.55.39.1 - - [12/Nov/2026:10:03:00 +0000] \"GET / HTTP/1.1\" 200 4096 \"-\" \"Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)\"\n54.36.148.1 - - [12/Nov/2026:10:03:30 +0000] \"GET /robots.txt HTTP/1.1\" 200 120 \"-\" \"Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)\"\n69.171.249.1 - - [12/Nov/2026:10:04:00 +0000] \"GET /og HTTP/1.1\" 200 300 \"-\" \"facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)\"\n216.144.248.1 - - [12/Nov/2026:10:04:30 +0000] \"HEAD / HTTP/1.1\" 200 0 \"-\" \"Mozilla/5.0 (compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)\"\n10.0.0.5 - - [12/Nov/2026:10:05:00 +0000] \"GET /api/v1/items HTTP/1.1\" 200 900 \"-\" \"python-requests/2.32.3\"\n10.0.0.6 - - [12/Nov/2026:10:05:10 +0000] \"GET /api/v1/items HTTP/1.1\" 200 900 \"-\" \"Go-http-client/2.0\"\n10.0.0.7 - - [12/Nov/2026:10:05:20 +0000] \"GET /app HTTP/1.1\" 200 900 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/131.0.0.0 Safari/537.36\"\n81.2.69.1 - - [12/Nov/2026:10:06:00 +0000] \"GET /pricing HTTP/1.1\" 200 5120 \"https://www.google.com/\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36\"\n81.2.69.2 - - [12/Nov/2026:10:06:05 +0000] \"GET /pricing HTTP/1.1\" 200 5120 \"-\" \"Mozilla/5.0 (iPhone; CPU iPhone OS 18_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.1 Mobile/15E148 Safari/604.1\"\n81.2.69.3 - - [12/Nov/2026:10:06:09 +0000] \"GET / HTTP/1.1\" 304 0 \"-\" \"-\"\n{\"time\":\"2026-11-12T10:07:00Z\",\"ip\":\"10.0.0.8\",\"request\":{\"headers\":{\"user-agent\":\"curl/8.10.1\"}}}",
"deduplicate": true,
"botFilter": "bots"
}What it costs
You pay for the events the actor reports, such as each result or page it produces. Platform usage is included in the event prices: compute, proxies, browser and storage cost nothing extra.
- User-Agent parsedfrom $0.00027One saved row: a User-Agent string parsed into browser, engine, OS, device, CPU and bot columns. Entries that are not User-Agents, log lines without one and rows left out by the bot filter are free.
When to use this example
From the User-Agent Parser readme:
How many strings can I parse?
As many as you like: there is no limit on the list, the log text or the dataset. Large inputs are saved in batches as they are parsed.
Known limit
count with merging on is over the whole input of the run.
Other ways to run User-Agent Parser
- Parse a mixed list of browsers, devices and botsDesktop, phone, tablet, TV, console and in-app browsers plus search bots, AI crawlers and HTTP clients, with repeats merged.
- One row per entry, browsers onlyKeep input order with one row per entry, drop bots and HTTP clients, and stop at a row limit.
About User-Agent Parser
Parse User-Agent strings in bulk into browser, version, OS, device type, brand and model, rendering engine and CPU, plus a bot verdict with the bot's name and category (search engines, AI crawlers, monitors, HTTP clients, headless browsers). Paste strings, raw access-log lines or a dataset. No login or API key needed.
- Price:
- from $0.27 / 1,000 results
- Developer:
- Querystone
- Categories:
- Developer tools, Automation
Related actors
RAG Web Browserby QuerystoneWeb browser for AI agents and RAG pipelines: search Google, fetch the top result pages, and return clean Markdown or text ready for LLMs.…
Google Search Scraperby QuerystoneScrape Google Search results by query and country in fast search mode: organic results, People Also Ask, and related searches with…
Website Content Crawlerby QuerystoneCrawl whole websites and extract clean text and Markdown for AI models, vector databases, and RAG pipelines. Removes navigation and…
Latest articles
GuidesWeb Scraper Chrome Extensions: 4 Good Ones, and When to Use a Cloud Scraper
The best web scraper Chrome extensions in 2026 (Web Scraper, Instant Data Scraper, Data Miner and Evomi Web Data), what each is good at, where every extension stops, and how a cloud scraper takes over for schedules, thousands of pages and API access.
Read the article
AI agents12 Best MCP Servers in 2026: Web Data, Code, Docs and Browsers
The best MCP servers in 2026, grouped by job: web data, documentation, code, browsers, errors, databases and the official reference servers. Each with its hosted URL or install command, how it signs in, and the caveats its own README states.
Read the article
Guidesn8n Puppeteer Node: Puppeteer and Playwright in n8n Without Self-Hosting Chrome
The n8n Puppeteer node needs Chromium inside your n8n container and installs only on self-hosted n8n. How it and the Playwright nodes install, where they break, and how to run the same browser scrape on Scrape.ly from n8n instead, with a real tested run.
Read the article
The internet is full of data.
Start owning yours.
Scrape.ly gives you the infrastructure, the tooling, and the team to extract, process, and operationalize web data at any scale. Setup takes minutes.