RAG Web Browser

🌐 RAG Web Browser

Search Google, open the pages that matter, and hand your model clean Markdown instead of a link it still has to read.

Built for RAG pipelines, agents, and research jobs that need the text of a public page in the dataset before the next step of the workflow runs.

✨ Features

• 🔎 query is a Google search or a single public URL. A URL skips search and fetches that page.
• ⚡ Google results come from fast search mode, which returns a results page in a second or two. Any page fast search mode can't return is loaded in browser mode instead, and the run carries on.
• 📄 maxResults chooses how many organic results to store, from 1 to 100 (the run keeps every extracted page in memory until it returns). A blocked search page is retried. A Google results page that still has no public links is loaded once in a remote scraping browser. A result URL that is not a public document is tried again on a new session, then in headless Chrome. Once maxResults pages are stored the run ends. If it is still short and a result URL is missing, the run fails and writes nothing. When Google returns no further public pages and every result URL was stored, the shorter clean set is stored.
• 📝 outputFormats returns markdown, text, html, or any combination. Markdown links are absolute, so they still work after the page is gone.
• 🧹 removeElementsCssSelector drops the matching nodes before conversion. Navigation, scripts, and the other default chrome stay out of the prompt.
• 🍪 removeCookieWarnings also strips common consent containers from the HTML before conversion.
• 📖 htmlTransformer set to readableText keeps the readable article when extraction is confident, and keeps the cleaned HTML when it is not. none only applies the element removal.
• 🏷 Each handled page can include metadata taken from that document: title, description, author, keywords, language, canonical URL, Open Graph tags, and JSON-LD. A field that the page does not have is left off the item.
• 🔁 A blocked page is fetched again on a new HTTP session. If HTTP still does not return it, headless Chrome loads that URL through the residential proxy. If the page is still missing, one remote scraping-browser session loads it. Reddit, Facebook, Wikipedia, and X also get one public alternate host for the same document. A login wall, cookie consent wall, browser check, or empty shell is not stored.
• 🎬 An image, audio, video, or font URL is not stored. The run fails instead of writing a skip row when that was the only requested target.

🚀 Why teams use this

HTTP first, for every page that already returns. Headless Chrome loads only a result URL that HTTP did not store, using the residential proxy and no saved cookies. Markdown is converted from the HTML the page sent.

It is a Scrapely run with a dataset, a timeout, and retries. requestTimeoutSecs bounds the whole query. serpMaxRetries retries search. maxRequestRetries retries a page. desiredConcurrency caps how many pages are open at once. Set it to 0 and the run picks a small pool from the memory it was given.

The dataset contains only handled public pages. A direct URL that cannot be stored fails the run and writes nothing. A search that stops while later Google pages are still unread fails the same way. A result URL that stays blocked is retried on a new session. Once maxResults pages are stored the run ends, even if an earlier result URL stayed blocked. If the run is still short and a result URL is missing, it fails and writes nothing. When Google has no further public pages and every result URL was stored, the shorter clean set is stored and the run says why.

proxyConfiguration chooses how target pages are fetched. When Scrapely Proxy is on and no group is set, pages use the residential group. Google Search runs in fast search mode (US results, English interface). If fast search mode fails for a results page (an error, a timeout, or no results on the first page), the log says Switching to browser mode for this query page and that page runs in browser mode: the platform's remote browser with a US exit when the run has one, otherwise a local headless Chromium behind the residential proxy. In browser mode, if Google keeps answering with its consent or unusual-traffic page, the run fails and says so.

💰 Pricing

Pay per event: SERP page, $1.50 per 1,000 Google results pages ($0.0015 each). One event is charged for each Google results page the search retrieved, in either mode; a run that reads 2 results pages to store 15 documents is charged 2 events. A direct URL doesn't search, so it isn't charged. A run that fails and writes nothing isn't charged. The run checks its maximum total charge before each results page; when the next page isn't covered, it stops searching and stores the clean pages it already has.

💡 Use cases

• Give an LLM or agent live web access: search Google and read the top pages as Markdown.
• Ground chatbot answers in fresh sources with their URLs.
• Turn a single URL into clean Markdown for retrieval-augmented generation.

🧪 Example input

{
  "htmlTransformer": "none",
  "proxyConfiguration": {
    "useScrapelyProxy": true
  },
  "query": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"
}

🧾 Sample output

One item from a real run of the example input, shortened. Long text is cut and lists show their first entries.

{
  "crawl": {
    "httpStatusCode": 200,
    "httpStatusMessage": "OK",
    "loadedAt": "2026-10-12T22:14:51.181Z",
    "uniqueKey": "AQUoPf3PyX",
    "requestStatus": "handled"
  },
  "metadata": {
    "title": "Retrieval-augmented generation - Wikipedia",
    "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
    "languageCode": "en",
    "canonicalUrl": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
    "openGraph": [
      {
        "property": "…",
        "content": "…"
      },
      {
        "property": "…",
        "content": "…"
      }
    ],
    "headers": {
      "accept-ranges": "bytes",
      "age": "7110",
      "cache-control": "private, s-maxage=0, max-age=0, must-revalidate, no-transform",
      "content-encoding": "gzip",
      "content-language": "en",
      "content-length": "46696",
      "content-security-policy": "script-src 'unsafe-eval' blob: 'self' meta.wikimedia.org *.wikimedia.org *.wikipedia.org *.wikinews.org *.wiktionary.org *.wikibooks.org *.wikiversity.org *.…",
      "content-type": "text/html; charset=UTF-8",
      "date": "Mon, 12 Oct 2026 20:16:18 GMT",
      "last-modified": "Mon, 12 Oct 2026 18:31:50 GMT",
      "nel": "{ \"report_to\": \"wm_nel\", \"max_age\": 604800, \"failure_fraction\": 0.05, \"success_fraction\": 0.0}",
      "report-to": "{ \"group\": \"wm_nel\", \"max_age\": 604800, \"endpoints\": [{ \"url\": \"https://intake-logging.wikimedia.org/v1/events?stream=w3c.reportingapi.network_error&schema_u…",
      "reporting-endpoints": "csp-report-to-endpoint='/w/api.php?action=cspreport&format=json';",
      "server": "mw-web.eqiad.main-6df4688ffd-gg2zh",
      "server-timing": "cache;desc=\"hit-front\", host;desc=\"cp3073\",co_id;desc=\"1572652044\"",
      "strict-transport-security": "max-age=106384710; includeSubDomains; preload",
      "vary": "Accept-Encoding,X-Subdomain,Cookie,Authorization,User-Agent",
      "x-cache": "cp3073 miss, cp3073 hit/1",
      "x-cache-status": "hit-front",
      "x-client-ip": "78.133.77.167",
      "x-content-type-options": "nosniff",
      "x-request-id": "a56e96a4-4cb2-42cb-8af6-761fc6df3095"
    }
  },
  "query": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
  "markdown": "Retrieval-augmented generation - Wikipedia\n[Jump to content](#bodyContent)\n[![](https://en.wikipedia.org/static/images/icons/enwiki-25.svg)\n![Wikipedia](http…"
}

📥 Input

query is required. Everything else has a default.

{
  "query": "retrieval-augmented generation site:en.wikipedia.org",
  "maxResults": 1,
  "outputFormats": ["markdown"],
  "scrapingTool": "raw-http",
  "proxyConfiguration": {
    "useScrapelyProxy": true
  }
}

📤 Output

One dataset item per fetched URL. Empty values are omitted. A real false or 0 would stay; this actor does not invent either one.

Search items include query, searchResult (title, url, resultType, and description when Google sent one), and crawl:

• crawl.httpStatusCode
• crawl.httpStatusMessage (OK when the page was handled)
• crawl.loadedAt
• crawl.uniqueKey
• crawl.requestStatus (handled for every stored page)
• crawl.debug only when debugMode is true

metadata is filled from the response and the document:

• metadata.url
• metadata.redirectedUrl when the final URL differs from the requested one
• metadata.title, metadata.description, metadata.author, metadata.keywords, metadata.languageCode
• metadata.canonicalUrl
• metadata.openGraph as {property, content} pairs
• metadata.jsonLd when a JSON-LD script parses
• metadata.headers from the response, without cookie headers

markdown, text, and html appear only for the formats you requested, and only when that conversion produced text. A direct URL has no searchResult. A media URL has no body.