GitHub Repo to LLM Context

🧠 GitHub Repo to LLM Context

Turn any public GitHub repository into one clean, LLM-ready document: a directory tree followed by every source file in a language-tagged code fence, with an exact token count. Paste it into ChatGPT, Claude, Gemini or your own agent, or feed it to a RAG pipeline. No GitHub token, no login, and a typical repository is done in seconds.

✨ Why use GitHub Repo to LLM Context?

• ⚡ Fast — downloads the whole repository as one archive instead of crawling file by file, so there are no API rate limits and no token to create.
• 🧹 Noise removed by default — skips node_modules, vendor, dist, build output, lockfiles, minified bundles, source maps, images and other binaries, so your context window holds code, not junk.
• 🔢 Exact token counts — every document, part and file is measured with the o200k_base tokenizer (GPT-4o family), so you know before you paste whether it fits.
• 🎯 Precise scope — pick a branch, tag or commit, a single folder (/tree/main/src), and .gitignore-style include and exclude patterns.
• 📦 Ready for agents — the full document is saved as a .md (or .xml) file with a download link, and the dataset can hold the text, a compact summary, or one row per file.
• 💸 Cheap — you pay per repository plus a small amount per 10,000 tokens of output.

🎯 Use cases

• 🤖 Ask an LLM about a codebase — "explain the architecture", "find the bug", "write tests for this module".
• 🧑‍💻 Coding agents and MCP tools — give an agent the whole library it is working with, in one call.
• 📚 RAG and embeddings — per-file rows with language, path, line count and token count, ready to chunk and index.
• 🔍 Code review and audits — line-numbered output for a folder or a release tag.
• 📝 Documentation — turn a project's docs/ folder into one file for summarising or translating.
• 📊 Repository comparison — run many repositories at once and compare size, languages and token footprint.

🚀 How to use

1. Paste one or more repositories into GitHub repositories (https://github.com/owner/repo, owner/repo, or a folder link such as https://github.com/owner/repo/tree/main/src).
2. Optionally set a branch or tag, include and exclude patterns, or a token limit.
3. Click Start.
4. Open the Download link for the full document, or export the dataset as JSON, CSV, Excel or Markdown, or read it through the API.

📥 Input

{
  "repoUrls": [
    "https://github.com/pallets/itsdangerous",
    "https://github.com/psf/requests/tree/main/src/requests"
  ],
  "excludePatterns": ["tests/", "docs/"],
  "datasetRows": "document"
}

📤 Output

One row per repository (or per part, for very large repositories):

{
  "type": "repository",
  "repository": "pallets/itsdangerous",
  "repoUrl": "https://github.com/pallets/itsdangerous",
  "ref": "main",
  "commitSha": "672971d66a2ef9f85151e53283113f33d642dabd",
  "subdirectory": null,
  "fileCount": 46,
  "totalFilesInRepo": 50,
  "tokenEstimate": 25513,
  "truncated": false,
  "skippedFiles": {"lockfiles": 1, "binary": 3, "vendoredOrGenerated": 0, "tooLarge": 0, "excludedByPattern": 0},
  "tokenizer": "o200k_base",
  "documentSizeBytes": 103190,
  "documentKey": "pallets-itsdangerous-main.md",
  "documentUrl": "https://api.scrape.ly/v2/key-value-stores/<store>/records/pallets-itsdangerous-main.md?signature=<signature>",
  "format": "markdown",
  "languageTokens": {"python": 13620, "rst": 5442, "yaml": 2104, "toml": 1294, "markdown": 785},
  "convertedAt": "2026-10-16T00:29:38+00:00",
  "part": 1,
  "totalParts": 1,
  "filesInPart": 46,
  "partTokenEstimate": 25513,
  "content": "# pallets/itsdangerous\n\n- Repository: https://github.com/pallets/itsdangerous\n- Ref: main (commit 672971d66a2e)\n- Files: 46 | Estimated tokens: 25,513\n\n## Directory structure\n\n```\nitsdangerous/\n├── .devcontainer/\n│   ├── devcontainer.json\n...\n```\n\n## Files\n\n### `README.md`\n\n```markdown\n# ItsDangerous\n..."
}

With datasetRows set to files or both, you also get one row per file (type: file) with path, language, lines, sizeBytes, tokenEstimate, a GitHub url pinned to the commit, and content. Each repository still gets its own type: repository row (the summary in files mode, the document in both mode), so files with maxFilesPerRepo 10 returns up to 11 rows per repository: 1 repository row plus at most 10 file rows.

A repository that fails (not found, wrong branch, nothing matched your filters) saves no row. It is listed under failedInputs (input, reason) in the run's OUTPUT record and named in the status message.

💰 Pricing

You pay $0.012 per repository converted plus $0.0005 per 10,000 tokens of output ($0.05 per million tokens), plus a little platform usage.

• A small library (about 25,000 tokens) costs about $0.0135 in events.
• A mid-size project (about 170,000 tokens) costs about $0.021.
• 100 small repositories cost about $1.35.

Repositories that fail (not found, wrong branch, nothing matched your filters) are never charged. Set Max tokens per repository to cap the cost of very large repositories.

❓ FAQ

Do I need a GitHub token or account? No. Public repositories are downloaded as an archive straight from GitHub, without the API.

Does it work with private repositories? No, only public ones.

How big can a repository be? Archives up to 300 MB by default (adjustable up to 2 GB). Documents larger than about 3 MB are split into several dataset rows; the download link always holds the complete file.

Which token count is it? Exact counts for OpenAI's o200k_base tokenizer. Other models (Claude, Gemini, Llama) tokenise differently, usually within about 10 to 20%.

What if a file is missing from the document? Check skippedFiles: it counts every file left out and why. Turn off a filter or raise the size limit to include it.

Can I schedule it or call it from code or an AI agent? Yes. Use the API, schedules or the MCP server; summary mode keeps responses small and gives the agent a download link.

⚠️ Limits

• Public GitHub repositories only (no GitLab, Bitbucket or private repositories yet).
• Git submodules and Git LFS objects are not downloaded (LFS files appear as small pointer files).
• Files over 1 MB are always skipped; binary files are never embedded.
• Repository metadata such as stars, topics and contributors is not included.