GitHub Scraper

🐙 GitHub Scraper

Turn GitHub search, Trending, topics and user or organisation pages into a clean spreadsheet of repositories. Every row has stars, forks, open issues, language, topics, license, homepage, owner and the created, updated and pushed dates. Trending rows add the stars gained today, this week or this month. No GitHub account or token needed, and it costs $0.80 per 1,000 repositories.

✨ Why use GitHub Scraper?

• 🔎 Every way into GitHub in one actor: search queries with full GitHub syntax, Trending by language and period, topics, all repositories of a user or organisation, and plain repository URLs.
• 🚀 Past the 1,000-result cap: GitHub stops every search at 1,000 results. Bigger queries are split by star range or creation date automatically, so one run can return 10,000+ repositories without duplicates.
• 📈 Trending with growth: stars gained in the period, growth percent, Trending rank and the top contributors, enriched with topics, license and dates.
• 🔐 No login, no token: public data only. Rate limits are handled for you by rotating proxy sessions and backing off when GitHub asks.
• 🧹 Clean, typed output: numbers are numbers, dates are ISO 8601, licenses are SPDX ids, and the same repository is never saved (or charged) twice.
• 💸 Cheap: $0.80 per 1,000 repositories, with optional README text and watcher counts.

🎯 Use cases

• 🧑‍💻 Developer tools discovery: find the most-starred libraries for a language, topic or problem.
• 📊 Open-source market research: track star growth and adoption of frameworks and competitors over time.
• 📰 Newsletters and curation: pull today's or this week's Trending repositories with descriptions and topics.
• 🤝 Lead generation for dev-tool companies: list the active repositories of organisations, with homepages and owners.
• 🤖 AI datasets: collect repositories with their README text for a topic such as llm or rag.
• 🏢 Portfolio monitoring: watch all repositories of your organisation or of the projects you depend on.

🚀 How to use

1. Type one or more search queries (for example web scraping or language:rust stars:>1000), and/or paste GitHub URLs (repositories, users, organisations, topics, Trending or search pages), and/or add topics.
2. Optional: turn on Scrape GitHub Trending and pick languages and a period.
3. Optional: narrow results with filters (language, stars, pushed and created dates, license, archived, forks) and choose a sort.
4. Set Max results and click Start.
5. Download the dataset as JSON, CSV, Excel or HTML, or read it through the API.

📥 Input

{
  "searchQueries": ["web scraping", "language:rust stars:>5000"],
  "startUrls": ["https://github.com/vercel", "https://github.com/trending/python?since=weekly", "psf/requests"],
  "topics": ["llm"],
  "maxResults": 200,
  "minStars": 100,
  "pushedAfter": "2026-01-01",
  "sortBy": "stars"
}

📤 Output

Example row from a test run (trimmed):

{
  "id": 529502,
  "fullName": "scrapy/scrapy",
  "owner": "scrapy",
  "ownerType": "Organization",
  "url": "https://github.com/scrapy/scrapy",
  "description": "Scrapy, a fast high-level web crawling & scraping framework for Python.",
  "homepage": "https://scrapy.org",
  "stars": 64807,
  "forks": 12002,
  "openIssues": 246,
  "language": "Python",
  "topics": ["crawler", "crawling", "framework", "python", "scraping", "web-scraping"],
  "license": "BSD-3-Clause",
  "isFork": false,
  "isArchived": false,
  "createdAt": "2010-02-22T02:01:14Z",
  "pushedAt": "2026-10-14T16:21:55Z",
  "rank": 1,
  "source": "search",
  "sourceInput": "web scraping"
}

A Trending row also carries (sample values):

{
  "trendingRank": 1,
  "trendingPeriod": "daily",
  "starsToday": 1520,
  "starsGrowthPercent": 4.2,
  "builtBy": [{"username": "example-dev", "avatar": "https://avatars.githubusercontent.com/u/0"}]
}

💰 Pricing

You pay $0.80 per 1,000 repositories ($0.0008 each), charged only for rows saved to the dataset. Platform usage (a little compute at 256 MB and datacenter proxy traffic) is billed on top and is usually a fraction of a cent per run.

Example: 1,000 repositories ≈ $0.80 plus about $0.01 of usage.

❓ FAQ

Is it legal to scrape GitHub? The actor reads only public repository data that GitHub shows to anyone without logging in. Check GitHub's terms and your local rules for your use case, especially before republishing data.

Do I need a GitHub account, token or my own proxy? No. Datacenter proxies are on by default and each proxy session has its own GitHub rate limit, so the actor rotates sessions when GitHub slows it down.

How many results can I get? As many as GitHub has. Searches with more than 1,000 matches are split by star range (when sorted by stars) or by creation date, so 10,000+ results per query are possible. Trending shows about 10 to 25 repositories per language and period.

Why is a field empty? Many repositories have no homepage, license or topics. watchers, networkCount and parentFullName are filled only with Include watchers turned on, and readmeContent only with Include README. Trending fields are filled only for Trending rows.

Can I schedule it or use the API? Yes. Schedule a daily Trending run, or start runs over the API and read the dataset as JSON or CSV.

Can I combine inputs? Yes. Queries, URLs, topics and Trending can be mixed in one run; a repository found twice is saved once.

⚠️ Limits

• Public repositories only. Private repositories, stars of a single user, issues, pull requests and commits are not covered.
• Trending lists exactly what github.com/trending shows (often 10 to 25 repositories). Trending developers are not included.
• watchers in GitHub's sense (people subscribed to notifications) needs one extra lookup per repository, so Include watchers makes big runs slower.
• Search results reflect GitHub's search index, which can lag a few minutes behind the repository page.
• A single day with more than 1,000 new matching repositories, or a single star count with more than 1,000 repositories, is capped at 1,000 by GitHub.