Tweet Scraper

🐦 Tweet Scraper

Public posts from X profiles, paginated to the size you ask for, with the author and the engagement counts that response actually includes.

Built for research, social listening, and anyone who needs a clean dataset of public posts without starting a browser.

✨ Features

• 👤 Profiles, page after page. A handle (NASA) or a profile URL walks the public timeline with the site’s own cursor until maxItems or the timeline ends.
• 🔗 Tweet URLs. A status URL returns that post plus the replies included on the logged-out status page.
• 💬 Conversation ids. A conversation id is read as that focal status page.
• 🔎 Search. A keyword, a search URL, or a wizard query is matched against Google's index of public x.com posts, then each post is read back from X before it is stored. Top returns the most relevant posts. Latest returns recent posts via the search index, newest first: it looks at the past day, then widens to the past week and month. It is typically a few hours behind live X, so posts from the last hour or two may be missing. Location operators (near:, geocode:, place:) are not supported by this route. from:handle with language, image, video, verified, quote, Twitter Blue, and minimum-count filters is still read from that profile timeline.
• 📅 Date window. start and end keep posts on or after the start date and before the end date, in UTC.
• 📊 Engagement that was sent. Retweets, replies, likes, quotes, and bookmarks stay on the item, including a real zero. View count is included when that response sends it.
• 🖼️ Media, cards, and mentions. Photos, videos, link cards, and @mentions are included when that post contains them.
• 🛟 One bad input does not abort the run. An unsupported URL or a closed surface is recorded and the next input is still fetched.
• 🧩 Your own fields. customMapFunction receives each post. The object it returns is the row that is stored.

🚀 Why teams use this

• Request-based. Profile timelines are cursor-paginated HTTP calls with Chrome’s TLS fingerprint. There is no browser startup.
• Production runs. The actor uses a residential proxy session. A blocked or rate-limited call is retried on a new session.
• A failed URL does not kill the run. The rest of the handles and URLs still land in the dataset.
• No invented fields. If a response does not include a flag or a count, that key is left off the item.

💡 Use cases

• Monitor what an account or brand posts on X, with engagement counts.
• Collect tweets for research, sentiment analysis, or news tracking.
• Archive public posts from a list of handles or tweet URLs.

🧪 Example input

{
  "maxItems": 20,
  "sort": "Latest",
  "tweetLanguage": "en",
  "twitterHandles": [
    "NASA"
  ]
}

🧾 Sample output

One item from a real run of the example input, shortened. Long text is cut and lists show their first entries.

{
  "type": "tweet",
  "id": "2109435410850193473",
  "url": "https://x.com/NASA/status/2109435410850193473",
  "twitterUrl": "https://twitter.com/NASA/status/2109435410850193473",
  "text": "RT @NASAJohnson: One. More. Day.\n\nFlight director applications close TOMORROW. Sharpen your pencils, steady your focus, and get ready to ma…",
  "retweetCount": 76,
  "replyCount": 0,
  "likeCount": 0,
  "quoteCount": 0,
  "createdAt": "Mon Oct 12 00:06:08 +0000 2026",
  "lang": "en",
  "bookmarkCount": 0,
  "author": {
    "type": "user",
    "userName": "NASA",
    "url": "https://x.com/NASA",
    "twitterUrl": "https://twitter.com/NASA",
    "id": "11348282",
    "name": "NASA",
    "isBlueVerified": true,
    "verifiedType": "Government",
    "profilePicture": "https://pbs.twimg.com/…",
    "followers": 92326677,
    "following": 115
  },
  "mentions": [
    {
      "id": "50115087",
      "userName": "NASAJohnson"
    }
  ],
  "isRetweet": true
}

💰 Pricing

You pay $0.0003 per tweet ($0.30 per 1,000 tweets) plus platform usage. Only saved posts are charged. Inputs that return nothing or fail never become dataset rows and cost nothing in event fees.

Worked example: 1,000 tweets ≈ $0.30 in event fees plus platform usage. On our test runs at the default 512 MB, usage was about $0.0017 for 100 tweets and about $0.0008 for the 20-tweet example input, so 1,000 tweets come to roughly $0.32 in total.

Set a maximum charge for the run if you want a hard cap. The run then stops cleanly once it is reached and keeps every post saved so far.

📥 Input

Use the same field names as the console form.

A /highlights profile URL reads that account’s highlights timeline. /media keeps posts that have a photo or video. /with_replies keeps replies written by that account. A list URL keeps posts from that list. /articles reads that account’s articles timeline. An account with no articles stores no posts and no failed item.

📤 Output

Each post is one dataset item. Keys appear only when the response sent a value.

A failed input does not create a dataset row. It is listed in the run's OUTPUT key-value record as failedInputs: [{input, reason}] and summarised in the status message. reason is a short status such as HTTP 404 list_unavailable, HTTP 200 no_timeline, user_unavailable, or no public posts matched the search. It does not include the response body.

Run locally

pip install -r requirements.txt
python -m src

On Scrapely the actor asks for a residential proxy session and 1024 MB.