Twitter Post Scraper | No Cookie | No Login

---

✨ Why this scraper?

⚡ Fast & cheap	Tier 1 uses direct HTTP with TLS-impersonation — high volume, minimal proxy bytes.
🛡️ Crash-proof	Three runtime guards always terminate cleanly with a specific terminationReason — it never falls over.
🔁 Self-healing	Automatically re-harvests queryId/features & regenerates x-client-transaction-id when X changes its API.
♻️ Duplicate-free resume	State & cursors are saved to the Key-Value Store; resumed runs never write the same tweet twice.
💰 Cost-aware	A Cost Circuit Breaker stops the run before proxy/dataset charges spiral out of control.
🔒 Safe & compliant	Read-only, public data only, and never logs tokens or secrets.

---

🚀 Quick Start

1. Provide at least one of handles or startUrls.
2. Use a RESIDENTIAL proxy (required — datacenter IPs are permanently banned by X.com).
3. Run the Actor. Scraped tweets land in the Apify Dataset.

{
    "handles": ["nasa", "spacex"],
    "maxTweetsPerUser": 100,
    "includeReplies": false,
    "includeRetweets": true,
    "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"] }
}

💡 Use cases: brand sentiment analysis, competitor research, keyword monitoring, academic research datasets, or archiving public account timelines.

---

🧭 How it works

INPUT ─▶ Fail-fast validation ─▶ Tier Router ─┬─▶ Tier 1: HTTP-direct (TLS-impersonation)   ◀── primary
        (handles/startUrls)                    └─▶ Tier 2: Playwright + CDP interception      ◀── fallback
                                                       │
                                    Dedup + Cursor + Enrichment (optional)
                                                       │
                                    Apify Dataset  +  Key-Value Store (STATE)

1. Fail-fast input validation — at least one of startUrls/handles is required, checked before any proxy or resource is used.
2. Tier router — auto (Hybrid), or force http / browser.
3. Per-target scrape — page-by-page with cursors, dedup, and state checkpoints.
4. Three runtime guards ensure the Actor always terminates cleanly (see Resilience).

🔀 Two transport tiers

	Tier 1 — HTTP-direct	Tier 2 — Browser-CDP
Engine	got-scraping + TLS-impersonation	Playwright + CDP request interception
Role	Primary (default)	Fallback / protected operations
Proxy cost	Low	10–50× higher
Used when	Normal volume	Self-heal fails / JS execution needed

Locked scope — public data only. No DMs, protected accounts, follower lists, or full Search. Read-only (no write actions). The authenticated path is an opt-in input only (authToken/csrfToken, secret) and is disabled by default — using it is at the user's own ToS risk.

---

⚙️ Input

Provide at least one of startUrls or handles.

Field	Type	Default	Description
startUrls	array	–	X.com profile or single-tweet URLs.
handles	array	–	X.com usernames without the leading @.
mode	auto|http|browser	auto	Force a transport tier or let the router decide.
maxTweetsPerUser	integer (≥1)	100	Max tweets per target. Ignored when noPerUserLimit is on.
noPerUserLimit	boolean	false	Collect all tweets per target (still bounded by maxRequestsPerCrawl).
maxRequestsPerCrawl	integer (≥1)	5000	Hard guard against runaway cost & memory.
proxyByteBudgetMb	integer (≥1)	–	Cost breaker: stop the run cleanly once proxy traffic exceeds the budget.
includeReplies	boolean	false	Include reply tweets.
includeRetweets	boolean	true	Include retweets.
dateFrom / dateTo	string	–	Date filter (ISO 8601 / YYYY-MM-DD), derived from the Snowflake ID.
enableEnrichment	boolean	false	Enable optional AI enrichment (non-blocking).
enrichment	object	–	Enrichment options, e.g. { "sentiment": true, "translateTo": "id" }.
proxyConfiguration	object	RESIDENTIAL	Apify proxy configuration (residential required).
authToken / csrfToken	string (secret)	–	Optional authenticated mode. Disabled by default.

No sentinel values. "Unlimited" is the boolean noPerUserLimit, not maxTweetsPerUser: 0 (the minimum stays 1).

---

📦 Output (Apify Dataset)

Each Dataset item is a single flat tweet record (<9 MB, media stored as URLs only).

{
    "tweetId": "1234567890123456789",
    "url": "https://x.com/exampleuser/status/1234567890123456789",
    "text": "This is an example tweet body.",
    "createdAt": "2025-01-15T12:00:00.000Z",
    "lang": "en",
    "author": {
        "id": "44196397",
        "handle": "exampleuser",
        "name": "Example User",
        "verified": true,
        "followersCount": 15000
    },
    "metrics": {
        "replyCount": 12,
        "retweetCount": 34,
        "likeCount": 560,
        "quoteCount": 7,
        "bookmarkCount": 21,
        "viewCount": 98000
    },
    "media": ["https://pbs.twimg.com/media/EXAMPLE1.jpg"],
    "isReply": false,
    "isRetweet": false,
    "isQuote": false,
    "conversationId": "1234567890123456789",
    "enrichment": null,
    "scrapedAt": "2025-01-15T12:05:00.000Z",
    "sourceTier": "http"
}

• createdAt is derived from the Snowflake ID: (id >> 22) + 1288834974657.
• sourceTier marks the tier (http/browser) that produced the record.
• enrichment is null when enrichment is disabled/skipped.

🗃️ Key-Value Store (State & Cache)

The Actor persists operational state for duplicate-free resume after persistState/migrating events.

Key	Contents	Purpose
STATE	Per-target cursors, dedup index, and request budget.	Persisted for safe resume; holds terminationReason when the run stops.
CACHE_QUERYIDS	GraphQL queryId + features toggles (TTL'd).	Reused until expiry to reduce re-harvesting.
TOKEN_POOL	Guest-token ↔ proxy-IP affinity (ephemeral, rotated).	No user credentials.

---

🛡️ Resilience

Three independent guards; all terminate cleanly (state flushed, resume-able, with a specific terminationReason):

1. 💰 Cost Circuit Breaker (cost-governor.ts) — input proxyByteBudgetMb + env ACTOR_MAX_TOTAL_CHARGE_USD / ACTOR_MAX_PAID_DATASET_ITEMS. SOFT 80% = stop new targets, force Tier 2→1; HARD 100% = clean exit cost_budget_exceeded. Has absolute authority.
2. 🔌 Error-Rate Circuit Breaker (circuit-breaker.ts) — CLOSED→OPEN→HALF_OPEN, sliding window of 20 per tier, counting auth-class errors only (401/403/tid_rejected/ queryId_unknown; 429 goes to proxy quarantine). OPEN triggers self-heal then Tier 2 failover; persistent failure → tid_algorithm_drift.
3. 📉 Degradation State Machine (orchestrator.ts) — HEALTHY→DEGRADED→ABORTED. Global guest-access revocation → DEGRADED → clean exit guest_access_revoked (or stay alive if the user supplied an authToken escape-hatch).

terminationReason	Meaning
completed	Finished normally.
cost_budget_exceeded	Cost breaker hit HARD 100%.
tid_algorithm_drift	Error-rate breaker OPEN & Tier 2 failover exhausted.
guest_access_revoked	Global guest access revoked, no auth escape-hatch.
max_requests_reached	maxRequestsPerCrawl limit reached.

---

❓ FAQ & Troubleshooting

---

📈 Pricing & Support

This Actor is a commercial product available on the Apify Store. Pricing follows your Apify plan and the proxy/compute resources consumed per run. For questions, feature requests, or custom scraping needs, contact us through the Actor's Apify Store page.

Compliance: Use for public data only and comply with X.com's Terms of Service and applicable law. This Actor is read-only and performs no write actions.