X Tweet Scraper

TypeScript Apify Actor for searching public X posts through an HTTP-only data source, normalizing them to the assessment contract, applying authoritative local filters, deduplicating results, and enforcing server-authoritative FREE/PAID entitlement limits.

Status

The Actor is deployed and has been validated end-to-end on Apify.

Live validation includes:

• FREE tier enforcement
• PAID entitlement resolution
• latest search
• top search
• multi-page pagination
• combined filters
• hashtag filtering
• canonical output validation
• residential Apify Proxy
• 100-result paid performance benchmark

The project is locally tested and clean-clone buildable.

Requirements

• Node.js 22+
• npm
• TWEETAPI_API_KEY in private configuration or a local .env
• Authenticated Apify CLI for deployment

The API key is never part of Actor input, source control, schemas, persisted state, or logs.

Local development

npm ci
npm test
npm run typecheck
npm run build

The production entrypoint is:

dist/main.js

Run locally with:

$npm start

or with the Apify CLI:

$apify run

A clean-clone validation was performed successfully with dependency installation, tests, type checking, and production build.

Deployment

$apify push --force

The repository includes:

• Dockerfile
• .actor/actor.json
• .actor/INPUT_SCHEMA.json
• strict TypeScript configuration
• production build configuration

The Actor uses Apify SDK v3 and runs natively on the Apify platform.

Execution flow

1. Initialize the Apify Actor.
2. Read and validate input using Zod.
3. Resolve the runner identity from the Apify runtime.
4. Resolve FREE/PAID entitlement against an owner-controlled private store.
5. Fail closed to FREE when entitlement cannot be verified.
6. Calculate the effective collection limit.
7. Restore resumable state and validate the input fingerprint.
8. Configure Apify Proxy when requested.
9. Build the provider HTTP query.
10. Fetch pages using cursor-based pagination.
11. Normalize provider responses into strict CanonicalTweet objects.
12. Apply authoritative local filters.
13. Deduplicate results.
14. Validate each item again at the output boundary.
15. Enforce the entitlement limit immediately before dataset emission.
16. Push valid results to the default Apify dataset.
17. Persist the final run summary to KVS OUTPUT.

Project structure

• src/main.ts — Actor lifecycle.
• src/orchestrator.ts — production orchestration.
• src/input.ts — strict runtime input validation.
• src/types/ — canonical tweet contract and schemas.
• src/providers/tweetapi/ — HTTP provider integration, query construction, response validation, retry handling, and transport.
• src/normalize.ts — provider-to-canonical normalization boundary.
• src/filter.ts — authoritative local filtering.
• src/pipeline/paginator.ts — cursor pagination and deduplication.
• src/pipeline/result-sink.ts — final validation, entitlement enforcement, and dataset output.
• src/entitlement/resolver.ts — runtime identity and fail-closed entitlement.
• src/entitlement/apify-store.ts — private Apify KVS entitlement lookup.
• src/state.ts — resumable state and deterministic input fingerprinting.
• src/integration.ts — Apify Proxy, migration, and reconciliation adapters.
• scripts/benchmark.ts — deterministic local processing benchmark.

Input

Primary selectors include:

• searchTerms
• fromUsers
• toUsers
• mentioning
• hashtags

At least one useful search selector must be supplied.

Additional filters include:

• date range
• language
• minimum likes
• minimum retweets
• minimum replies
• verified-only filtering
• media type
• replies
• retweets
• ordering (latest / top)
• maxResults
• proxyConfiguration

Important semantics:

• Unspecified includeReplies means no reply constraint.
• includeReplies: false excludes replies.
• includeRetweets defaults to false.
• FREE users receive at most min(maxResults, 10).
• PAID users may receive up to their requested maxResults.
• Entitlement cannot be selected through Actor input.

Input is runtime-validated. Undocumented fields cannot be used to grant PAID access or bypass the output limit.

HTTP-only extraction

Extraction is browserless and HTTP-only.

The implementation does not use:

• Playwright
• Puppeteer
• Selenium
• browserless
• headless Chromium
• personal X cookies
• auth_token
• ct0
• a logged-in personal X account

The final implementation uses TweetAPI as the assessment-permitted equivalent HTTP data source.

Direct logged-out X SearchTimeline / guest-token global search was investigated during implementation but was not usable for the required global-search flow in the tested environment. Rather than introducing browser automation or a personal authenticated X session, the Actor uses an HTTP provider boundary.

Because the selected equivalent HTTP provider does not expose X guest tokens, X guest-token rotation is not applicable to this implementation. Equivalent transport resilience is implemented at the HTTP provider boundary through bounded retries, exponential backoff with jitter, Retry-After handling, timeouts, pagination, and configurable Apify residential proxy support.

The provider is isolated behind its own integration layer so the extraction source can be replaced without changing the canonical output, filtering, entitlement, or dataset boundaries.

Canonical output

Every dataset item is validated as a strict CanonicalTweet.

The output contains string IDs, ISO-8601 UTC timestamps, author information, engagement metrics, relationship flags, hashtags, mentions, expanded URLs, normalized media, source information, and scrapedAt.

Missing permitted values are represented as null rather than silently omitted.

Example:

{
  "id": "1770000000000000001",
  "url": "https://x.com/example/status/1770000000000000001",
  "text": "Example post",
  "lang": "en",
  "createdAt": "2024-03-20T09:12:00.000Z",
  "conversationId": null,
  "isReply": false,
  "isRetweet": false,
  "isQuote": false,
  "inReplyToId": null,
  "quotedTweetId": null,
  "author": {
    "id": "42",
    "username": "example",
    "name": "Example",
    "verified": false,
    "followers": 0,
    "following": 0
  },
  "metrics": {
    "likes": 0,
    "retweets": 0,
    "replies": 0,
    "quotes": 0,
    "bookmarks": null,
    "views": null
  },
  "entities": {
    "hashtags": [],
    "mentions": [],
    "urls": [],
    "media": []
  },
  "source": null,
  "scrapedAt": "2024-03-20T09:13:00.000Z"
}

Provider-specific response shapes are normalized before reaching the dataset. For example, TweetAPI Top responses may contain wrapped content and non-tweet modules; valid tweet content is unwrapped while non-tweet modules are ignored. IDs are never fabricated to satisfy the output schema.

Filtering

Provider-side query construction is used to reduce unnecessary traffic, but local filtering remains authoritative.

This means the Actor does not rely solely on the upstream provider to honor the requested constraints.

Live runs have validated combinations including:

• language
• minimum likes
• replies exclusion
• retweets exclusion
• hashtags
• latest
• top

Filter conditions are applied together using AND semantics where applicable.

Free-tier protection

Free-tier protection is enforced server-side and does not trust Actor input.

The runner identity is obtained from the Apify runtime using Actor.getEnv().userId.

That identity is resolved against an owner-controlled private Apify key-value store. The entitlement source is not part of the public repository and cannot be modified through Actor input.

Valid private entitlement records resolve to FREE or PAID. Unknown users, missing runtime identity, malformed records, lookup failures, and unavailable entitlement infrastructure all fail closed to FREE.

There is no:

• paid input field
• user-controlled tier input
• PAID=true environment override
• hardcoded public list of paid users
• client-side-only entitlement decision

Enforcement

The requested limit and effective limit are deliberately separate.

For FREE:

effectiveLimit = min(requestedMaxResults, 10)

For PAID:

effectiveLimit = requestedMaxResults

The effective limit constrains collection so a FREE run stops collecting once the required valid result count has been reached.

More importantly, the limit is enforced again in ResultSink immediately before Actor.pushData.

This provides defense in depth: changing pagination behavior or manipulating maxResults cannot by itself cause the output path to emit item 11 for a FREE run.

When the FREE cap is applied, the run summary reports:

{
  "limited": true,
  "reason": "free_tier",
  "cap": 10
}

The Actor also emits a clear sanitized log when FREE-tier limiting is applied.

Live entitlement validation

The entitlement path was tested on the deployed Actor.

A FREE run requesting substantially more than 10 results produced:

effectiveLimit: 10
pushed: 10
limited: true
reason: free_tier
cap: 10

A controlled PAID identity requesting 25 results produced:

effectiveLimit: 25
pushed: 25
limited: false
cap: null

Anti-fork reasoning

A fork can modify or remove checks in its own copy of the source code. No public-source-only mechanism can prevent somebody from changing code they own.

The security boundary for the canonical Actor therefore does not rely on the public repository.

PAID entitlement for the canonical deployment requires a successful lookup against an owner-controlled private entitlement source. A user can fork the repository and change their own Actor, but they cannot use Actor input, user-controlled environment variables, or modifications to the public source to grant PAID entitlement in the canonical deployment.

A production billing system could replace the private KVS lookup with a signed entitlement API or billing service without changing the enforcement boundary.

Resilience and rate limiting

The HTTP client implements:

• request timeouts
• bounded retry budgets
• exponential backoff
• jitter
• HTTP 429 classification
• HTTP 5xx retry handling
• network failure classification
• Retry-After support
• fatal authentication/request classification
• external cancellation handling
• response-shape validation

A single transient network failure, 429, or retryable server response therefore does not immediately hard-crash the run.

Pagination is cursor-based and results are deduplicated by tweet ID.

Apify Proxy

proxyConfiguration is supported as Actor input.

When enabled, the Actor creates an Apify ProxyConfiguration and routes HTTP requests through the configured proxy.

Residential proxy operation was validated on the deployed Actor with:

{
  "useApifyProxy": true,
  "apifyProxyGroups": ["RESIDENTIAL"]
}

The Node.js HTTP transport uses Undici's ProxyAgent through the supported dispatcher mechanism.

A stable Apify proxy session is used during the run to avoid unnecessary identity churn. Proxy URLs and credentials are never written to logs or persisted state.

Without proxyConfiguration, requests continue through the normal direct HTTP transport.

Resumable state

KVS STATE contains only safe execution state such as:

• cursor
• pushed count
• seen IDs
• input fingerprint
• query state
• statistics

API keys, proxy credentials, private entitlement data, private URLs, and raw provider responses are not persisted.

The input fingerprint prevents incompatible state from being resumed using a different query.

The state/reconciliation layer supports continuation without blindly restarting the same workload from zero.

Tests

The Vitest suite covers:

• input validation
• normalization
• filtering
• pagination
• deduplication
• HTTP response validation
• retry/backoff behavior
• entitlement resolution
• fail-closed entitlement cases
• FREE cap enforcement
• ResultSink
• persisted state
• integration adapters
• proxy transport
• proxy credential redaction

Run:

$npm test

Also verify:

npm run typecheck
npm run build

The project has been validated from a clean clone.

Performance benchmark

A live end-to-end paid Apify run was executed using:

• broad keyword query
• sortBy: latest
• maxResults: 100
• Apify residential proxy
• PAID entitlement
• no rate-limit failure
• no dropped-to-error run
• no duplicate outputs

Observed result:

Requested:       100
Effective limit: 100
Raw fetched:     120
Normalized:      107
Filtered:        7
Duplicates:      0
Pushed:          100
Pages fetched:   6
Run status:      SUCCEEDED
Apify duration:  22 s
Run cost:        ~$0.007

The deployed Actor therefore returned 100 valid dataset items in a successful 22-second Apify run.

Under the assessment performance rubric:

Grade A: < 30 seconds

this observed run falls within Grade A.

The reported 22 seconds is the full duration displayed by Apify. The assessment timer starts at the first outbound request and stops when the 100th schema-conforming item is pushed, excluding Actor cold-start/build time. Therefore the assessment-defined request-to-result interval for this run cannot be greater than the reported full-run duration.

The observed compute/run cost extrapolates to approximately $0.07 per 1,000 results if cost scaled linearly. This is an estimate only: actual cost varies with filtering selectivity, number of pages, proxy traffic, retries, execution time, and Apify pricing.

Live validation summary

The deployed Actor has been successfully exercised for:

• FREE cap enforcement
• PAID entitlement
• latest
• top
• multi-page pagination
• combined filters
• hashtag search/filtering
• canonical normalization
• residential proxy routing
• cross-run global deduplication
• overlapping-query global seen-set behavior
• optional finish webhook delivery
• 100-result end-to-end performance

The 100-result benchmark required six provider pages and completed without duplicates or rate-limit failure.

Global seen-set behavior was also validated live with overlapping runs. Globally seen items were excluded from the effective target, duplicate counters were reported, and pagination continued until new unique items satisfied the requested limit.

The optional finish webhook was validated live and delivered only the safe run summary metadata.

Empty-result and unavailable-account handling are covered by deterministic provider and pipeline tests. A final live empty-result validation could not be completed because the upstream TweetAPI request allowance was exhausted; the resulting HTTP 429 was correctly classified after bounded retries.

Compliance and production considerations

This project accesses public X data through an HTTP data provider and was built for the assessment.

Before deploying an equivalent system for a production client, I would review:

• X's current Terms of Service and developer/platform policies
• the upstream HTTP provider's terms and permitted uses
• applicable robots/access restrictions
• rate limits and acceptable request volume
• privacy and personal-data obligations
• data retention requirements
• jurisdiction-specific legal requirements
• downstream redistribution/storage restrictions

The Actor intentionally avoids personal logged-in X sessions and browser automation.

Public availability of a post should not by itself be treated as permission for unrestricted collection, retention, redistribution, or commercial reuse. Production deployment should be reviewed against the specific use case and current platform/provider terms.

Known limitations and trade-offs

The implementation intentionally uses TweetAPI as the assessment-permitted equivalent HTTP data source instead of depending on a personal X session or browser automation.

Direct guest-token SearchTimeline global search was investigated but was not used for the final extraction path. Consequently, X guest-token rotation is not implemented because the selected provider does not expose those tokens.

Instead, resilience is handled at the provider HTTP boundary with retries, backoff, jitter, Retry-After support, timeouts, cursor pagination, response classification, and configurable Apify residential proxy support.

Global cross-run deduplication is implemented through a separate private Apify KVS containing tweet IDs only. Because Apify KVS does not provide an atomic compare-and-set primitive for this design, simultaneous overlapping runs retain a small distributed race window; exactly-once delivery is therefore not claimed.

The optional finish webhook is implemented as best-effort delivery. Webhook failure does not invalidate an otherwise successful scraping run.

Empty result sets and known protected/private, suspended, deleted, or other unavailable-account provider entries are handled gracefully at the provider boundary. These cases are covered by deterministic tests. Exact future TweetAPI response representations cannot be exhaustively predicted and are not claimed as supported until verified.

The Actor also depends on the availability and request quota of the upstream HTTP provider. Provider-side rate limits or exhausted quotas are classified and handled by the retry policy, but cannot be bypassed by the Actor.

Delivery

Apify Actor

https://console.apify.com/actors/XaAxi9PSS9kTiOjcj

GitHub repository

https://gitlab.com/lucasacci/x-tweet-scraper#

Optional finish webhook

Set finishWebhookUrl to an absolute HTTP or HTTPS URL to receive the final safe run summary after a successful run:

{ "searchTerms": ["openai"], "maxResults": 10, "finishWebhookUrl": "https://example.com/hooks/scraper" }

This bonus webhook is best-effort. It retries network, 429, and 5xx failures with a bounded budget and never fails an otherwise successful Actor run. Only safe summary metadata is sent; tweets, credentials, proxy URLs, and entitlement data are excluded.

Empty and unavailable accounts

Empty search pages complete successfully with zero dataset items and a normal sourceExhausted summary. Known non-tweet or unavailable-account entries from TweetAPI are skipped safely; valid tweets on the same page continue through the pipeline. No tweet or ID is fabricated. Exact protected/suspended/deleted representations remain provider-specific and depend on the upstream HTTP response shape.

Optional global seen-set

The canonical Actor can maintain cross-run deduplication in the separate private Apify KVS x-tweet-scraper-global-seen. Records are bucketed as seen_<sha256-prefix> and contain only string tweet IDs, so overlapping queries share the same set independently of query fingerprints. An ID is marked only after a successful dataset push; failed writes do not poison the set. KVS failures degrade to the existing same-run deduplication behavior. Apify KVS does not provide an atomic compare-and-set here, so simultaneous overlapping runs retain a small race window and exactly-once delivery is not claimed. Bucket compaction/TTL is an operational retention task.