# X Tweet Scraper (`lucasacci/x-tweet-scraper`) Actor

Browserless X/Twitter tweet scraper built with Node.js and TypeScript. Extracts and filters tweets using HTTP requests only, with pagination, rate-limit handling, retries, deduplication, resumable state, proxy support, and strict free-tier result limits.

- **URL**: https://apify.com/lucasacci/x-tweet-scraper.md
- **Developed by:** [Luciano Casacci](https://apify.com/lucasacci) (community)
- **Categories:** Automation, Developer tools, Social media
- **Stats:** 1 total users, 1 monthly users, 94.4% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## X Tweet Scraper

TypeScript Apify Actor for searching public X posts through an HTTP-only data
source, normalizing them to the assessment contract, applying authoritative
local filters, deduplicating results, and enforcing server-authoritative
FREE/PAID entitlement limits.

### Status

The Actor is deployed and has been validated end-to-end on Apify.

Live validation includes:

- FREE tier enforcement
- PAID entitlement resolution
- `latest` search
- `top` search
- multi-page pagination
- combined filters
- hashtag filtering
- canonical output validation
- residential Apify Proxy
- 100-result paid performance benchmark

The project is locally tested and clean-clone buildable.

### Requirements

- Node.js 22+
- npm
- `TWEETAPI_API_KEY` in private configuration or a local `.env`
- Authenticated Apify CLI for deployment

The API key is never part of Actor input, source control, schemas, persisted
state, or logs.

### Local development

```bash
npm ci
npm test
npm run typecheck
npm run build
```

The production entrypoint is:

```text
dist/main.js
```

Run locally with:

```bash
npm start
```

or with the Apify CLI:

```bash
apify run
```

A clean-clone validation was performed successfully with dependency
installation, tests, type checking, and production build.

### Deployment

```bash
apify push --force
```

The repository includes:

- `Dockerfile`
- `.actor/actor.json`
- `.actor/INPUT_SCHEMA.json`
- strict TypeScript configuration
- production build configuration

The Actor uses Apify SDK v3 and runs natively on the Apify platform.

### Execution flow

1. Initialize the Apify Actor.
2. Read and validate input using Zod.
3. Resolve the runner identity from the Apify runtime.
4. Resolve FREE/PAID entitlement against an owner-controlled private store.
5. Fail closed to FREE when entitlement cannot be verified.
6. Calculate the effective collection limit.
7. Restore resumable state and validate the input fingerprint.
8. Configure Apify Proxy when requested.
9. Build the provider HTTP query.
10. Fetch pages using cursor-based pagination.
11. Normalize provider responses into strict `CanonicalTweet` objects.
12. Apply authoritative local filters.
13. Deduplicate results.
14. Validate each item again at the output boundary.
15. Enforce the entitlement limit immediately before dataset emission.
16. Push valid results to the default Apify dataset.
17. Persist the final run summary to KVS `OUTPUT`.

### Project structure

- `src/main.ts` — Actor lifecycle.
- `src/orchestrator.ts` — production orchestration.
- `src/input.ts` — strict runtime input validation.
- `src/types/` — canonical tweet contract and schemas.
- `src/providers/tweetapi/` — HTTP provider integration, query construction,
  response validation, retry handling, and transport.
- `src/normalize.ts` — provider-to-canonical normalization boundary.
- `src/filter.ts` — authoritative local filtering.
- `src/pipeline/paginator.ts` — cursor pagination and deduplication.
- `src/pipeline/result-sink.ts` — final validation, entitlement enforcement,
  and dataset output.
- `src/entitlement/resolver.ts` — runtime identity and fail-closed entitlement.
- `src/entitlement/apify-store.ts` — private Apify KVS entitlement lookup.
- `src/state.ts` — resumable state and deterministic input fingerprinting.
- `src/integration.ts` — Apify Proxy, migration, and reconciliation adapters.
- `scripts/benchmark.ts` — deterministic local processing benchmark.

### Input

Primary selectors include:

- `searchTerms`
- `fromUsers`
- `toUsers`
- `mentioning`
- `hashtags`

At least one useful search selector must be supplied.

Additional filters include:

- date range
- language
- minimum likes
- minimum retweets
- minimum replies
- verified-only filtering
- media type
- replies
- retweets
- ordering (`latest` / `top`)
- `maxResults`
- `proxyConfiguration`

Important semantics:

- Unspecified `includeReplies` means no reply constraint.
- `includeReplies: false` excludes replies.
- `includeRetweets` defaults to `false`.
- FREE users receive at most `min(maxResults, 10)`.
- PAID users may receive up to their requested `maxResults`.
- Entitlement cannot be selected through Actor input.

Input is runtime-validated. Undocumented fields cannot be used to grant PAID
access or bypass the output limit.

### HTTP-only extraction

Extraction is browserless and HTTP-only.

The implementation does **not** use:

- Playwright
- Puppeteer
- Selenium
- browserless
- headless Chromium
- personal X cookies
- `auth_token`
- `ct0`
- a logged-in personal X account

The final implementation uses TweetAPI as the assessment-permitted equivalent
HTTP data source.

Direct logged-out X SearchTimeline / guest-token global search was investigated
during implementation but was not usable for the required global-search flow
in the tested environment. Rather than introducing browser automation or a
personal authenticated X session, the Actor uses an HTTP provider boundary.

Because the selected equivalent HTTP provider does not expose X guest tokens,
X guest-token rotation is not applicable to this implementation. Equivalent
transport resilience is implemented at the HTTP provider boundary through
bounded retries, exponential backoff with jitter, `Retry-After` handling,
timeouts, pagination, and configurable Apify residential proxy support.

The provider is isolated behind its own integration layer so the extraction
source can be replaced without changing the canonical output, filtering,
entitlement, or dataset boundaries.

### Canonical output

Every dataset item is validated as a strict `CanonicalTweet`.

The output contains string IDs, ISO-8601 UTC timestamps, author information,
engagement metrics, relationship flags, hashtags, mentions, expanded URLs,
normalized media, source information, and `scrapedAt`.

Missing permitted values are represented as `null` rather than silently
omitted.

Example:

```json
{
  "id": "1770000000000000001",
  "url": "https://x.com/example/status/1770000000000000001",
  "text": "Example post",
  "lang": "en",
  "createdAt": "2024-03-20T09:12:00.000Z",
  "conversationId": null,
  "isReply": false,
  "isRetweet": false,
  "isQuote": false,
  "inReplyToId": null,
  "quotedTweetId": null,
  "author": {
    "id": "42",
    "username": "example",
    "name": "Example",
    "verified": false,
    "followers": 0,
    "following": 0
  },
  "metrics": {
    "likes": 0,
    "retweets": 0,
    "replies": 0,
    "quotes": 0,
    "bookmarks": null,
    "views": null
  },
  "entities": {
    "hashtags": [],
    "mentions": [],
    "urls": [],
    "media": []
  },
  "source": null,
  "scrapedAt": "2024-03-20T09:13:00.000Z"
}
```

Provider-specific response shapes are normalized before reaching the dataset.
For example, TweetAPI `Top` responses may contain wrapped content and non-tweet
modules; valid tweet content is unwrapped while non-tweet modules are ignored.
IDs are never fabricated to satisfy the output schema.

### Filtering

Provider-side query construction is used to reduce unnecessary traffic, but
local filtering remains authoritative.

This means the Actor does not rely solely on the upstream provider to honor the
requested constraints.

Live runs have validated combinations including:

- language
- minimum likes
- replies exclusion
- retweets exclusion
- hashtags
- `latest`
- `top`

Filter conditions are applied together using AND semantics where applicable.

### Free-tier protection

Free-tier protection is enforced server-side and does not trust Actor input.

The runner identity is obtained from the Apify runtime using
`Actor.getEnv().userId`.

That identity is resolved against an owner-controlled private Apify
key-value store. The entitlement source is not part of the public repository
and cannot be modified through Actor input.

Valid private entitlement records resolve to FREE or PAID. Unknown users,
missing runtime identity, malformed records, lookup failures, and unavailable
entitlement infrastructure all **fail closed to FREE**.

There is no:

- `paid` input field
- user-controlled `tier` input
- `PAID=true` environment override
- hardcoded public list of paid users
- client-side-only entitlement decision

#### Enforcement

The requested limit and effective limit are deliberately separate.

For FREE:

```text
effectiveLimit = min(requestedMaxResults, 10)
```

For PAID:

```text
effectiveLimit = requestedMaxResults
```

The effective limit constrains collection so a FREE run stops collecting once
the required valid result count has been reached.

More importantly, the limit is enforced again in `ResultSink` immediately
before `Actor.pushData`.

This provides defense in depth: changing pagination behavior or manipulating
`maxResults` cannot by itself cause the output path to emit item 11 for a FREE
run.

When the FREE cap is applied, the run summary reports:

```json
{
  "limited": true,
  "reason": "free_tier",
  "cap": 10
}
```

The Actor also emits a clear sanitized log when FREE-tier limiting is applied.

#### Live entitlement validation

The entitlement path was tested on the deployed Actor.

A FREE run requesting substantially more than 10 results produced:

```text
effectiveLimit: 10
pushed: 10
limited: true
reason: free_tier
cap: 10
```

A controlled PAID identity requesting 25 results produced:

```text
effectiveLimit: 25
pushed: 25
limited: false
cap: null
```

#### Anti-fork reasoning

A fork can modify or remove checks in its own copy of the source code. No
public-source-only mechanism can prevent somebody from changing code they own.

The security boundary for the canonical Actor therefore does not rely on the
public repository.

PAID entitlement for the canonical deployment requires a successful lookup
against an owner-controlled private entitlement source. A user can fork the
repository and change their own Actor, but they cannot use Actor input,
user-controlled environment variables, or modifications to the public source
to grant PAID entitlement in the canonical deployment.

A production billing system could replace the private KVS lookup with a signed
entitlement API or billing service without changing the enforcement boundary.

### Resilience and rate limiting

The HTTP client implements:

- request timeouts
- bounded retry budgets
- exponential backoff
- jitter
- HTTP 429 classification
- HTTP 5xx retry handling
- network failure classification
- `Retry-After` support
- fatal authentication/request classification
- external cancellation handling
- response-shape validation

A single transient network failure, 429, or retryable server response therefore
does not immediately hard-crash the run.

Pagination is cursor-based and results are deduplicated by tweet ID.

### Apify Proxy

`proxyConfiguration` is supported as Actor input.

When enabled, the Actor creates an Apify `ProxyConfiguration` and routes HTTP
requests through the configured proxy.

Residential proxy operation was validated on the deployed Actor with:

```json
{
  "useApifyProxy": true,
  "apifyProxyGroups": ["RESIDENTIAL"]
}
```

The Node.js HTTP transport uses Undici's `ProxyAgent` through the supported
`dispatcher` mechanism.

A stable Apify proxy session is used during the run to avoid unnecessary
identity churn. Proxy URLs and credentials are never written to logs or
persisted state.

Without `proxyConfiguration`, requests continue through the normal direct HTTP
transport.

### Resumable state

KVS `STATE` contains only safe execution state such as:

- cursor
- pushed count
- seen IDs
- input fingerprint
- query state
- statistics

API keys, proxy credentials, private entitlement data, private URLs, and raw
provider responses are not persisted.

The input fingerprint prevents incompatible state from being resumed using a
different query.

The state/reconciliation layer supports continuation without blindly restarting
the same workload from zero.

### Tests

The Vitest suite covers:

- input validation
- normalization
- filtering
- pagination
- deduplication
- HTTP response validation
- retry/backoff behavior
- entitlement resolution
- fail-closed entitlement cases
- FREE cap enforcement
- ResultSink
- persisted state
- integration adapters
- proxy transport
- proxy credential redaction

Run:

```bash
npm test
```

Also verify:

```bash
npm run typecheck
npm run build
```

The project has been validated from a clean clone.

### Performance benchmark

A live end-to-end paid Apify run was executed using:

- broad keyword query
- `sortBy: latest`
- `maxResults: 100`
- Apify residential proxy
- PAID entitlement
- no rate-limit failure
- no dropped-to-error run
- no duplicate outputs

Observed result:

```text
Requested:       100
Effective limit: 100
Raw fetched:     120
Normalized:      107
Filtered:        7
Duplicates:      0
Pushed:          100
Pages fetched:   6
Run status:      SUCCEEDED
Apify duration:  22 s
Run cost:        ~$0.007
```

The deployed Actor therefore returned 100 valid dataset items in a successful
22-second Apify run.

Under the assessment performance rubric:

```text
Grade A: < 30 seconds
```

this observed run falls within **Grade A**.

The reported 22 seconds is the full duration displayed by Apify. The assessment
timer starts at the first outbound request and stops when the 100th
schema-conforming item is pushed, excluding Actor cold-start/build time.
Therefore the assessment-defined request-to-result interval for this run cannot
be greater than the reported full-run duration.

The observed compute/run cost extrapolates to approximately **$0.07 per 1,000
results** if cost scaled linearly. This is an estimate only: actual cost varies
with filtering selectivity, number of pages, proxy traffic, retries, execution
time, and Apify pricing.

### Live validation summary

The deployed Actor has been successfully exercised for:

- FREE cap enforcement
- PAID entitlement
- `latest`
- `top`
- multi-page pagination
- combined filters
- hashtag search/filtering
- canonical normalization
- residential proxy routing
- cross-run global deduplication
- overlapping-query global seen-set behavior
- optional finish webhook delivery
- 100-result end-to-end performance

The 100-result benchmark required six provider pages and completed without
duplicates or rate-limit failure.

Global seen-set behavior was also validated live with overlapping runs. Globally
seen items were excluded from the effective target, duplicate counters were
reported, and pagination continued until new unique items satisfied the
requested limit.

The optional finish webhook was validated live and delivered only the safe run
summary metadata.

Empty-result and unavailable-account handling are covered by deterministic
provider and pipeline tests. A final live empty-result validation could not be
completed because the upstream TweetAPI request allowance was exhausted; the
resulting HTTP 429 was correctly classified after bounded retries.

### Compliance and production considerations

This project accesses public X data through an HTTP data provider and was built
for the assessment.

Before deploying an equivalent system for a production client, I would review:

- X's current Terms of Service and developer/platform policies
- the upstream HTTP provider's terms and permitted uses
- applicable robots/access restrictions
- rate limits and acceptable request volume
- privacy and personal-data obligations
- data retention requirements
- jurisdiction-specific legal requirements
- downstream redistribution/storage restrictions

The Actor intentionally avoids personal logged-in X sessions and browser
automation.

Public availability of a post should not by itself be treated as permission for
unrestricted collection, retention, redistribution, or commercial reuse.
Production deployment should be reviewed against the specific use case and
current platform/provider terms.

### Known limitations and trade-offs

The implementation intentionally uses TweetAPI as the assessment-permitted
equivalent HTTP data source instead of depending on a personal X session or
browser automation.

Direct guest-token SearchTimeline global search was investigated but was not
used for the final extraction path. Consequently, X guest-token rotation is not
implemented because the selected provider does not expose those tokens.

Instead, resilience is handled at the provider HTTP boundary with retries,
backoff, jitter, `Retry-After` support, timeouts, cursor pagination, response
classification, and configurable Apify residential proxy support.

Global cross-run deduplication is implemented through a separate private Apify
KVS containing tweet IDs only. Because Apify KVS does not provide an atomic
compare-and-set primitive for this design, simultaneous overlapping runs retain
a small distributed race window; exactly-once delivery is therefore not
claimed.

The optional finish webhook is implemented as best-effort delivery. Webhook
failure does not invalidate an otherwise successful scraping run.

Empty result sets and known protected/private, suspended, deleted, or other
unavailable-account provider entries are handled gracefully at the provider
boundary. These cases are covered by deterministic tests. Exact future
TweetAPI response representations cannot be exhaustively predicted and are not
claimed as supported until verified.

The Actor also depends on the availability and request quota of the upstream
HTTP provider. Provider-side rate limits or exhausted quotas are classified and
handled by the retry policy, but cannot be bypassed by the Actor.

### Delivery

#### Apify Actor

`https://console.apify.com/actors/XaAxi9PSS9kTiOjcj`

#### GitHub repository

`https://gitlab.com/lucasacci/x-tweet-scraper#`

### Optional finish webhook

Set `finishWebhookUrl` to an absolute HTTP or HTTPS URL to receive the final
safe run summary after a successful run:

```json
{ "searchTerms": ["openai"], "maxResults": 10, "finishWebhookUrl": "https://example.com/hooks/scraper" }
```

This bonus webhook is best-effort. It retries network, 429, and 5xx failures
with a bounded budget and never fails an otherwise successful Actor run. Only
safe summary metadata is sent; tweets, credentials, proxy URLs, and entitlement
data are excluded.

### Empty and unavailable accounts

Empty search pages complete successfully with zero dataset items and a normal
`sourceExhausted` summary. Known non-tweet or unavailable-account entries from
TweetAPI are skipped safely; valid tweets on the same page continue through the
pipeline. No tweet or ID is fabricated. Exact protected/suspended/deleted
representations remain provider-specific and depend on the upstream HTTP
response shape.

### Optional global seen-set

The canonical Actor can maintain cross-run deduplication in the separate
private Apify KVS `x-tweet-scraper-global-seen`. Records are bucketed as
`seen_<sha256-prefix>` and contain only string tweet IDs, so overlapping queries
share the same set independently of query fingerprints. An ID is marked only
after a successful dataset push; failed writes do not poison the set. KVS
failures degrade to the existing same-run deduplication behavior. Apify KVS
does not provide an atomic compare-and-set here, so simultaneous overlapping
runs retain a small race window and exactly-once delivery is not claimed.
Bucket compaction/TTL is an operational retention task.

# Actor input Schema

## `searchTerms` (type: `array`):

Search terms or phrases to match. At least one of searchTerms, fromUsers, or hashtags must be provided.

## `fromUsers` (type: `array`):

Usernames to include as tweet authors. Do not include the @ prefix.

## `toUsers` (type: `array`):

Usernames whose replies should be included. Do not include the @ prefix.

## `mentioning` (type: `array`):

Usernames mentioned by tweets. Do not include the @ prefix.

## `hashtags` (type: `array`):

Hashtags that tweets must contain. Do not include the # prefix.

## `since` (type: `string`):

Inclusive UTC start date in YYYY-MM-DD format.

## `until` (type: `string`):

Inclusive UTC end date in YYYY-MM-DD format.

## `language` (type: `string`):

ISO-639-1 language code to match.

## `minLikes` (type: `integer`):

Minimum inclusive likes threshold.

## `minRetweets` (type: `integer`):

Minimum inclusive retweets threshold.

## `minReplies` (type: `integer`):

Minimum inclusive replies threshold.

## `onlyVerified` (type: `boolean`):

When true, include only tweets from verified authors.

## `mediaType` (type: `string`):

Restrict results to any, text-only, images, video, or links.

## `includeReplies` (type: `boolean`):

Reply filter: unset means no constraint, false excludes replies, true allows replies.

## `includeRetweets` (type: `boolean`):

Whether retweets are allowed. Defaults to false.

## `sortBy` (type: `string`):

Search ordering mode.

## `maxResults` (type: `integer`):

Maximum number of results requested. Free-tier users are limited separately by server-side entitlement enforcement.

## `proxyConfiguration` (type: `object`):

Optional Apify Proxy configuration for HTTP requests.

## `finishWebhookUrl` (type: `string`):

Optional HTTP or HTTPS URL that receives the final safe run summary after a successful run.

## Actor input object example

```json
{
  "searchTerms": [
    "openai"
  ],
  "fromUsers": [],
  "toUsers": [],
  "mentioning": [],
  "hashtags": [],
  "mediaType": "any",
  "includeRetweets": false,
  "sortBy": "latest",
  "maxResults": 10
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("lucasacci/x-tweet-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("lucasacci/x-tweet-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call lucasacci/x-tweet-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,lucasacci/x-tweet-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/XaAxi9PSS9kTiOjcj/builds/tKqHbmUtLpIPw14jf/openapi.json