X Feed Monitor

Monitor X/Twitter for keywords, hashtags, or @mentions with cursor-based incremental sync and webhook notifications. Posts are retrieved through Grok's x_search tool (xAI). Optional AI sentiment/topic analysis. Cost-controlled and schedule-ready with a stable, versioned output schema.

Requires an xAI (Grok) API key. Without one the actor runs in demo mode (no data). It is not a direct X API/scraper client — results are whatever the Grok x_search tool returns.

Features

• Search X/Twitter by keyword, hashtag, or @mention (via Grok x_search)
• Engagement metrics: likes, reposts, replies (the views field is reserved and not currently populated)
• Incremental sync with cursor persistence (only new posts each run)
• Webhook notification on completion (Slack/Discord/Zapier/any HTTP endpoint)
• Cost control: hard-cap projected USD spend (maxCostUsd)
• Dry-run mode (estimateOnly): preview projected cost without spending
• Optional sentiment and topic analysis
• Dataset output plus optional JSON/CSV export to the key-value store

Quick start

1. Get an xAI (Grok) API key at console.x.ai. (Pricing and any free credits are set by xAI and may change.)
2. Run the actor with this input:

{
  "queries": ["@anthropic", "#AI"],
  "estimateOnly": true,
  "grokApiKey": "xai-..."
}

3. estimateOnly: true previews the cost without spending API credits. Remove it and rerun to collect posts.

Live monitoring workflow

The common use case is a self-updating feed:

1. Run once with persistState: true (the default). The actor saves a cursor to the key-value store.
2. Schedule the actor in Apify Console → Schedules at your chosen cadence (every 15 min, hourly, daily).
3. Set a webhook URL to receive a POST on every completion.

Each scheduled run fetches only new posts since the last run and pings the webhook.

Input

Queries and scope

• queries (array, required): strings or objects {q, type?, label?, limit?}.
• maxResults (integer, 1-100, default 50): cap on posts requested per query.
• maxPages (integer, 1-10, default 1): currently used only in the cost estimate. Collection makes a single x_search call per query; multi-page pagination is not yet implemented.
• sinceId (string): only fetch posts newer than this ID (passed to Grok as a hint).
• sinceMinutes (integer): only fetch posts from the last N minutes (passed to Grok as a prompt hint, not a hard server-side filter).
• includeReplies (boolean, default true): include replies.
• includeReposts (boolean, default true): include reposts.
• includeQuotes (boolean, default true): include quote posts.

Analysis

• mode (enum, default collect): collect, analyze, or both.
• analysisLevel (enum, default none): none, light, or full.
• includeMetrics (boolean, default true): include likes/retweets/replies/views.

Scheduling and state

• persistState (boolean, default true): save cursor between runs.
• stateKey (string): KV store namespace for cursor (marked secret to satisfy Apify's name-pattern scanner).
• dedupe (boolean, default true): remove duplicates within a run.

Output and cost controls

• outputFormat (enum, default dataset): dataset, json, or csv. The dataset is always written; json/csv add an export to the key-value store.
• estimateOnly (boolean, default false): output a projected-cost estimate without fetching. Note: the input form pre-fills this to true, so a first console run is a dry run until you clear it.
• maxCostUsd (number): abort before fetching if the projected cost exceeds this amount.
• maxApiCalls (integer): accepted for forward-compatibility but not currently enforced; use maxCostUsd to cap spend.
• webhook (string, secret): URL to POST a summary on completion.
• grokApiKey (string, secret): xAI (Grok) API key. Can also be supplied via the GROK_API_KEY environment variable.

Output

Each post is pushed to the default dataset in this canonical (nested) shape:

{
  "schema_version": "1.0.0",
  "platform": "x",
  "post_id": "1234567890",
  "url": "https://x.com/anthropic/status/1234567890",
  "created_at": "2026-04-15T10:30:00Z",
  "collected_at": "2026-04-15T10:35:12Z",
  "author_handle": "anthropic",
  "author_id": null,
  "post_type": "post",
  "text": "Claude 4 is now available.",
  "lang": null,
  "metrics": {
    "likes": 42,
    "reposts": 10,
    "replies": 5,
    "views": null
  },
  "analysis": {
    "level": "light",
    "model": "grok-2-latest",
    "prompt_version": "1.0.0",
    "sentiment": "positive",
    "sentiment_score": 0.8,
    "topics": ["product-launch"],
    "summary": null,
    "risk_flags": []
  },
  "query_context": {
    "query": "@anthropic",
    "query_hash": "…",
    "label": null,
    "tool": "x_search",
    "run_id": "…",
    "cursor": null
  },
  "source": { "provider": "grok", "endpoint": "x_search", "api_version": "1.0" }
}

Notes on fields:

• metrics is null when includeMetrics is false. metrics.reposts (not retweets) holds the repost count; metrics.views is reserved and is currently always null.
• analysis is null unless analysisLevel is light or full and mode is analyze or both. summary is populated only at full level.
• author_id and lang are not provided by x_search and are currently null.

A run summary (counts, per-query results, errors, estimated cost, dataset URL) is written separately to the OUTPUT key in the key-value store. The dataset may also contain occasional auxiliary records — a demo record when no API key is set, an estimate record in estimateOnly mode, an error record on invalid input or cost abort, or a status record when all queries returned no results.

Webhook payload

If webhook is set, the actor POSTs this JSON after each run:

{
  "run_id": "abc123",
  "status": "success",
  "total_posts": 47,
  "dataset_url": "https://api.apify.com/v2/datasets/.../items",
  "errors_count": 0
}

status is one of success, partial, or failed. Webhook delivery failures are logged and do not fail the run.

Examples

Brand monitoring

{
  "queries": ["@yourbrand", "\"your product\""],
  "mode": "both",
  "analysisLevel": "light",
  "persistState": true,
  "webhook": "https://hooks.slack.com/services/T00/B00/XXX"
}

Competitor tracking (daily digest)

{
  "queries": ["@competitor1", "@competitor2"],
  "maxResults": 100,
  "persistState": true,
  "sinceMinutes": 1440
}

Trend analysis (full analysis, cost-capped)

{
  "queries": ["#trending", "#breaking"],
  "mode": "analyze",
  "analysisLevel": "full",
  "maxResults": 100,
  "maxCostUsd": 1.00
}

Cost estimation

The actor computes a projected cost from a built-in token model (an upper-bound estimate assuming a conservative per-post token count plus a 20% safety margin), not from live billing. Collection scales roughly with the number of queries × maxResults; enabling analysis adds a per-post analysis cost on top. Full analysis costs more than light, which costs more than collect-only.

Actual spend depends on current xAI/Grok pricing and the real token usage of each request, so treat the estimate as a ceiling rather than an exact figure. Run with estimateOnly: true to see the projection for your specific input before spending, and set maxCostUsd to abort automatically if the projection exceeds your budget.

Limitations

• Requires an xAI (Grok) API key. With no key (input grokApiKey unset and no GROK_API_KEY env var), the actor does not fetch data — it pushes a single demo record explaining how to add a key and exits successfully.
• X/Twitter access is indirect and best-effort. Posts are retrieved via Grok's x_search tool, not the official X API. Coverage, ordering, freshness, and completeness are whatever the tool returns and can vary run to run. This is not a guaranteed exhaustive scrape.
• Rate limits and availability. The underlying X/Twitter and xAI endpoints may be rate-limited (HTTP 429), throttled, temporarily unavailable, or return no results. The actor retries transient errors with backoff, then degrades gracefully: it records the error, and if every query comes back empty it logs a warning, pushes a status: no_results record, and still exits SUCCESS (it does not hard-fail the run).
• maxPages does not paginate. Exactly one x_search call is made per query; maxPages currently only feeds the cost projection.
• maxApiCalls is not enforced. It is accepted but has no effect; use maxCostUsd to cap spend.
• sinceMinutes / sinceId are hints. They are passed into the search prompt, not applied as strict server-side filters, so some out-of-window posts may occasionally appear.
• Not all fields are populated. metrics.views, author_id, and lang are not available from x_search and are always null. post_type is inferred heuristically (e.g. RT @ → repost) and may be imperfect.
• Analysis quality. Sentiment/topics/summaries are produced by an LLM and are approximate; analysis failures are isolated per post (the post is still returned with analysis: null).
• Cost figures are estimates, not live billing — see Cost estimation.

Privacy and security

• grokApiKey, webhook, and stateKey are marked isSecret: true. Apify encrypts them at rest and redacts from run logs.
• Fetched posts are public X/Twitter content; no private data is collected.

Troubleshooting

• 401 Unauthorized: invalid or missing Grok key. Regenerate at console.x.ai and paste into grokApiKey.
• 429 Too Many Requests: reduce maxResults, maxPages, or query count.
• Webhook never fires: verify the URL with curl -X POST; confirm endpoint is active.
• Zero new posts after first run: cursor working as intended. Set persistState: false to re-fetch.

Changelog

• v1.4: Added .actor/dataset_schema.json (overview view) and store categories; corrected README to match the actual nested output schema, the real webhook payload, and true maxPages/maxApiCalls/views behavior; added a Limitations section; hardened the no-results path to exit SUCCESS with a status record instead of only logging an error.
• v1.3 (2026-04-18): stateKey marked isSecret to clear Apify name-pattern warning; README simplified.
• v1.2 (2026-04-18): webhook marked isSecret (signed URLs may contain auth tokens); expanded README.
• v1.0: initial release with collect, analyze, incremental sync, cost controls.

Resources

• xAI API Console - create a Grok/xAI API key
• Apify Schedules - automate runs
• Apify Webhooks - advanced routing