Threads Scraper — Keyword Search, Posts & Profiles (No Login)

Scrape public Meta Threads data at scale: search posts by keyword or hashtag, pull public profiles, and fetch a user's posts — returned as clean, LLM-ready JSON. No login, public data only.

Why this exists: the official Threads API has no keyword search and no streaming — you cannot monitor a topic, brand, or hashtag through it. This scraper closes that gap. Keyword/hashtag search is the flagship capability.

---

Features

• 🔎 Keyword & hashtag search — the thing the official API can't do.
• 👤 Public profiles — bio, links, follower count, verification, recent posts.
• 🧵 User posts — a specific public account's posts, with scroll pagination.
• 🤖 LLM-ready output — flat, predictable JSON that an AI agent can consume directly. Also exportable as CSV / Excel / JSON via the Apify dataset.
• 🛡️ Reliable by design — Camoufox anti-fingerprint browser, residential proxy, retries with exponential backoff, and a GraphQL-response capture strategy (not brittle DOM scraping).
• 🔒 Privacy-first — logged-out public data only, nothing persisted beyond the request.

Modes

mode	What you get	Required input
search (default)	Public posts matching a keyword / #hashtag	query
profile	One public profile + its recent posts	username
posts	A specific user's public posts	username

Quick start

Runs out of the box — no configuration needed. The default run searches Threads for openai and returns 25 posts.

Input

{
  "mode": "search",
  "query": "openai",
  "maxResults": 25,
  "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"] }
}

Profile example:

{ "mode": "profile", "username": "natgeo", "maxResults": 10 }

Output (real example, mode: search)

{
  "id": "3939159366922654199",
  "code": "DaqtJLbE9H3",
  "url": "https://www.threads.com/@technewsonweb/post/DaqtJLbE9H3",
  "text": "OpenAI Bio Bug Bounty.\nTesting for universal jailbreaks for biorisks.",
  "published_at": "2026-07-11T21:13:31+00:00",
  "like_count": 0,
  "reply_count": 0,
  "repost_count": 0,
  "quote_count": 0,
  "author": {
    "username": "technewsonweb",
    "user_pk": "63104067007",
    "is_verified": false,
    "profile_pic_url": "https://.../profile.jpg"
  },
  "media": [],
  "has_audio": false,
  "is_reply": false,
  "scraped_at": "2026-07-12T00:00:00+00:00"
}

Profile output (mode: profile) adds full_name, biography, bio_links, follower_count, is_verified, is_private, and a posts[] array of the above shape.

Pricing (pay-per-event)

Single, simple per-result price — you only pay for items actually returned:

Event	Price
Per result returned (any mode)	$0.002

≈ $2 per 1,000 results. In search / posts each post is one result; in profile the profile (with its bundled recent posts) is one result. Charged automatically via Apify's apify-default-dataset-item event, and the SDK enforces your max-charge budget.

Legal & privacy

• Scrapes only logged-out, publicly visible data — no login, no authentication, no private endpoints replayed.
• Aligned with the Meta v. Bright Data (2024) precedent on public data.
• No personal data is stored or logged beyond the lifetime of a single request (GDPR hygiene). You are responsible for how you use the output.

---

Repository architecture

One codebase, three delivery channels:

core/       Pure scraping logic (search/profile/posts) → structured objects.
            Knows nothing about Apify or HTTP. Fully unit-tested.
src/        Apify Actor entrypoint — thin wrapper over core/ (input, billing).
.actor/     Apify config: actor.json, input_schema.json, dataset_schema.json,
            Dockerfile. (Pricing is set in the Apify Console — see DEPLOY.md.)
server/     FastAPI HTTP API + MCP server (thin adapter over the Apify Actor,
            deployed to Render — no browser, free-tier friendly).
tests/      Hermetic extractor tests + a manual live smoke check.

Why Camoufox over stock Playwright? Threads fingerprints the browser even when logged out. Camoufox is a hardened Firefox with anti-fingerprinting built in and a Playwright-compatible API — fewer blocks, no brittle stealth patches. The code falls back to stock Playwright Firefox when Camoufox isn't installed, so it stays importable and testable anywhere.

How extraction works. Threads is a React/Relay app; logged-out profile/post data is not in the initial HTML — it arrives via GraphQL/XHR after hydration. The browser layer captures those JSON responses and the extractor walks them for thread_items / user objects. This is far more robust than DOM scraping.

Local development

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt          # core + actor
playwright install firefox               # or: python -m camoufox fetch

# Run unit tests (no network):
pip install pytest pytest-asyncio && pytest -q

# Live smoke test (hits real Threads):
python -m tests.live_check

Use core directly:

import asyncio
from core import search, profile

posts = asyncio.run(search("#ai", limit=20))
natgeo = asyncio.run(profile("natgeo", post_limit=10))

HTTP API + MCP server (thin adapter)

The server is a thin adapter over the Apify Actor — it doesn't run a browser, so it fits a Render free instance. Set APIFY_TOKEN and it forwards to the Actor.

pip install -r server/requirements.txt
export APIFY_TOKEN=apify_api_...        # required
uvicorn server.app:app --reload

• POST /search { "query": "openai", "limit": 25 }
• POST /profile { "username": "natgeo", "limit": 10 }
• POST /posts { "username": "natgeo", "limit": 25 }
• GET /health — liveness / keepalive
• POST /mcp — MCP server exposing threads_search, threads_profile, threads_posts for AI agents (see MCP.md).

Optional x402 payment gating is available behind X402_ENABLED (off by default).

Deployment

• Apify Actor — built from this Git repo (see DEPLOY.md).
• Render — HTTP + MCP server via render.yaml Blueprint, with a keepalive cron for the free tier.

License

MIT