🕷️ Website Crawler — Full-Site Scraping for AI avatar

🕷️ Website Crawler — Full-Site Scraping for AI

Pricing

from $20.00 / 1,000 results

Go to Apify Store
🕷️ Website Crawler — Full-Site Scraping for AI

🕷️ Website Crawler — Full-Site Scraping for AI

Crawl entire websites for clean text, markdown or HTML. Perfect for RAG pipelines, AI training & content analysis. Handles JS-rendered pages. Alternative to Firecrawl & Jina. Pay per page.

Pricing

from $20.00 / 1,000 results

Rating

0.0

(0)

Developer

NexGenData

NexGenData

Maintained by Community

Actor stats

1

Bookmarked

23

Total users

4

Monthly active users

9 days ago

Last modified

Categories

Share

Website Crawler

Full-Site Scraping for AI

Public web content, for research and RAG.

📊 What you get

Clean JSON output (7 fields):

  • url — Url
  • title — Title
  • text — Text
  • markdown — Markdown
  • html — Html
  • metadata — Metadata
  • crawledAt — Crawled At

Pricing: $0.020 per page (Pay-Per-Event) — about 50 pages per $1.

🤖 Use with AI agents

Point Claude, the OpenAI Agents SDK, an n8n flow or any MCP-aware client at it.

Sample agent prompt:

Run Website Crawler on my input and return the structured results.

Agentic payments (x402): Supports agentic payment via x402 — call this actor with USDC, no API key required.

RAG Web Browser


A NexGenData utility actor.