AI-Ready Content Extractor — Structured Web Data for LLM & MCP

Extract fully structured JSON content from any URL — optimized for LLM consumption, RAG pipelines, and MCP (Model Context Protocol) tool integration. One Actor call returns everything you need: title, sections, headings, contact info, internal/external links, JSON-LD structured data, and clean plain text.

Works seamlessly with Claude, GPT-4, LangChain, LlamaIndex, and any AI framework.

Why Use This vs. a Regular Scraper?

Feature	Regular Scraper	This Actor
Output format	Raw HTML	Clean JSON
LLM-ready	❌ needs post-processing	✅ ready to paste into prompt
Contact extraction	❌ manual regex	✅ built-in
Structured data	❌ ignored	✅ JSON-LD parsed
Section splitting	❌ flat blob	✅ heading hierarchy
MCP compatible	❌	✅

Use Cases

• AI agents: Give your Claude/GPT agent the ability to read and understand any webpage
• RAG pipelines: Feed structured sections directly into vector databases
• Lead enrichment: Extract emails, phones, and addresses from company websites
• Content auditing: Audit page structure, headings, and link patterns at scale
• Competitive research: Extract structured data from competitor pages
• MCP tool integration: Use as a tool in your MCP server to give AI access to web content

Input

{
  "urls": [
    "https://example.com",
    "https://another-site.com/about"
  ],
  "extractSections": true,
  "extractContactInfo": true,
  "extractLinks": true,
  "extractStructuredData": true,
  "includeRawText": true,
  "maxContentLength": 10000
}

Field	Type	Default	Description
urls	Array	(required)	List of URLs to extract
extractSections	Boolean	true	Split content into heading-based sections
extractContactInfo	Boolean	true	Find emails, phones in page text
extractLinks	Boolean	true	Collect internal + external links
extractStructuredData	Boolean	true	Parse JSON-LD (schema.org) if present
includeRawText	Boolean	true	Include full plain text for LLM prompt injection
maxContentLength	Integer	0	Truncate textForLlm at N chars (0 = unlimited)

Output

Each dataset record:

{
  "url": "https://example.com",
  "canonical": "https://example.com/",
  "title": "Example Domain",
  "lang": "en",
  "meta": {
    "description": "This domain is for illustrative examples.",
    "keywords": null
  },
  "wordCount": 312,
  "hasEmail": true,
  "hasPhone": false,
  "contactInfo": {
    "emails": ["contact@example.com"],
    "phones": []
  },
  "sections": [
    {
      "heading": "About Us",
      "level": 2,
      "content": ["We are a company that...", "Founded in 2020..."]
    }
  ],
  "links": {
    "internal": [
      { "url": "https://example.com/about", "text": "About" }
    ],
    "external": [
      { "url": "https://twitter.com/example", "text": "Follow us" }
    ]
  },
  "structuredData": [
    { "@context": "https://schema.org", "@type": "Organization", "name": "Example" }
  ],
  "textForLlm": "Example Domain This domain is for illustrative examples...",
  "extractedAt": "2026-07-04T10:00:00.000Z"
}

MCP Integration

This Actor is designed to be called from MCP tool definitions. Example MCP tool wrapper:

# In your MCP server
@server.tool()
async def read_webpage(url: str) -> str:
    """Read and extract structured content from a webpage for AI analysis."""
    from apify_client import ApifyClient
    client = ApifyClient("YOUR_TOKEN")
    run = client.actor("yuchiaoniu/ai-content-extractor").call(run_input={
        "urls": [url],
        "includeRawText": True,
        "maxContentLength": 8000,
        "extractContactInfo": True,
    })
    items = client.dataset(run["defaultDatasetId"]).list_items().items
    return items[0]["textForLlm"] if items else "Could not extract content."

Once wrapped, your AI agent can call read_webpage("https://...") and get clean, LLM-ready text back instantly.

Use with LangChain

from langchain.tools import tool
from apify_client import ApifyClient

@tool
def extract_web_content(url: str) -> dict:
    """Extract structured content from a URL for analysis."""
    client = ApifyClient("YOUR_TOKEN")
    run = client.actor("yuchiaoniu/ai-content-extractor").call(run_input={
        "urls": [url],
        "extractSections": True,
        "maxContentLength": 6000,
    })
    items = client.dataset(run["defaultDatasetId"]).list_items().items
    return items[0] if items else {}

Use with JavaScript / Claude API

import Anthropic from '@anthropic-ai/sdk';
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const anthropic = new Anthropic();

// Extract page content
const run = await client.actor('yuchiaoniu/ai-content-extractor').call({
    urls: ['https://competitor.com/pricing'],
    includeRawText: true,
    maxContentLength: 8000,
});
const [page] = (await client.dataset(run.defaultDatasetId).listItems()).items;

// Feed directly into Claude
const response = await anthropic.messages.create({
    model: 'claude-opus-4-5',
    max_tokens: 1024,
    messages: [{
        role: 'user',
        content: `Analyze this pricing page and summarize the plans:\n\n${page.textForLlm}`,
    }],
});

Pair with RAG Pipeline Scraper

For full RAG pipelines, pair this Actor with RAG Pipeline Scraper:

• This Actor → structured JSON with contact info, sections, links (best for targeted pages)
• RAG Pipeline Scraper → multi-page crawl with Markdown + JSONL chunks (best for entire sites)

FAQ

Does it handle JavaScript-rendered pages? This Actor uses Cheerio (HTML parser) for maximum speed. For heavily JS-rendered sites (SPAs), consider pairing with a Playwright-based crawler first.

How many URLs can I process per run? No hard limit — tested with 500+ URLs in a single run. Set maxConcurrency in the Actor settings if you want to throttle.

What languages does it support? Any language — the extractor is language-agnostic. The lang field in output reflects the page's declared language.

Can I use this to build a RAG chatbot? Yes! Extract content with extractSections: true, then embed each section individually in your vector database for fine-grained retrieval.