RAG-Ready Website Research & Knowledge Crawler

Turn public web pages into clean research records with text quality, provenance, fingerprints, chunk-ready content and change signals for RAG and AI-agent pipelines. Returns structured, machine-readable results for repeatable research, monitoring, automation, Apify Tasks, API.

Business use and first result

For: RAG knowledge-base builders and AI research teams.

Why run it: Prepare public or licensed page content as structured research records with provenance and repeat-run change signals.

What to inspect: Cleaned text, headings, metadata, fingerprints and chunk-ready content where supported.

First run: use the documented sample input below, inspect the Dataset rows and source links, then reuse the saved output for a later comparison only where snapshots are supported. Confirm the input scope and current Pay Per Event pricing before scheduling ongoing use.

Scope and limitations: Page accessibility and text quality vary. The Actor does not grant permission to access restricted content.

Quick start

Start with this working example and replace the target values with your own:

{
  "currentItems": [
    {
      "id": "r1",
      "url": "https://docs.example.com/guide",
      "title": "API Guide",
      "description": "Integration documentation",
      "text": "This guide explains authentication, endpoints, pagination, retries and examples. This guide explains authentication, endpoints, pagination, retries and examples. This guide explains authentication, endpoints, pagination, retries and examples. This guide explains authentication, endpoints, pagination, retries and examples. This guide explains authentication, endpoints, pagination, retries and examples. This guide explains authentication, endpoints, pagination, retries and examples. This guide explains authentication, endpoints, pagination, retries and examples. This guide explains authentication, endpoints, pagination, retries and examples. This guide explains authentication, endpoints, pagination, retries and examples. This guide explains authentication, endpoints, pagination, retries and examples. This guide explains authentication, endpoints, pagination, retries and examples. This guide explains authentication, endpoints, pagination, retries and examples. ",
      "headings": [
        "Authentication",
        "Endpoints",
        "Pagination"
      ],
      "jsonLd": [
        {
          "@type": "TechArticle"
        }
      ]
    },
    {
      "id": "r2",
      "url": "https://docs.example.com/changelog",
      "title": "Changelog",
      "description": "Product updates",
      "text": "Version 2.4 adds batch operations and retry controls. Version 2.4 adds batch operations and retry controls. Version 2.4 adds batch operations and retry controls. Version 2.4 adds batch operations and retry controls. Version 2.4 adds batch operations and retry controls. Version 2.4 adds batch operations and retry controls. Version 2.4 adds batch operations and retry controls. Version 2.4 adds batch operations and retry controls. ",
      "headings": [
        "Version 2.4"
      ]
    }
  ],
  "previousItems": [
    {
      "id": "r1",
      "url": "https://docs.example.com/guide",
      "title": "API Guide",
      "description": "Integration documentation",
      "text": "This guide explains authentication, endpoints and examples. This guide explains authentication, endpoints and examples. This guide explains authentication, endpoints and examples. This guide explains authentication, endpoints and examples. This guide explains authentication, endpoints and examples. This guide explains authentication, endpoints and examples. This guide explains authentication, endpoints and examples. This guide explains authentication, endpoints and examples. This guide explains authentication, endpoints and examples. ",
      "headings": [
        "Authentication",
        "Endpoints"
      ],
      "jsonLd": [
        {
          "@type": "TechArticle"
        }
      ]
    }
  ],
  "maxItems": 20000
}

The Actor writes structured results to the default Apify Dataset and can be used from the Store, API, schedules, Tasks, automations and MCP-compatible AI workflows.

Input

• currentItems — Current source records: Optional current source rows. If supplied, these are analyzed directly and no source fetch is required.
• currentDatasetId — Current Apify Dataset ID: Optional current Dataset from an upstream extraction workflow.
• previousDatasetId — Previous Apify Dataset ID: Optional prior Dataset for recurring comparisons.
• queries — Queries: Direct-source queries for Google Maps or Google Search modes.
• urls — URLs: Direct public URLs for web or social best-effort modes.
• directFetch — Use direct-source mode when no Dataset/rows are supplied: Enable direct-source retrieval only when inline rows and Dataset input are not supplied.

All integration and advanced analysis fields are optional. The default example is intentionally runnable without configuring MCP or a previous-run baseline.

Output

The Dataset exposes predictable machine-readable output. Representative fields include results, report.

Pricing

This Actor uses Pay Per Event. The current factory base price is $0.003000 per result event. The Apify Store remains the source of truth for the price and plan/tier details shown to the buyer.

Use cases

• Run the buyer-ready workflows exposed as Apify Tasks without preparing a custom integration first.
• Use the Actor from API or schedules for recurring collection, comparison or monitoring.
• Feed the structured Dataset output into spreadsheets, databases, automations or AI agents.
• Compare repeat runs when the product supports snapshots or previous-run inputs.

Automation and AI

Use the same Actor through Apify API, schedules, public Tasks and the Apify MCP server. Outputs are structured for downstream workflows and AI agents rather than requiring manual copy/paste.

Limitations

• Public websites and upstream APIs can change markup, access rules, rate limits or field availability without notice.
• Fields that are not publicly available are returned as unavailable or omitted rather than fabricated.
• Analytical outputs depend on the quality and coverage of the supplied or collected source data.
• Treat marketplace, reputation, workforce, safety or commercial signals as decision support and validate material decisions against the underlying source evidence.

Detailed documentation

Turn public web pages into clean research records with text quality, provenance, fingerprints, chunk-ready content and change signals for RAG and AI-agent pipelines.

Why use this Actor

Web crawls frequently require a second pipeline for cleaning, provenance, deduplication, chunk readiness and deciding which changed pages need re-indexing. This Actor converts structured source data into a decision-ready result so recurring workflows do not need a separate manual analysis step. It supports repeatable monitoring, scheduled execution, dashboards, and agent workflows. Scores are deterministic and traceable to observable input fields.

Key features

• HTTP-first page extraction with clean text, headings, metadata and provenance.
• Content fingerprints for deterministic deduplication and re-index decisions.
• RAG-readiness score based on text density, structure and source metadata.
• Previous-snapshot change score to re-index only meaningful changes.
• Agent-ready INDEX/REINDEX/SKIP actions and Dataset ingestion fallback.
• Structured Dataset output for API, Task, schedule, and agent use.
• Current-versus-previous snapshot analysis when prior records are supplied.
• Limited-permission, data-first execution design.

Input

The JSON input schema includes explicit fields and prefilled examples. Important fields include currentItems, currentDatasetId, previousItems, previousDatasetId, queries, urls, directFetch, language, countryCode, maxResults, maxItems. Inline records are convenient for small tests. Dataset identifiers are better for recurring pipelines because one Actor can consume the structured output of another without copying rows manually. Stable entity identifiers improve snapshot comparison.

Output

Results are written to the default Apify Dataset as structured JSON. Output fields include ok, sourceType, analysisKind, entityId, signalScore, changeScore, agentAction, agentReason, metrics, changes, observedAt, source. These fields are designed for dashboards, databases, automation tools, and downstream Actors. Decision fields summarize observable evidence instead of returning only a large unranked export.

Example

Run the prefilled example once in the Apify Console. For a recurring workflow, save the current Dataset or upstream Dataset ID and provide a later snapshot on the next run. Compare the emitted scores, changes, and action fields. Through API or public Tasks, keep the same field mapping between runs so time-series comparisons remain meaningful.

Use cases

• recurring competitive monitoring.
• AI-agent and automation workflows.
• market research and prospect prioritization.
• period-over-period change detection.
• Scheduled daily or weekly monitoring with repeatable decision logic.
• Agent workflows that need compact actions rather than raw rows.
• Research pipelines that need stable JSON and traceable signals.

Pricing

The Actor uses pay-per-event pricing with one primary measurable result. The base design price is USD 0.0030 per primary result before Apify tier discounts. There is no mandatory external AI subscription in the deterministic scoring path. Platform usage varies with run duration, memory, and data volume. The release gate checks margin from real validation costs before publication.

Limitations

• Only public or appropriately licensed data should be supplied or fetched.
• Instagram and TikTok direct public-URL mode is best-effort; Dataset input is the stable mode when those sites block anonymous requests.
• Signals are deterministic decision support, not guarantees.
• Public-source availability and upstream schemas can change over time.
• Missing source fields can reduce the strength of a score or comparison.
• Decision scores are prioritization signals, not promises of commercial or operational outcomes.
• Users remain responsible for source access rules and platform terms.

Operational notes

For production use, prefer scheduled runs with stable input definitions. Review the Dataset after the first production run before connecting downstream systems. Preserve raw snapshots when a source changes schema so transformations remain auditable. Market evidence and competitor capabilities are rechecked before publication because Store conditions can change.