RAG-Ready Website Research & Knowledge Crawler

Turn public web pages into clean research records with text quality, provenance, fingerprints, chunk-ready content and change signals for RAG and AI-agent pipelines.

Why use this Actor

Web crawls frequently require a second pipeline for cleaning, provenance, deduplication, chunk readiness and deciding which changed pages need re-indexing. This Actor converts structured source data into a decision-ready result so recurring workflows do not need a separate manual analysis step. It supports repeatable monitoring, scheduled execution, dashboards, and agent workflows. Scores are deterministic and traceable to observable input fields.

Key features

• HTTP-first page extraction with clean text, headings, metadata and provenance.
• Content fingerprints for deterministic deduplication and re-index decisions.
• RAG-readiness score based on text density, structure and source metadata.
• Previous-snapshot change score to re-index only meaningful changes.
• Agent-ready INDEX/REINDEX/SKIP actions and Dataset ingestion fallback.
• Structured Dataset output for API, Task, schedule, and agent use.
• Current-versus-previous snapshot analysis when prior records are supplied.
• Limited-permission, data-first execution design.

Input

The JSON input schema includes explicit fields and prefilled examples. Important fields include currentItems, currentDatasetId, previousItems, previousDatasetId, queries, urls, directFetch, language, countryCode, maxResults, maxItems. Inline records are convenient for small tests. Dataset identifiers are better for recurring pipelines because one Actor can consume the structured output of another without copying rows manually. Stable entity identifiers improve snapshot comparison.

Output

Results are written to the default Apify Dataset as structured JSON. Output fields include ok, sourceType, analysisKind, entityId, signalScore, changeScore, agentAction, agentReason, metrics, changes, observedAt, source. These fields are designed for dashboards, databases, automation tools, and downstream Actors. Decision fields summarize observable evidence instead of returning only a large unranked export.

Example

Run the prefilled example once in the Apify Console. For a recurring workflow, save the current Dataset or upstream Dataset ID and provide a later snapshot on the next run. Compare the emitted scores, changes, and action fields. Through API or public Tasks, keep the same field mapping between runs so time-series comparisons remain meaningful.

Use cases

• recurring competitive monitoring.
• AI-agent and automation workflows.
• market research and prospect prioritization.
• period-over-period change detection.
• Scheduled daily or weekly monitoring with repeatable decision logic.
• Agent workflows that need compact actions rather than raw rows.
• Research pipelines that need stable JSON and traceable signals.

Pricing

The Actor uses pay-per-event pricing with one primary measurable result. The base design price is USD 0.0030 per primary result before Apify tier discounts. There is no mandatory external AI subscription in the deterministic scoring path. Platform usage varies with run duration, memory, and data volume. The release gate checks margin from real validation costs before publication.

Limitations

• Only public or appropriately licensed data should be supplied or fetched.
• Instagram and TikTok direct public-URL mode is best-effort; Dataset input is the stable mode when those sites block anonymous requests.
• Signals are deterministic decision support, not guarantees.
• Public-source availability and upstream schemas can change over time.
• Missing source fields can reduce the strength of a score or comparison.
• Decision scores are prioritization signals, not promises of commercial or operational outcomes.
• Users remain responsible for source access rules and platform terms.

Operational notes

For production use, prefer scheduled runs with stable input definitions. Review the Dataset after the first production run before connecting downstream systems. Preserve raw snapshots when a source changes schema so transformations remain auditable. Market evidence and competitor capabilities are rechecked before publication because Store conditions can change.