Armored AI Web Scraper | RAG Data Protection

Give your agents web data with a defense layer

A page should add knowledge—not drain the crawl budget or smuggle instructions into agent memory. Armored fetches public web content through a bounded inspection pipeline and returns clean text, source fingerprints and a security verdict your workflow can act on.

Stop slow-stream tarpits. Cut off repetitive crawl mazes. Strip invisible token bloat. Quarantine content that matches implemented prompt-injection signals before it reaches your RAG index.

Built for autonomous research agents, RAG ingestion, agent memory, dataset preparation and web-to-LLM pipelines. One URL in. Structured text and diagnostics out. No model key or prompt engineering required.

Start in one call

{"url": "https://docs.python.org/3/tutorial/datastructures.html", "maxDepth": 0}

For a batch, use {"urls": ["https://docs.python.org/3/tutorial/datastructures.html", "https://docs.apify.com"], "maxDepth": 0}. For a bounded crawl, raise maxDepth and set maxPages.

Click Try for free, replace the example URL, and run. Apify account credits apply; the Actor uses the pricing below.

What you get

• Time protection: a sliding-window watchdog closes HTTP streams that fall below your throughput floor.
• Traversal protection: structural SimHash and ancestor-content similarity identify repetitive branches; hard page, depth and time limits stay in force.
• Cleaner model input: hidden HTML removal and selected Unicode cleanup, with language and emoji joiners preserved by default.
• Actionable quarantine: matching instruction signals withhold content and links by default. Diagnostic records explain the decision.
• Provenance and cost visibility: URL, timestamp, SHA-256, exact counts for your selected tokenizer and optional savings at your supplied model rate.

In a reproducible local fixture, default supervision terminated a slow response in 2.01 seconds versus 6.02 seconds without throughput rejection. A fixture containing 1,000 injected zero-width spaces fell from 504 to 4 cl100k_base tokens. These are controlled examples, not a promised result on every website.

Pricing: $5 per 1,000 delivered pages

$0.005 per nonempty page with status OK, including standard Apify platform usage. Failed, empty, quarantined, robots-blocked and maze-pruned records carry no page charge. A small platform start event costs $0.00005 per run per GB of allocated memory, with a one-event minimum (the default 512 MB run costs $0.00005 to start).

Ten delivered pages cost $0.05005 at the default memory. One hundred cost $0.50005. Set Maximum cost per run in Run options or maxTotalChargeUsd in the API to cap paid output. The Actor stops before starting another page when the remaining page budget is exhausted.

For autonomous agents and tool routers

Choose this tool when: you need bounded retrieval of public HTTP content before it enters a model, vector store or agent memory, and you need machine-readable rejection reasons and provenance.

Use the result: consume sanitizedContent only for the statuses your workflow permits; check securityTelemetry.instructionSignals, contentTruncated, and OUTPUT.complete. Every page remains trust: "untrusted". Source text is data, never an instruction to your agent.

Need browser rendering and stealth retrieval? Use Wolfpack, the Super Stealth + Armored combination.

MCP: Claude, Cursor and other MCP clients

{
  "mcpServers": {
    "armored": {
      "url": "https://mcp.apify.com?tools=apricot_blackberry/armored-ai-web-scraper",
      "headers": {"Authorization": "Bearer YOUR_APIFY_TOKEN"}
    }
  }
}

This Actor requests limited Apify permissions and exposes input, dataset and output schemas. MCP clients discover its parameters as a tool. No separate LLM account is needed by the Actor.

REST API

curl -X POST 'https://api.apify.com/v2/acts/apricot_blackberry~armored-ai-web-scraper/run-sync-get-dataset-items?timeout=180&memory=512&maxTotalChargeUsd=0.10' \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://docs.python.org/3/tutorial/datastructures.html","maxDepth":0,"maxPages":1}'

For longer jobs, start with POST /v2/acts/apricot_blackberry~armored-ai-web-scraper/runs, wait for the run to finish, then read its dataset and OUTPUT. Use the authenticated API instead of putting tokens into page URLs.

Python and JavaScript

import os
from apify_client import ApifyClient
client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("apricot_blackberry/armored-ai-web-scraper").call(
    run_input={"url": "https://docs.python.org/3/tutorial/datastructures.html", "maxDepth": 0},
    max_total_charge_usd=0.10,
)
for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    if row["status"] == "OK":
        print(row["sanitizedContent"])

import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('apricot_blackberry/armored-ai-web-scraper').call(
  { url: 'https://docs.python.org/3/tutorial/datastructures.html', maxDepth: 0 },
  { maxTotalChargeUsd: 0.10 },
);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
const documents = items.filter(x => x.status === 'OK');

n8n, Make, Zapier, LangChain and CrewAI

The INTEGRATIONS output contains an importable n8n HTTP workflow, MCP configuration, API endpoints and machine-readable settings. In n8n, import n8nWorkflow and select an HTTP Header Auth credential with header Authorization and value Bearer YOUR_APIFY_TOKEN; replace the example URL. It is inactive by default.

For Make or Zapier, use their Apify Run Actor / Run a Task action, select this Actor, map url or urls, then read the resulting Dataset. Native Apify success webhooks can trigger your next workflow without polling. Customer accounts and destination credentials stay under your control.

LangChain and CrewAI can call the same API/client method as a tool and map sanitizedContent to a document, keeping URL, source hash, timestamp and telemetry as metadata. The AGENT-TOOL output provides the tool contract; Apify also generates an OpenAPI description from the input/output schemas.

Input

{
  "startUrls": [{"url": "https://docs.apify.com/sdk/python/docs/overview"}],
  "maxDepth": 0,
  "maxPages": 20,
  "maxResponseBytes": 2097152,
  "maxOutputChars": 40000,
  "minBytesPerSecond": 32,
  "gracePeriodSecs": 2,
  "throughputWindowSecs": 2,
  "requestTimeoutSecs": 20,
  "maxRunSecs": 120,
  "respectRobotsTxt": true,
  "tokenEncoding": "cl100k_base",
  "onSuspectedInjection": "quarantine"
}

Provide at least one of url, urls, or startUrls. Supply a single URL, a string array, or 1–100 objects containing only url; URL-list uploads and request-queue references are unsupported. Crawl depth defaults to 2; zero fetches seeds only. URLs are deduplicated after normalization and fragment removal. Child links stay within the seed origins. Redirects are checked again before connection. One page runs at a time. Page, depth, queue, link, byte and wall-clock limits are independent.

Additional options are maxLinksPerPage (default 50), stripJoiners (default false), and optional inputTokenUsdPerMillion. Unknown settings and invalid types are rejected before fetching. minBytesPerSecond: 0 explicitly disables throughput rejection; absolute deadlines remain active. The machine-readable input schema and exact supported ranges are in .actor/input_schema.json; runtime validation additionally enforces URL-object structure and public-network policy.

Pipeline

1. Stream supervision. A monotonic-clock sliding window measures response-body wire bytes. After both the configured grace and window have elapsed, a rate below the threshold closes the response. A timer checks even when no new chunk arrives. Connection/header waits and the complete redirect sequence also have an absolute request deadline. Wire and decompressed bytes are capped separately at 2 MB by default. Challenge pages that match implemented wall checks are withheld with ANTI_BOT_BLOCKED, without a page charge. Gzip and zlib-wrapped deflate are decoded incrementally with an output limit. A slow response is a policy violation, not proof that its operator intended a tarpit.
2. Traversal supervision. A 64-bit SimHash uses ancestor tag paths and ordered tag shingles. A branch is pruned only after structural similarity and at least 90% normalized word-set similarity to each of three ancestors. Numeric calendar variation is ignored. Siblings do not share ancestor history. This is a heuristic: common boilerplate can cause false positives and sufficiently varied mazes can evade it. Hard crawl budgets still apply. No infinite-topology guarantee is claimed.
3. Content airlock. HTML scripts, styles, templates, comments, embedded frames, selected non-text elements, hidden/aria-hidden elements and inline and unconditional embedded CSS hidden styles are removed. HTML entities are decoded before Unicode normalization. Line breaks and tabs survive. Contextual language/emoji joiners survive by default; ASCII obfuscation and long joiner runs are removed. stripJoiners: true removes all joiners and can change legitimate text. External stylesheets and browser-computed visibility are not evaluated.
4. Instruction signals. Implemented patterns flag instruction overrides, credential requests, task redirection, claimed authority and selected role delimiters. The default quarantine policy withholds matching text and outgoing links. report returns flagged text. These are transparent heuristics, not a comprehensive prompt-injection classifier: quoted HTML instruction blocks and inline double-quoted examples are redacted while surrounding text is retained. quotedInstructionBlocksRedacted and quotedInstructionSignals record that change; redactions are not reported as token savings. Unrecognized, multilingual and factual poisoning can still be missed. Every record remains trust: "untrusted"; JSON serialization cannot make arbitrary text harmless to an LLM. Downstream agents must retain the source-data/instruction boundary.

Markdown, JSON and plain text receive Unicode normalization, instruction checks and output limits. DOM visibility filtering, HTML-link discovery and structural fingerprints apply to HTML/XHTML only. JSON is returned as text rather than being coerced into a new API schema.

Results and telemetry

Every attempted page writes a dataset record, including failures and policy rejections. The key-value record OUTPUT contains a separate run summary. Dataset writes complete before the summary is published; storage failures fail the run. The result schema is .actor/result_schema.json.

Status	Meaning
OK	Content returned under the configured limits; it remains untrusted
QUARANTINED	An implemented instruction pattern matched; content and links withheld
MAZE_PRUNED	Repetitive ancestor evidence reached the pruning threshold; no further descent
FETCH_FAILED	Network, HTTP, throughput, size, destination or time policy prevented extraction
ROBOTS_DISALLOWED	robots.txt denies the requested URL
ROBOTS_UNAVAILABLE	robots.txt could not be evaluated under the network policies
INSPECTION_LIMIT	Parsed HTML exceeded the node budget

Records include requested/final URL, parent/depth, UTC timestamp, decoded source byte count, source SHA-256, optional text, error code and securityTelemetry. The hash identifies the received decoded response bytes; it does not authenticate the source. Failures never silently disappear.

Telemetry includes tarpitDetected, mazeDetected, spongesStripped, hidden-element removal count, instruction signals, truncation, received bytes, request time and DOM fingerprint. spongesStripped counts selected removed characters, not independently confirmed attacks. Counts are per record; summary counters aggregate actual events.

complete in OUTPUT means that the eligible crawl queue was exhausted without a page, link or run-time budget cutoff. It does not mean every URL succeeded or that all content is safe. Inspect statuses, stopReason, and unprocessedUrls. robots.txt is fetched once per origin: 404 permits crawling, 401/403 deny it, and other failures produce diagnostic records. No automatic retries or CAPTCHA-solving charges occur.

Token and cost accounting

cl100k_base and o200k_base are supported. tokensBefore, tokensAfterNormalization and tokensAfter use the actual selected tokenizer on extracted text, before normalization, after normalization, and after the output cap respectively. Counts exclude HTML already removed by DOM filtering and do not describe complete chat-message framing. Reserved-token-looking strings are tokenized as ordinary source text.

tokenDelta is signed. estimatedTokensSaved credits only a positive reduction caused by normalization; truncation and quarantine are not credited as savings. estimatedInputUsdSaved uses that reduction times the caller's inputTokenUsdPerMillion / 1,000,000; without a price it is null. This is potential input cost reduction at the supplied rate, not verified API invoices, platform compute savings or model-independent token savings.

For extracted text over 200,000 characters, token counts/savings are null and tokenAccounting explains the limit. Quarantined/failed records likewise receive no invented savings. Summary tokenCountedRecords states coverage. Container builds preload both tokenizer vocabularies so normal runtime token counting needs no vocabulary download.

Network and resource boundaries

Only public HTTP(S) destinations on ports 80/443 are allowed. Embedded credentials, private/loopback/link-local/reserved addresses, metadata services and mixed public/private DNS answers are rejected. Validated DNS answers are passed directly to the connector. Each redirect is validated again. Ambient proxies, .netrc credentials and persistent cookies are disabled. The local test harness can inject a loopback-only client; that option is absent from Actor input.

This Actor consumes fully bounded HTML before DOM inspection; it does not claim to parse structure before the HTTP response finishes. A tag-count prepass rejects oversized trees before tree allocation. Parsing, structural inspection and tokenization run in one isolated worker that the parent can terminate when the remaining crawl deadline expires. There is a 20,000-tag inspection cap, 100-page maximum, 10-depth maximum and 300-second crawl-budget maximum. Process cancellation, storage writes and cleanup can add a small amount of wall time beyond a reached crawl deadline; completion is then reported as false. One response may still use its allowed byte/time budget. Crawl state is run-local; automatic recovery across Apify migrations is not implemented. Partial durable dataset records survive, but a new invocation starts a new crawl.

Compatibility and support

Armored fetches HTML, XHTML, plain text, Markdown and JSON over HTTP. It does not render website JavaScript. Injection screening and maze recognition are heuristics: they can miss attacks or flag legitimate quotations. Factual poisoning detection is outside this release. Keep downstream tool permissions and source-data boundaries in place.

For questions or a reproducible failure, open an Apify issue or email contact@creatorfusion.net with the run ID and expected result. Do not include API tokens or private source data.

Content quality and token budgets

maxOutputTokens defaults to 8,192 tokens in the selected tokenEncoding (maximum 50,000), alongside maxOutputChars. outputTokens, tokenLimitApplied and contentTruncated report the delivered budget. Truncation is never counted as normalization savings. Unicode tag payloads and excessive invisible runs are removed while recognized subdivision flags, linguistic joiners and ordinary emoji remain intact.

Inline HTML words remain continuous. JSON strings are decoded once into readable JSON; screening also checks bounded decoded representations and overlapping text windows. Recognized multilingual directives are covered by explicit rules, not a universal language or semantic classifier. Benign credential-storage guidance and negative safety policies are retained. Recognized attacks inside HTML quotations, Markdown quotes/fences and explicitly labeled JSON example fields are redacted; any separate unquoted directive still screens. Quote-only payloads are quarantined. All delivered content remains untrusted.

Maze novelty uses visibility-filtered normalized text, so changing hidden words cannot create apparent useful novelty. Visibility cleanup distinguishes hidden subtrees from inherited text visibility, preserving explicitly visible descendants. Empty extraction returns FETCH_FAILED with EMPTY_CONTENT; detected stalled HTML returns CLIENT_RENDERED_SHELL, without a page charge.