Agent Memory Feeder

Turn web pages and docs into clean, chunked markdown memory your AI agent can store in a vector DB and retrieve. Point it at a list of URLs; get back deduplicated, sentence-aware markdown chunks with token estimates and a source hash for each — ready to embed.

Built for agents: direct-first fetching (proxy only as a paid fallback), boilerplate/nav/ad stripping via Mozilla Readability, HTML→markdown conversion, overlapping chunking on sentence boundaries, and near-duplicate removal (Jaccard shingling).

Input

Field	Type	Required	Default	Description
urls	string[]	yes	—	Page/doc URLs to convert into memory.
maxCharsPerChunk	integer	no	1200	Target max chunk size in characters (~4 chars/token).
chunkOverlap	integer	no	150	Characters of context carried between adjacent chunks.
proxyConfiguration	object	no	Apify Proxy	Proxy used only when a direct fetch is blocked (403/429/503). Billed to you.

Output

One dataset row per memory chunk:

{
  "url": "https://example.com/page",
  "title": "Page Title",
  "chunkIndex": 0,
  "totalChunks": 7,
  "text": "# Heading\n\nClean markdown chunk...",
  "charCount": 1187,
  "tokenEstimate": 297,
  "sourceHash": "9f2c...e1"
}

sourceHash is the SHA-256 of the full extracted source text — use it to detect when a page changed and re-embed only what moved.

Behavior

• Direct-first, proxy-fallback. Fetches go out on the datacenter IP first; the caller proxy is used only when a page returns 403/429/503. All proxy is billed to you.
• Partial success. If some URLs fail, you still get chunks for the ones that worked; failures are logged.
• Fail-loud. If every URL yields no content, the run exits non-zero with a status message (nothing is billed for empty URLs).

Run it from an agent

MCP (Apify Actors MCP server)

Expose this Actor as a tool via the Apify MCP server, then call:

{
  "tool": "apricot_blackberry/agent-memory-feeder",
  "input": {
    "urls": ["https://en.wikipedia.org/wiki/Retrieval-augmented_generation"],
    "maxCharsPerChunk": 1200,
    "chunkOverlap": 150
  }
}

curl (run and fetch items)

curl -X POST "https://api.apify.com/v2/acts/apricot_blackberry~agent-memory-feeder/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://en.wikipedia.org/wiki/Retrieval-augmented_generation"]}'

JavaScript (apify-client)

import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: 'YOUR_TOKEN' });

const run = await client.actor('apricot_blackberry/agent-memory-feeder').call({
    urls: ['https://en.wikipedia.org/wiki/Retrieval-augmented_generation'],
    maxCharsPerChunk: 1200,
    chunkOverlap: 150,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const chunk of items) {
    // await vectorStore.add({ id: `${chunk.sourceHash}:${chunk.chunkIndex}`, text: chunk.text });
    console.log(chunk.chunkIndex, chunk.tokenEstimate, chunk.title);
}

Python (apify-client)

from apify_client import ApifyClient

client = ApifyClient("YOUR_TOKEN")
run = client.actor("apricot_blackberry/agent-memory-feeder").call(run_input={
    "urls": ["https://en.wikipedia.org/wiki/Retrieval-augmented_generation"],
    "maxCharsPerChunk": 1200,
    "chunkOverlap": 150,
})
for chunk in client.dataset(run["defaultDatasetId"]).iterate_items():
    # vector_store.add(id=f'{chunk["sourceHash"]}:{chunk["chunkIndex"]}', text=chunk["text"])
    print(chunk["chunkIndex"], chunk["tokenEstimate"], chunk["title"])

Pricing

Pay-per-event. A small actor-start fee plus a per-chunk fee for each memory chunk produced. Empty/failed URLs are never charged. Proxy usage, when the fallback triggers, is billed to your account.