Jina Scraper

Crawl any website and get clean, LLM-ready Markdown for every page, in a single run.

This Actor uses Jina Reader as its fetching engine. Jina renders the page, including JavaScript-heavy sites, and returns clean Markdown instead of raw HTML. No selectors to write, no HTML parsing.

Jina Reader only reads one page at a time. This Actor adds the crawling around it: pagination, link discovery, glob filtering, deduplication and concurrency.

What it does

1. Reads your start URL and pulls every link on the page.
2. Keeps only the links matching your detail-page glob.
3. Walks the pagination URLs and repeats, stopping automatically when a page yields nothing new.
4. Fetches each matched page through Jina and returns the Markdown.

One run replaces the usual two-step setup of a listing scraper followed by a separate detail scraper.

Use cases

• Blogs, news archives, documentation sites, job boards, product catalogues, directories, listings
• Building RAG or LLM pipelines that need clean text rather than HTML
• Sites where conventional scrapers fail because content is rendered client-side

Input

Field	Type	Default	Notes
startUrls	array	required	Listing, search or index pages to start from
detailPageGlob	string		Glob for detail pages. * matches within one path segment, ** crosses segments
paginationUrlGlob	string		Glob where * is the page number, e.g. https://site.com/blog?page=*
maxPages	integer	2	Listing pages to walk
maxDetailPages	integer	5	Hard cap on detail pages fetched. 0 means unlimited
jinaApiKey	secret		Strongly recommended, see below
stripMedia	boolean	true	Removes images, video embeds and media links from the Markdown
retainImages	select	none	Only used when stripMedia is off
targetSelector	string		CSS selector for the listing container. Cuts token cost significantly
engine	select	direct	browser for JS-heavy sites, direct for static HTML
concurrency	integer	2	Parallel Jina requests during extraction
requestDelayMs	integer	0	Politeness delay per worker

Example input

{
    "startUrls": ["https://example.com/blog"],
    "detailPageGlob": "https://example.com/blog/*",
    "paginationUrlGlob": "https://example.com/blog?page=*",
    "maxPages": 5,
    "maxDetailPages": 100,
    "engine": "browser",
    "concurrency": 5
}

Output

One dataset item per detail page:

{
    "url": "https://example.com/blog/how-we-scaled-ingest",
    "title": "How we scaled ingest to 10M docs a day",
    "description": "A walkthrough of the architecture changes...",
    "markdown": "# How we scaled ingest\n\nWe started with a single queue...",
    "markdownLength": 2841,
    "listingUrl": "https://example.com/blog?page=2",
    "foundOnPage": 2,
    "scrapedAt": "2026-07-27T09:15:00.000Z",
    "error": null
}

title and description come from the page's <title> and meta description tags, so they are null on sites that do not set them. markdown is the field to rely on.

Pages that fail are still pushed, with error populated and markdown set to null. Filter on error = null downstream.

A RUN_SUMMARY record is written to the key-value store at the end of every run:

{
    "discovered": 47,
    "attempted": 47,
    "succeeded": 46,
    "failed": 1,
    "jinaCalls": 52,
    "totalTokens": 384210,
    "estimatedCostUsd": 0.00768
}

You need a Jina API key

Get one at jina.ai/reader and paste it into the jinaApiKey input field.

Anonymous requests are limited to roughly 20 per minute, and Jina rejects anonymous traffic from many datacenter IP ranges outright with a 401. Without a key this Actor will fail on most runs.

New keys include a free token allowance. Reader is billed per token thereafter, so maxDetailPages and targetSelector are your two main cost controls. Check RUN_SUMMARY.estimatedCostUsd after a test run and scale from there.

Writing globs

* matches within a single path segment. ** crosses segments.

Glob	Matches	Does not match
https://site.com/blog/*	/blog/my-post	/blog/tags/rust
https://site.com/blog/**	both of the above
https://site.com/docs/*/index.html	/docs/intro/index.html	/docs/api/v2/index.html

Start narrow. If the log shows 0 matched, widen the glob rather than raising maxPages.

Reading the log

Page 1 [https://site.com/blog]: 47 links, 12 matched, 12 new
Page 2 [https://site.com/blog?page=2]: 51 links, 14 matched, 14 new

• 0 matched means the detail glob is wrong.
• 0 new on page 2 means the pagination glob is not advancing, and the crawl stops.

Limitations

• Jina Reader is not a bot-protection bypass. Sites behind aggressive Cloudflare challenges may still fail.
• Pagination assumes a numeric page parameter. Cursor-based or infinite-scroll pagination is not supported.
• The crawl stops as soon as a listing page produces no new matches, which can end a run early if a page renders slowly. Increase requestDelayMs or switch engine to browser if that happens.
• Link discovery reads links present in the rendered page. Links injected after user interaction are not seen.

Tips

• Set targetSelector to the listing container (.post-list, #results). It removes navigation and footer noise before you are billed for it.
• Use engine: direct for static HTML. It is faster and cheaper. Switch to browser only when a page comes back empty.
• Lower concurrency and raise requestDelayMs if you see repeated 429 warnings.