# Jina Scraper (`sshdopey/jina-scraper`) Actor

Crawl any website and get clean, LLM-ready Markdown for every page. Powered by Jina Reader, so JavaScript-heavy sites that break normal scrapers just work. Handles pagination, link discovery and glob filtering in one run. No selectors to write. Perfect for RAG pipelines and AI agents.

- **URL**: https://apify.com/sshdopey/jina-scraper.md
- **Developed by:** [Destiny](https://apify.com/sshdopey) (community)
- **Categories:** Automation, AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Jina Scraper

Crawl any website and get clean, LLM-ready Markdown for every page, in a single run.

This Actor uses [Jina Reader](https://jina.ai/reader) as its fetching engine. Jina renders the page, including JavaScript-heavy sites, and returns clean Markdown instead of raw HTML. No selectors to write, no HTML parsing.

Jina Reader only reads one page at a time. This Actor adds the crawling around it: pagination, link discovery, glob filtering, deduplication and concurrency.

### What it does

1. Reads your start URL and pulls every link on the page.
2. Keeps only the links matching your detail-page glob.
3. Walks the pagination URLs and repeats, stopping automatically when a page yields nothing new.
4. Fetches each matched page through Jina and returns the Markdown.

One run replaces the usual two-step setup of a listing scraper followed by a separate detail scraper.

### Use cases

- Blogs, news archives, documentation sites, job boards, product catalogues, directories, listings
- Building RAG or LLM pipelines that need clean text rather than HTML
- Sites where conventional scrapers fail because content is rendered client-side

### Input

| Field | Type | Default | Notes |
|---|---|---|---|
| `startUrls` | array | required | Listing, search or index pages to start from |
| `detailPageGlob` | string | | Glob for detail pages. `*` matches within one path segment, `**` crosses segments |
| `paginationUrlGlob` | string | | Glob where `*` is the page number, e.g. `https://site.com/blog?page=*` |
| `maxPages` | integer | 2 | Listing pages to walk |
| `maxDetailPages` | integer | 5 | Hard cap on detail pages fetched. `0` means unlimited |
| `jinaApiKey` | secret | | Strongly recommended, see below |
| `stripMedia` | boolean | true | Removes images, video embeds and media links from the Markdown |
| `retainImages` | select | none | Only used when `stripMedia` is off |
| `targetSelector` | string | | CSS selector for the listing container. Cuts token cost significantly |
| `engine` | select | direct | `browser` for JS-heavy sites, `direct` for static HTML |
| `concurrency` | integer | 2 | Parallel Jina requests during extraction |
| `requestDelayMs` | integer | 0 | Politeness delay per worker |

#### Example input

```json
{
    "startUrls": ["https://example.com/blog"],
    "detailPageGlob": "https://example.com/blog/*",
    "paginationUrlGlob": "https://example.com/blog?page=*",
    "maxPages": 5,
    "maxDetailPages": 100,
    "engine": "browser",
    "concurrency": 5
}
```

### Output

One dataset item per detail page:

```json
{
    "url": "https://example.com/blog/how-we-scaled-ingest",
    "title": "How we scaled ingest to 10M docs a day",
    "description": "A walkthrough of the architecture changes...",
    "markdown": "# How we scaled ingest\n\nWe started with a single queue...",
    "markdownLength": 2841,
    "listingUrl": "https://example.com/blog?page=2",
    "foundOnPage": 2,
    "scrapedAt": "2026-07-27T09:15:00.000Z",
    "error": null
}
```

`title` and `description` come from the page's `<title>` and meta description tags, so they are `null` on sites that do not set them. `markdown` is the field to rely on.

Pages that fail are still pushed, with `error` populated and `markdown` set to `null`. Filter on `error = null` downstream.

A `RUN_SUMMARY` record is written to the key-value store at the end of every run:

```json
{
    "discovered": 47,
    "attempted": 47,
    "succeeded": 46,
    "failed": 1,
    "jinaCalls": 52,
    "totalTokens": 384210,
    "estimatedCostUsd": 0.00768
}
```

### You need a Jina API key

Get one at [jina.ai/reader](https://jina.ai/reader) and paste it into the `jinaApiKey` input field.

Anonymous requests are limited to roughly 20 per minute, and Jina rejects anonymous traffic from many datacenter IP ranges outright with a 401. Without a key this Actor will fail on most runs.

New keys include a free token allowance. Reader is billed per token thereafter, so `maxDetailPages` and `targetSelector` are your two main cost controls. Check `RUN_SUMMARY.estimatedCostUsd` after a test run and scale from there.

### Writing globs

`*` matches within a single path segment. `**` crosses segments.

| Glob | Matches | Does not match |
|---|---|---|
| `https://site.com/blog/*` | `/blog/my-post` | `/blog/tags/rust` |
| `https://site.com/blog/**` | both of the above | |
| `https://site.com/docs/*/index.html` | `/docs/intro/index.html` | `/docs/api/v2/index.html` |

Start narrow. If the log shows `0 matched`, widen the glob rather than raising `maxPages`.

### Reading the log

```
Page 1 [https://site.com/blog]: 47 links, 12 matched, 12 new
Page 2 [https://site.com/blog?page=2]: 51 links, 14 matched, 14 new
```

- `0 matched` means the detail glob is wrong.
- `0 new` on page 2 means the pagination glob is not advancing, and the crawl stops.

### Limitations

- Jina Reader is not a bot-protection bypass. Sites behind aggressive Cloudflare challenges may still fail.
- Pagination assumes a numeric page parameter. Cursor-based or infinite-scroll pagination is not supported.
- The crawl stops as soon as a listing page produces no new matches, which can end a run early if a page renders slowly. Increase `requestDelayMs` or switch `engine` to `browser` if that happens.
- Link discovery reads links present in the rendered page. Links injected after user interaction are not seen.

### Tips

- Set `targetSelector` to the listing container (`.post-list`, `#results`). It removes navigation and footer noise before you are billed for it.
- Use `engine: direct` for static HTML. It is faster and cheaper. Switch to `browser` only when a page comes back empty.
- Lower `concurrency` and raise `requestDelayMs` if you see repeated 429 warnings.

# Actor input Schema

## `startUrls` (type: `array`):

Listing, search or index pages to start crawling from.

## `detailPageGlob` (type: `string`):

Glob matching the detail pages you want to scrape. \* matches within one path segment, \*\* crosses segments. Example: https://site.com/article/\*

## `paginationUrlGlob` (type: `string`):

Glob where \* is the page number, e.g. https://site.com/search?page=\*

## `maxPages` (type: `integer`):

How many listing pages to walk before stopping.

## `maxDetailPages` (type: `integer`):

Hard cap on detail pages fetched per run. Set to 0 for unlimited. This is your main cost guard.

## `jinaApiKey` (type: `string`):

Your Jina Reader API key from jina.ai. Strongly recommended: anonymous requests are heavily rate limited and are rejected from many datacenter IPs. Overrides the JINA\_API\_KEY environment variable.

## `stripMedia` (type: `boolean`):

Removes images, video embeds, media links and Jina image placeholders. Leave on for clean LLM input.

## `retainImages` (type: `string`):

Only applies when Strip media is unchecked. none removes images, alt keeps alt text, all keeps image URLs.

## `targetSelector` (type: `string`):

Optional CSS selector for the content container on listing pages. Cuts token cost significantly.

## `engine` (type: `string`):

browser handles JS-heavy sites. direct is faster and cheaper for static HTML.

## `concurrency` (type: `integer`):

Parallel Jina requests during extraction. Lower this if you hit 429s.

## `requestDelayMs` (type: `integer`):

Politeness delay per worker. Set to 500-1000 if a site rate limits you.

## Actor input object example

```json
{
  "startUrls": [
    "https://books.toscrape.com/"
  ],
  "detailPageGlob": "https://books.toscrape.com/catalogue/*/index.html",
  "paginationUrlGlob": "https://books.toscrape.com/catalogue/page-*.html",
  "maxPages": 2,
  "maxDetailPages": 5,
  "stripMedia": true,
  "retainImages": "none",
  "engine": "direct",
  "concurrency": 2,
  "requestDelayMs": 0
}
```

# Actor output Schema

## `results` (type: `string`):

One item per detail page, with title, description and clean Markdown. Failed pages are included with the error field populated.

## `runSummary` (type: `string`):

Counts of discovered, succeeded and failed pages, plus Jina call volume, token usage and estimated cost in USD.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://books.toscrape.com/"
    ],
    "detailPageGlob": "https://books.toscrape.com/catalogue/*/index.html",
    "paginationUrlGlob": "https://books.toscrape.com/catalogue/page-*.html"
};

// Run the Actor and wait for it to finish
const run = await client.actor("sshdopey/jina-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://books.toscrape.com/"],
    "detailPageGlob": "https://books.toscrape.com/catalogue/*/index.html",
    "paginationUrlGlob": "https://books.toscrape.com/catalogue/page-*.html",
}

# Run the Actor and wait for it to finish
run = client.actor("sshdopey/jina-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://books.toscrape.com/"
  ],
  "detailPageGlob": "https://books.toscrape.com/catalogue/*/index.html",
  "paginationUrlGlob": "https://books.toscrape.com/catalogue/page-*.html"
}' |
apify call sshdopey/jina-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=sshdopey/jina-scraper",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/rBJgCVBlZTm6lc2EP/builds/bus7NkM74I0fy2SVu/openapi.json