# Obsidian Web Clipper Scraper (`automation-lab/website-to-obsidian-markdown`) Actor

Convert public web pages into clean Obsidian-ready Markdown with source metadata, links, and stable filenames.

- **URL**: https://apify.com/automation-lab/website-to-obsidian-markdown.md
- **Developed by:** [Stas Persiianenko](https://apify.com/automation-lab) (community)
- **Categories:** Developer tools, Integrations
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.15 / 1,000 item extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Obsidian Web Clipper Scraper

Turn supplied public web pages into clean, portable, Obsidian-ready Markdown notes.

This **Obsidian web clipper** Actor extracts readable page content, preserves useful headings and links, adds source metadata, and creates a stable `.md` filename for each URL. It is built for repeatable server-side capture: use it from Apify Console, a schedule, the API, Make, Zapier, n8n, or an MCP client without installing a browser extension.

### What does Obsidian Web Clipper Scraper do?

Give the Actor one or more anonymous public HTTP(S) page URLs. For every page it can read, it returns:

- clean Markdown converted from the page's readable content;
- optional Obsidian YAML frontmatter;
- source, canonical URL, title, description, author, publication date, language, and site metadata when available;
- a stable filename based on the canonical URL;
- absolute links retained in the note;
- an extracted-link list for downstream processing;
- per-URL success or error status.

The default dataset contains one record per requested URL. Successful notes are charged as items. Error records are useful diagnostics and do not incur the item event.

### Who is this web-to-Obsidian converter for?

#### Researchers

Capture source pages with citations and canonical URLs before organizing them in an Obsidian vault.

#### Developers and technical writers

Convert documentation pages into Markdown for an offline reference library, internal notes, or review workflows.

#### Analysts

Collect a bounded batch of public articles or reference pages in a consistent format for text analysis or knowledge-base ingestion.

#### Automation builders

Replace manual copy-and-paste with scheduled Apify runs and feed the resulting Markdown to cloud storage, Git, an ETL tool, or a custom Obsidian sync process.

### Why use this instead of copying page text?

Manual copying commonly loses source metadata, resolves links inconsistently, includes navigation noise, and creates filenames that change from one capture to the next.

This Actor provides a repeatable data contract:

1. public destinations and redirects are validated;
2. readable content is selected from server-rendered HTML;
3. unsafe and noisy elements are removed;
4. relative links and image references become absolute URLs;
5. HTML becomes clean Markdown;
6. metadata becomes optional YAML frontmatter;
7. a deterministic canonical-URL hash keeps the filename stable across recurring runs.

The result is ready to write as a Markdown file, while the dataset fields remain available for integrations.

### Getting started

1. Open the Actor input.
2. Add one or more public page URLs to **Web page URLs**.
3. Keep **Include Obsidian frontmatter** enabled for vault-ready notes.
4. Add optional shared tags such as `research` or `documentation`.
5. Choose whether to keep image references.
6. Click **Start**.
7. Open the default dataset.
8. Save each `markdown` value using its corresponding `filename`.

The prefilled Obsidian Docs and MDN URLs are real public pages and can be used for a first run.

### Input parameters

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `startUrls` | array | required | One to 100 anonymous public HTTP(S) page URLs. Duplicate normalized URLs are processed once. |
| `maxItems` | integer | `10` | Maximum number of unique supplied URLs to process, from 1 to 100. |
| `includeFrontmatter` | boolean | `true` | Include Obsidian-compatible YAML frontmatter before the Markdown body. |
| `includeImages` | boolean | `true` | Keep image references as public absolute URLs. Image files are not downloaded. |
| `tags` | string array | `[]` | Up to 20 tags added to each successful note's frontmatter. |
| `maxConcurrency` | integer | `4` | Number of pages fetched in parallel, from 1 to 10. |
| `requestTimeoutSecs` | integer | `30` | Per-request timeout, from 5 to 120 seconds. |
| `maxRequestRetries` | integer | `2` | Retries for transient network, 408, 429, and server failures, from 0 to 4. |

#### Example input

```json
{
  "startUrls": [
    { "url": "https://obsidian.md/help/web-clipper" },
    { "url": "https://developer.mozilla.org/en-US/docs/Web/HTML" }
  ],
  "maxItems": 2,
  "includeFrontmatter": true,
  "includeImages": false,
  "tags": ["research", "documentation"]
}
```

### What data does the Actor return?

| Field | Meaning |
| --- | --- |
| `url` | Normalized requested URL. |
| `finalUrl` | Final destination after validated redirects. |
| `canonicalUrl` | Page canonical URL, or the final URL when no canonical link exists. |
| `status` | `success` or `error`. |
| `statusCode` | Final successful HTTP status when available. |
| `title` | Extracted page title. |
| `description` | Meta description or readable excerpt. |
| `author` | Page author when declared or inferred by the readability parser. |
| `publishedAt` | Publication timestamp in ISO 8601 form when declared. |
| `language` | Declared page language. |
| `siteName` | Open Graph/readability site name or source hostname. |
| `filename` | Stable Markdown filename using a readable slug and canonical-URL hash. |
| `markdown` | Complete note, including frontmatter when enabled. |
| `contentMarkdown` | Markdown body without frontmatter. |
| `wordCount` | Approximate readable-content word count. |
| `links` | Unique absolute HTTP(S) links retained in the note. |
| `metadata` | Structured values used to generate frontmatter. |
| `error` | Per-URL failure reason. Present on failed records. |
| `scrapedAt` | Capture time in ISO 8601 form. |

Fields can be absent or null when a source page does not publish the corresponding metadata.

### Example Obsidian Markdown output

A successful Obsidian Docs capture produces a filename such as:

```text
introduction-to-obsidian-web-clipper-obsidian-help-96418c6a7b.md
```

Its `markdown` field begins like this:

```markdown
---
title: "Introduction to Obsidian Web Clipper - Obsidian Help"
source: "https://obsidian.md/help/web-clipper"
requested_url: "https://obsidian.md/help/web-clipper"
site: "obsidian.md"
language: "en"
clipped_at: "2026-08-30T12:00:00.000Z"
tags: ["obsidian", "documentation"]
---

Developer Documentation

Build plugins and themes for Obsidian.
```

Actual content and metadata reflect the live source at run time.

### Stable filename behavior

The filename combines:

- a human-readable slug from the extracted title; and
- the first ten hexadecimal characters of a SHA-256 hash of the canonical URL.

For the same canonical URL and title, repeated runs produce the same filename. The URL hash prevents two pages with the same title from colliding. If a publisher changes the title or canonical URL, the filename can change; retain `canonicalUrl` as the durable source identity in downstream systems.

### How much does it cost to convert websites to Obsidian Markdown?

The Actor uses pay-per-event pricing:

- **$0.005** when a run starts;
- **$0.00192 per successful note** at the Bronze tier;
- failed URL records have no item charge.

Example Bronze-tier totals:

| Successful notes | Estimated total |
| ---: | ---: |
| 1 | $0.00692 |
| 10 | $0.0242 |
| 25 | $0.0530 |
| 100 | $0.1970 |

Higher subscription tiers receive lower per-note rates shown in the Actor pricing panel. These examples assume one run and exclude any separate charges imposed by your own downstream services.

### Recurring Obsidian knowledge-base capture

Apify schedules can run the same input repeatedly. Stable filenames make the results suitable for an upsert workflow:

1. schedule this Actor;
2. read successful dataset rows;
3. write each `markdown` value to `filename` in your storage or vault sync layer;
4. overwrite the previous file when the filename matches;
5. retain `scrapedAt` if capture provenance matters.

The Actor captures current page content. It does not maintain history, calculate diffs, send change alerts, or write directly into a local Obsidian vault.

### Integration patterns

#### Save notes to cloud storage

Use Make, Zapier, or n8n to iterate over dataset rows where `status` is `success`, then create a text file named by `filename` with the value from `markdown`.

#### Commit notes to Git

A CI job can run the Actor through the API, download dataset items, write the note files into a repository, and commit changed files for an auditable research archive.

#### Feed a RAG pipeline

Use `contentMarkdown` as clean document text and keep `canonicalUrl`, `title`, and `scrapedAt` as retrieval metadata. Chunking, embeddings, vector storage, and answer generation are intentionally outside this Actor's scope.

#### Export links for another crawl

Use the `links` array to select follow-up URLs. The Actor processes only supplied URLs and does not automatically crawl discovered links.

### Run with the Apify API using cURL

Replace `YOUR_APIFY_TOKEN` with your token:

```bash
curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~website-to-obsidian-markdown/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls": [{"url": "https://obsidian.md/help/web-clipper"}],
    "maxItems": 1,
    "includeFrontmatter": true,
    "tags": ["obsidian"]
  }'
```

For larger batches, start an asynchronous run and poll its status instead of holding a synchronous request open.

### Run with JavaScript

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/website-to-obsidian-markdown').call({
  startUrls: [
    { url: 'https://developer.mozilla.org/en-US/docs/Web/HTML' },
  ],
  maxItems: 1,
  includeFrontmatter: true,
  tags: ['reference'],
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const item of items.filter((row) => row.status === 'success')) {
  console.log(item.filename, item.markdown.length);
}
```

### Run with Python

```python
import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("automation-lab/website-to-obsidian-markdown").call(
    run_input={
        "startUrls": [
            {"url": "https://developer.mozilla.org/en-US/docs/Web/HTTP"}
        ],
        "maxItems": 1,
        "includeFrontmatter": True,
        "tags": ["web-standards"],
    }
)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    if item["status"] == "success":
        print(item["filename"])
```

### Use the Actor through MCP

Add the Apify MCP server to Claude Code:

```bash
claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=automation-lab/website-to-obsidian-markdown"
```

#### Claude Desktop setup

Add this server object to your Claude Desktop MCP configuration:

```json
{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/website-to-obsidian-markdown"
    }
  }
}
```

#### Cursor setup

In Cursor MCP settings, add a remote HTTP server named `apify` with this URL:

```text
https://mcp.apify.com?tools=automation-lab/website-to-obsidian-markdown
```

#### VS Code setup

In VS Code's MCP server configuration, add the same remote HTTP endpoint under the server name `apify`. Authenticate through the Apify MCP flow when prompted.

Example prompts:

- “Convert this public documentation URL into an Obsidian note and return the filename and Markdown.”
- “Clip these three public research pages with the tags `research` and `sources`.”
- “Capture this page without image references and summarize which links were retained.”

### Reliability and retry behavior

The Actor retries only failures likely to be temporary: network errors, timeouts, HTTP 408, HTTP 429, and server errors. It uses bounded exponential backoff and never retries forever.

Every destination is checked before connection. Redirect destinations are checked again. Requests are pinned to a validated public IP while retaining the original hostname for TLS and HTTP routing, reducing DNS rebinding and server-side request forgery risk.

A mixed batch can succeed even if some pages fail. The run exits with failure when no supplied page can be converted, so automated workflows do not mistake an empty result for success.

### Limits and unsupported pages

The Actor is intentionally an anonymous public-page converter. It does not bypass:

- login walls or private pages;
- paywalls;
- CAPTCHA or anti-bot challenges;
- JavaScript-only pages that provide no useful server-rendered HTML;
- non-HTML files such as PDFs;
- local, private-network, link-local, or credential-bearing URLs;
- pages larger than the bounded HTML download limit.

The output may omit metadata that a page does not declare. Readability extraction favors main content and can exclude navigation, comments, sidebars, footers, and interactive widgets.

### Images and attachments

When `includeImages` is enabled, Markdown can contain absolute image references from the source page. The Actor does not download, copy, resize, or permanently host those images. A source may later change or remove them.

Disable image references when you need a text-only note, smaller downstream payload, or a self-managed attachment workflow.

### Responsible and legal use

Only process pages you are allowed to access and reuse. Review the source website's terms, robots policy, copyright, database rights, and applicable law. Keep concurrency conservative and do not use the Actor to evade authentication, technical restrictions, or access controls.

The Actor records source URLs to support attribution, but a source field does not itself grant permission to republish content. You are responsible for downstream storage, sharing, retention, and compliance.

### Troubleshooting

#### Why did a URL return an error record?

Check `error` for the specific reason. Common causes include HTTP 404, non-HTML content, a challenge page, a timeout, or insufficient readable content. Open the URL anonymously in a normal browser to confirm it is public, then retry only if the condition is temporary.

#### Why is navigation or a sidebar missing?

The converter intentionally extracts readable main content and removes common navigation and interactive elements. It is a note creator, not a pixel-perfect page archive.

#### Why is the author or publication date null?

Many pages do not publish those fields consistently. The Actor does not invent missing metadata.

#### Why did the filename change?

The publisher may have changed the page title or canonical URL. Use `canonicalUrl` for identity and implement an explicit rename policy if your vault must preserve a previous filename.

#### Can it crawl an entire website?

No. Supply the exact public pages you want to capture. This bounded behavior keeps scope, cost, and source selection under your control.

### FAQ

#### Does this install the Obsidian browser extension?

No. It provides a server-side conversion workflow with a similar capture job and structured dataset output. It does not install or control an Obsidian web clipper extension.

#### Does it write directly to my vault?

No. Download or integrate the dataset, then save `markdown` under `filename` in your own vault or sync destination.

#### Can I capture more than one page?

Yes. Add up to 100 URLs and set `maxItems` to the desired batch size.

#### Are failed pages charged as notes?

No. The item event is emitted only after a useful Markdown record is created. The one-time run-start event still applies.

#### Can I disable YAML frontmatter?

Yes. Set `includeFrontmatter` to `false`; `markdown` will contain only the content body, while structured metadata remains available in dataset fields.

### Related Automation Lab Actors

- [HTML Readability to Markdown Converter](https://apify.com/automation-lab/html-readability-markdown-converter) — convert one URL or supplied raw HTML into generic Markdown with a content hash; use this Obsidian Actor instead for batch URLs, YAML frontmatter, tags, and stable note filenames.
- [Article Content Scraper](https://apify.com/automation-lab/multi-site-article-content-extractor) — extract normalized article text, HTML, metadata, and links when Markdown note formatting is not required.
- [LLM Web Page Research Browser](https://apify.com/automation-lab/llm-web-page-research-browser) — produce citation-oriented page records for research workflows.
- [PDF to Structured Markdown Converter](https://apify.com/automation-lab/pdf-to-structured-markdown-converter) — convert supplied PDF files or URLs when the source is not an HTML webpage.

### Support

If a public server-rendered HTML page fails unexpectedly, include the run link, the non-sensitive input URL, and the per-URL error in an Apify Actor issue. Do not include tokens, cookies, private URLs, or confidential page content.

# Actor input Schema

## `startUrls` (type: `array`):

Public HTTP(S) pages to convert. Duplicate URLs are processed once; up to 100 URLs per run.

## `maxItems` (type: `integer`):

Maximum number of unique input pages to process.

## `includeFrontmatter` (type: `boolean`):

Add YAML frontmatter with title, source, author, publication date, site, language, capture time, and tags.

## `includeImages` (type: `boolean`):

Keep public image references in Markdown. The Actor does not download image files.

## `tags` (type: `array`):

Optional tags added to every generated note's frontmatter.

## `maxConcurrency` (type: `integer`):

Number of pages fetched in parallel. Lower this for sensitive websites.

## `requestTimeoutSecs` (type: `integer`):

Timeout in seconds for each page request.

## `maxRequestRetries` (type: `integer`):

Retries for transient network errors, rate limits, and server errors.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://obsidian.md/help/web-clipper"
    },
    {
      "url": "https://developer.mozilla.org/en-US/docs/Web/HTML"
    }
  ],
  "maxItems": 10,
  "includeFrontmatter": true,
  "includeImages": true,
  "tags": [],
  "maxConcurrency": 4,
  "requestTimeoutSecs": 30,
  "maxRequestRetries": 2
}
```

# Actor output Schema

## `overview` (type: `string`):

Default dataset view containing converted notes and diagnostic error rows.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://obsidian.md/help/web-clipper"
        },
        {
            "url": "https://developer.mozilla.org/en-US/docs/Web/HTML"
        }
    ],
    "maxItems": 10,
    "includeFrontmatter": true,
    "includeImages": true
};

// Run the Actor and wait for it to finish
const run = await client.actor("automation-lab/website-to-obsidian-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [
        { "url": "https://obsidian.md/help/web-clipper" },
        { "url": "https://developer.mozilla.org/en-US/docs/Web/HTML" },
    ],
    "maxItems": 10,
    "includeFrontmatter": True,
    "includeImages": True,
}

# Run the Actor and wait for it to finish
run = client.actor("automation-lab/website-to-obsidian-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://obsidian.md/help/web-clipper"
    },
    {
      "url": "https://developer.mozilla.org/en-US/docs/Web/HTML"
    }
  ],
  "maxItems": 10,
  "includeFrontmatter": true,
  "includeImages": true
}' |
apify call automation-lab/website-to-obsidian-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,automation-lab/website-to-obsidian-markdown"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/RtauUYOv3l2Bos5jr/builds/PCa56sf9lAVkXaOw7/openapi.json