Obsidian Web Clipper Scraper

Turn supplied public web pages into clean, portable, Obsidian-ready Markdown notes.

This Obsidian web clipper Actor extracts readable page content, preserves useful headings and links, adds source metadata, and creates a stable .md filename for each URL. It is built for repeatable server-side capture: use it from Apify Console, a schedule, the API, Make, Zapier, n8n, or an MCP client without installing a browser extension.

What does Obsidian Web Clipper Scraper do?

Give the Actor one or more anonymous public HTTP(S) page URLs. For every page it can read, it returns:

• clean Markdown converted from the page's readable content;
• optional Obsidian YAML frontmatter;
• source, canonical URL, title, description, author, publication date, language, and site metadata when available;
• a stable filename based on the canonical URL;
• absolute links retained in the note;
• an extracted-link list for downstream processing;
• per-URL success or error status.

The default dataset contains one record per requested URL. Successful notes are charged as items. Error records are useful diagnostics and do not incur the item event.

Who is this web-to-Obsidian converter for?

Researchers

Capture source pages with citations and canonical URLs before organizing them in an Obsidian vault.

Developers and technical writers

Convert documentation pages into Markdown for an offline reference library, internal notes, or review workflows.

Analysts

Collect a bounded batch of public articles or reference pages in a consistent format for text analysis or knowledge-base ingestion.

Automation builders

Replace manual copy-and-paste with scheduled Apify runs and feed the resulting Markdown to cloud storage, Git, an ETL tool, or a custom Obsidian sync process.

Why use this instead of copying page text?

Manual copying commonly loses source metadata, resolves links inconsistently, includes navigation noise, and creates filenames that change from one capture to the next.

This Actor provides a repeatable data contract:

1. public destinations and redirects are validated;
2. readable content is selected from server-rendered HTML;
3. unsafe and noisy elements are removed;
4. relative links and image references become absolute URLs;
5. HTML becomes clean Markdown;
6. metadata becomes optional YAML frontmatter;
7. a deterministic canonical-URL hash keeps the filename stable across recurring runs.

The result is ready to write as a Markdown file, while the dataset fields remain available for integrations.

Getting started

1. Open the Actor input.
2. Add one or more public page URLs to Web page URLs.
3. Keep Include Obsidian frontmatter enabled for vault-ready notes.
4. Add optional shared tags such as research or documentation.
5. Choose whether to keep image references.
6. Click Start.
7. Open the default dataset.
8. Save each markdown value using its corresponding filename.

The prefilled Obsidian Docs and MDN URLs are real public pages and can be used for a first run.

Input parameters

Field	Type	Default	Description
startUrls	array	required	One to 100 anonymous public HTTP(S) page URLs. Duplicate normalized URLs are processed once.
maxItems	integer	10	Maximum number of unique supplied URLs to process, from 1 to 100.
includeFrontmatter	boolean	true	Include Obsidian-compatible YAML frontmatter before the Markdown body.
includeImages	boolean	true	Keep image references as public absolute URLs. Image files are not downloaded.
tags	string array	[]	Up to 20 tags added to each successful note's frontmatter.
maxConcurrency	integer	4	Number of pages fetched in parallel, from 1 to 10.
requestTimeoutSecs	integer	30	Per-request timeout, from 5 to 120 seconds.
maxRequestRetries	integer	2	Retries for transient network, 408, 429, and server failures, from 0 to 4.

Example input

{
  "startUrls": [
    { "url": "https://obsidian.md/help/web-clipper" },
    { "url": "https://developer.mozilla.org/en-US/docs/Web/HTML" }
  ],
  "maxItems": 2,
  "includeFrontmatter": true,
  "includeImages": false,
  "tags": ["research", "documentation"]
}

What data does the Actor return?

Field	Meaning
url	Normalized requested URL.
finalUrl	Final destination after validated redirects.
canonicalUrl	Page canonical URL, or the final URL when no canonical link exists.
status	success or error.
statusCode	Final successful HTTP status when available.
title	Extracted page title.
description	Meta description or readable excerpt.
author	Page author when declared or inferred by the readability parser.
publishedAt	Publication timestamp in ISO 8601 form when declared.
language	Declared page language.
siteName	Open Graph/readability site name or source hostname.
filename	Stable Markdown filename using a readable slug and canonical-URL hash.
markdown	Complete note, including frontmatter when enabled.
contentMarkdown	Markdown body without frontmatter.
wordCount	Approximate readable-content word count.
links	Unique absolute HTTP(S) links retained in the note.
metadata	Structured values used to generate frontmatter.
error	Per-URL failure reason. Present on failed records.
scrapedAt	Capture time in ISO 8601 form.

Fields can be absent or null when a source page does not publish the corresponding metadata.

Example Obsidian Markdown output

A successful Obsidian Docs capture produces a filename such as:

introduction-to-obsidian-web-clipper-obsidian-help-96418c6a7b.md

Its markdown field begins like this:

---
title: "Introduction to Obsidian Web Clipper - Obsidian Help"
source: "https://obsidian.md/help/web-clipper"
requested_url: "https://obsidian.md/help/web-clipper"
site: "obsidian.md"
language: "en"
clipped_at: "2026-08-30T12:00:00.000Z"
tags: ["obsidian", "documentation"]
---

Developer Documentation

Build plugins and themes for Obsidian.

Actual content and metadata reflect the live source at run time.

Stable filename behavior

The filename combines:

• a human-readable slug from the extracted title; and
• the first ten hexadecimal characters of a SHA-256 hash of the canonical URL.

For the same canonical URL and title, repeated runs produce the same filename. The URL hash prevents two pages with the same title from colliding. If a publisher changes the title or canonical URL, the filename can change; retain canonicalUrl as the durable source identity in downstream systems.

How much does it cost to convert websites to Obsidian Markdown?

The Actor uses pay-per-event pricing:

• $0.005 when a run starts;
• $0.00192 per successful note at the Bronze tier;
• failed URL records have no item charge.

Example Bronze-tier totals:

Successful notes	Estimated total
1	$0.00692
10	$0.0242
25	$0.0530
100	$0.1970

Higher subscription tiers receive lower per-note rates shown in the Actor pricing panel. These examples assume one run and exclude any separate charges imposed by your own downstream services.

Recurring Obsidian knowledge-base capture

Apify schedules can run the same input repeatedly. Stable filenames make the results suitable for an upsert workflow:

1. schedule this Actor;
2. read successful dataset rows;
3. write each markdown value to filename in your storage or vault sync layer;
4. overwrite the previous file when the filename matches;
5. retain scrapedAt if capture provenance matters.

The Actor captures current page content. It does not maintain history, calculate diffs, send change alerts, or write directly into a local Obsidian vault.

Integration patterns

Save notes to cloud storage

Use Make, Zapier, or n8n to iterate over dataset rows where status is success, then create a text file named by filename with the value from markdown.

Commit notes to Git

A CI job can run the Actor through the API, download dataset items, write the note files into a repository, and commit changed files for an auditable research archive.

Feed a RAG pipeline

Use contentMarkdown as clean document text and keep canonicalUrl, title, and scrapedAt as retrieval metadata. Chunking, embeddings, vector storage, and answer generation are intentionally outside this Actor's scope.

Export links for another crawl

Use the links array to select follow-up URLs. The Actor processes only supplied URLs and does not automatically crawl discovered links.

Run with the Apify API using cURL

Replace YOUR_APIFY_TOKEN with your token:

curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~website-to-obsidian-markdown/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls": [{"url": "https://obsidian.md/help/web-clipper"}],
    "maxItems": 1,
    "includeFrontmatter": true,
    "tags": ["obsidian"]
  }'

For larger batches, start an asynchronous run and poll its status instead of holding a synchronous request open.

Run with JavaScript

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/website-to-obsidian-markdown').call({
  startUrls: [
    { url: 'https://developer.mozilla.org/en-US/docs/Web/HTML' },
  ],
  maxItems: 1,
  includeFrontmatter: true,
  tags: ['reference'],
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const item of items.filter((row) => row.status === 'success')) {
  console.log(item.filename, item.markdown.length);
}

Run with Python

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("automation-lab/website-to-obsidian-markdown").call(
    run_input={
        "startUrls": [
            {"url": "https://developer.mozilla.org/en-US/docs/Web/HTTP"}
        ],
        "maxItems": 1,
        "includeFrontmatter": True,
        "tags": ["web-standards"],
    }
)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    if item["status"] == "success":
        print(item["filename"])

Use the Actor through MCP

Add the Apify MCP server to Claude Code:

claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=automation-lab/website-to-obsidian-markdown"

Claude Desktop setup

Add this server object to your Claude Desktop MCP configuration:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/website-to-obsidian-markdown"
    }
  }
}

Cursor setup

In Cursor MCP settings, add a remote HTTP server named apify with this URL:

https://mcp.apify.com?tools=automation-lab/website-to-obsidian-markdown

VS Code setup

In VS Code's MCP server configuration, add the same remote HTTP endpoint under the server name apify. Authenticate through the Apify MCP flow when prompted.

Example prompts:

• “Convert this public documentation URL into an Obsidian note and return the filename and Markdown.”
• “Clip these three public research pages with the tags research and sources.”
• “Capture this page without image references and summarize which links were retained.”

Reliability and retry behavior

The Actor retries only failures likely to be temporary: network errors, timeouts, HTTP 408, HTTP 429, and server errors. It uses bounded exponential backoff and never retries forever.

Every destination is checked before connection. Redirect destinations are checked again. Requests are pinned to a validated public IP while retaining the original hostname for TLS and HTTP routing, reducing DNS rebinding and server-side request forgery risk.

A mixed batch can succeed even if some pages fail. The run exits with failure when no supplied page can be converted, so automated workflows do not mistake an empty result for success.

Limits and unsupported pages

The Actor is intentionally an anonymous public-page converter. It does not bypass:

• login walls or private pages;
• paywalls;
• CAPTCHA or anti-bot challenges;
• JavaScript-only pages that provide no useful server-rendered HTML;
• non-HTML files such as PDFs;
• local, private-network, link-local, or credential-bearing URLs;
• pages larger than the bounded HTML download limit.

The output may omit metadata that a page does not declare. Readability extraction favors main content and can exclude navigation, comments, sidebars, footers, and interactive widgets.

Images and attachments

When includeImages is enabled, Markdown can contain absolute image references from the source page. The Actor does not download, copy, resize, or permanently host those images. A source may later change or remove them.

Disable image references when you need a text-only note, smaller downstream payload, or a self-managed attachment workflow.

Responsible and legal use

Only process pages you are allowed to access and reuse. Review the source website's terms, robots policy, copyright, database rights, and applicable law. Keep concurrency conservative and do not use the Actor to evade authentication, technical restrictions, or access controls.

The Actor records source URLs to support attribution, but a source field does not itself grant permission to republish content. You are responsible for downstream storage, sharing, retention, and compliance.

Troubleshooting

Why did a URL return an error record?

Check error for the specific reason. Common causes include HTTP 404, non-HTML content, a challenge page, a timeout, or insufficient readable content. Open the URL anonymously in a normal browser to confirm it is public, then retry only if the condition is temporary.

Why is navigation or a sidebar missing?

The converter intentionally extracts readable main content and removes common navigation and interactive elements. It is a note creator, not a pixel-perfect page archive.

Why is the author or publication date null?

Many pages do not publish those fields consistently. The Actor does not invent missing metadata.

Why did the filename change?

The publisher may have changed the page title or canonical URL. Use canonicalUrl for identity and implement an explicit rename policy if your vault must preserve a previous filename.

Can it crawl an entire website?

No. Supply the exact public pages you want to capture. This bounded behavior keeps scope, cost, and source selection under your control.

FAQ

Does this install the Obsidian browser extension?

No. It provides a server-side conversion workflow with a similar capture job and structured dataset output. It does not install or control an Obsidian web clipper extension.

Does it write directly to my vault?

No. Download or integrate the dataset, then save markdown under filename in your own vault or sync destination.

Can I capture more than one page?

Yes. Add up to 100 URLs and set maxItems to the desired batch size.

Are failed pages charged as notes?

No. The item event is emitted only after a useful Markdown record is created. The one-time run-start event still applies.

Can I disable YAML frontmatter?

Yes. Set includeFrontmatter to false; markdown will contain only the content body, while structured metadata remains available in dataset fields.

Related Automation Lab Actors

• HTML Readability to Markdown Converter — convert one URL or supplied raw HTML into generic Markdown with a content hash; use this Obsidian Actor instead for batch URLs, YAML frontmatter, tags, and stable note filenames.
• Article Content Scraper — extract normalized article text, HTML, metadata, and links when Markdown note formatting is not required.
• LLM Web Page Research Browser — produce citation-oriented page records for research workflows.
• PDF to Structured Markdown Converter — convert supplied PDF files or URLs when the source is not an HTML webpage.

Support

If a public server-rendered HTML page fails unexpectedly, include the run link, the non-sensitive input URL, and the per-URL error in an Apify Actor issue. Do not include tokens, cookies, private URLs, or confidential page content.