SourceLens: Website to Markdown & JSON for AI

Turn any public webpage into clean, LLM-ready Markdown or structured JSON in seconds. SourceLens strips menus, footers, cookie banners and scripts, keeps the content that matters, and hands it back in a format your AI agent, RAG pipeline or spreadsheet can use straight away.

You only pay for pages that succeed. Failed, blocked or empty pages are returned as free error rows, so you can see exactly what happened without paying for it.

Built and maintained by ONSQ Enterprises.

🤖 Want specific fields without writing CSS selectors? Try SourceLens AI. Describe what you want in plain English, bring your own OpenAI or Claude key, and get JSON where every value is backed by a quote from the page.

What can SourceLens do?

• 📝 Clean Markdown. It finds the main content of each page and converts headings, lists, tables and links into tidy Markdown. Navigation, footers, forms and scripts are removed.
• 🎯 Structured JSON with CSS selectors. Name the fields you want, point each one at a CSS selector, and get one JSON record per page. Each value comes with the matching page text as evidence.
• 🕸️ Same-site crawling. Start from a homepage or a docs index and follow internal links up to the depth and page count you choose. Filter links with include and exclude patterns.
• 🧠 Extra metadata for free. Every page includes its title, meta description, language, canonical URL, word count and any JSON-LD structured data the site publishes, such as products, articles and organizations.
• 🛡️ Polite and safe. It respects robots.txt by default, retries temporary errors once, and never visits private or local network addresses.
• ⚡ Fast and cheap. Plain HTTP requests with no browser, so runs finish quickly and stay inexpensive.

Who is it for?

• AI and LLM developers feeding websites into RAG, vector databases, fine-tuning sets or chatbots.
• AI agents that need to read a webpage and get clean text back through the Apify API or MCP.
• Marketers and analysts pulling prices, headlines, product names or links from many pages into a spreadsheet.
• Researchers archiving articles and documentation as Markdown.

How to use it

1. Click Try for free.
2. Paste one or more URLs. You can leave off https://.
3. Choose Clean Markdown or Structured JSON from CSS selectors.
4. Optionally set a Crawl depth to follow links on the same website.
5. Click Start, then download your results as JSON, CSV, Excel or HTML, or read them through the API.

Input examples

Markdown for a list of pages

{
  "urls": ["https://www.python.org/about/", "en.wikipedia.org/wiki/Web_scraping"],
  "mode": "markdown"
}

Crawl a documentation site

{
  "urls": ["https://docs.example.com/"],
  "mode": "markdown",
  "maxCrawlDepth": 2,
  "maxCrawlPages": 200,
  "includeUrlPattern": "/docs/",
  "excludeUrlPattern": "/(changelog|tags)/"
}

Structured JSON with CSS selectors

{
  "urls": ["https://news.ycombinator.com/"],
  "mode": "selectors",
  "fields": [
    { "name": "stories", "selector": ".titleline > a", "multiple": true },
    { "name": "storyLinks", "selector": ".titleline > a", "attribute": "href", "multiple": true }
  ]
}

Each field takes a name, a CSS selector, an optional attribute and an optional multiple. The attribute is text by default. You can also use html, href, src or any other attribute name. Set multiple to true to get every match as a list.

Setting	What it does	Default
urls	Pages to extract, up to 1,000 per run	required
mode	markdown or selectors	markdown
fields	Fields to extract in selectors mode	none
maxCrawlDepth	How many link levels to follow on the same site. 0 means no crawling	0
maxCrawlPages	Most extra pages crawling may add	100
includeUrlPattern	Only follow links matching this regular expression	none
excludeUrlPattern	Never follow links matching this regular expression	none
maxChars	Most Markdown characters kept per page	100,000
concurrency	Pages fetched at the same time	5
timeoutSeconds	Time limit per page	30
respectRobotsTxt	Skip pages that robots.txt disallows	true

Output

You get one dataset row per page. A Markdown result looks like this:

{
  "url": "https://www.python.org/about/",
  "status": "ok",
  "mode": "markdown",
  "title": "About Python™ | Python.org",
  "description": "The official home of the Python Programming Language",
  "language": "en",
  "wordCount": 331,
  "markdown": "## Getting Started\n\nPython can be easy to pick up whether you're a first time programmer or you're experienced with other languages...\n\n- [Beginner's Guide, Programmers](https://wiki.python.org/moin/BeginnersGuide/Programmers)\n...",
  "jsonLd": [{ "@context": "https://schema.org", "@type": "WebSite", "url": "https://www.python.org/" }],
  "truncated": false,
  "depth": 0,
  "statusCode": 200,
  "billed": true
}

A selectors result has a data object with your fields and an evidence object showing the page text each value came from. Pages that fail have "status": "error", an error message and "billed": false.

A run summary with counts of successful, failed and skipped pages is saved in the key-value store as SUMMARY.

How much does it cost?

You pay a small fee for each page that is extracted successfully. Markdown pages and selector pages have separate prices, which are shown on the Pricing tab. There are no proxy or browser costs.

• Failed pages are free. That includes pages that time out, are blocked, are disallowed by robots.txt, are empty, or match none of your selectors.
• Your spending limit is respected. If you set a maximum cost per run, SourceLens stops cleanly when it is reached and keeps everything extracted so far.

Use it from code or an AI agent

Call SourceLens through the Apify API from any language, or use the Apify Python and JavaScript clients:

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("onsq/sourcelens").call(run_input={"urls": ["https://www.python.org/about/"]})
for page in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(page["title"], page["wordCount"])

AI agents can also call SourceLens through the Apify MCP server. It works with Make, Zapier, n8n, LangChain, LlamaIndex and Google Sheets through Apify integrations.

Limits and good to know

• SourceLens reads the HTML a server returns. It does not run JavaScript, so pages that only build their content in the browser may come back nearly empty.
• Sites with strong bot protection, such as some large news sites, may refuse the request. Those pages come back as free error rows.
• It does not log in, solve CAPTCHAs or read PDFs and images.
• Crawling always stays on the same website as the start URL. www. and non-www. count as the same site.

FAQ

Is it legal to scrape websites? SourceLens only fetches public pages and respects robots.txt by default. Results can still contain personal or copyrighted data. Make sure you have a legitimate reason to collect it, follow each website's terms, and consult a lawyer if you are unsure.

Why did a page return an error? The error field explains it. Common reasons are HTTP 403 (the site blocks bots), Disallowed by robots.txt, or a timeout. You are not charged for these pages.

Can I crawl a whole website? Yes, within limits. Set a crawl depth and a page limit of up to 5,000 extra pages per run.

Can it pull specific facts without selectors? Yes, with SourceLens AI. You list the fields in plain English, it uses your own OpenAI or Claude key, and it checks every value against a quote from the page. It can also turn a product grid or job board into one row per item.

Something is broken or missing. Open an issue on the Issues tab. Feature requests are welcome too.