Website Content to Markdown for LLM Training
Try for free
2 hours trial then $19.99/month - No credit card required now
Go to Store
Website Content to Markdown for LLM Training
easyapi/website-content-to-markdown-for-llm-training
Try for free
2 hours trial then $19.99/month - No credit card required now
🚀 Transform web content into clean, LLM-ready Markdown! 📘 Scrape multiple pages, extract main content, and convert to Markdown format. Perfect for AI researchers, data scientists, and LLM developers. Fast, efficient, and customizable. Supercharge your AI training data today! 🌐📝🧠
Developer
EasyApiMaintained by Community
Actor Metrics
22 monthly users
5.0 / 5 (1)
No bookmarks yet
>99% runs succeeded
Created in Jan 2025
Modified 8 days ago
Categories
📘 Website Content to Markdown Scraper for LLM Training
This powerful Apify Actor transforms web content into clean, readable Markdown format, perfect for training Large Language Models (LLMs). It's an essential tool for AI researchers, data scientists, and developers working on natural language processing tasks.
✨ Features
- 🌐 Scrape content from multiple web pages
- 📝 Convert HTML to clean Markdown format
- 🧠 Generate high-quality training data for LLMs
- 🔍 Intelligent main content extraction
- 🚀 Fast and efficient with concurrent scraping
- 🕵️♂️ Stealth mode to avoid detection
📥 Input
Configure your scraping job with these options:
urls: List of URLs to start scraping from
📤 Output
For each scraped page, you'll get:
- 🔗 URL of the page
- 📄 Main content in Markdown format, ideal for LLM training
💡 Use Cases
- 🤖 LLM Training: Prepare web content as high-quality training data for language models
- 📚 Content Aggregation: Collect articles and blog posts for research or curation
- 📊 Web Analysis: Extract text content for sentiment analysis or topic modeling
- 📑 Documentation: Convert web-based documentation into Markdown for easy integration
- 🔍 SEO Analysis: Extract and analyze content from competitor websites
🚀 Getting Started
- Set your input parameters in the Apify console or via API
- Run the Actor and watch as it transforms web content into Markdown
- Access your results in JSON format, with Markdown content ready for LLM training or further processing
🆘 Support
If you encounter any issues or have questions, please reach out through Apify's support channels.
Transform web content into clean, LLM-ready Markdown with just a few clicks! 🚀📝🧠
Input Example
A full explanation of an input example in JSON.
1{ 2 "urls": [ 3 "https://apify.com", 4 "https://www.google.com" 5 ] 6}
Output sample
The results will be wrapped into a dataset which you can always find in the Storage tab. Here's an excerpt from the data you'd get if you apply the input parameters above:
And here is the same data but in JSON. You can choose in which format to download your data: JSON, JSONL, Excel spreadsheet, HTML table, CSV, or XML.
1[ 2 { 3 "url": "https://apify.com", 4 "markdown": "# Apify: Full-stack web scraping and data extraction platform\nApify is the largest ecosystem where developers build, deploy, and publish data extraction and web automation tools. We call them Actors.\n\n[\n\n\n\n### TikTok Data Extractor\n\nclockworks/free-tiktok-scraper\n\nExtract data about videos, users, and channels based on hashtags or scrape full user profiles including posts, total likes, name, nickname, numbers of comments, shares, followers, following, and more.\n\n](https://apify.com/clockworks/free-tiktok-scraper)[\n\n\n\n### Google Maps Extractor\n\ncompass/google-maps-extractor\n\nExtract data from hundreds of places fast. Scrape Google Maps by keyword, category, location, URLs & other filters. Get addresses, contact info, opening hours, popular times, prices, menus & more. Export scraped data, run the scraper via API, schedule and monitor runs, or integrate with other tools.\n\n\n\nCompass\n\n\n\n](https://apify.com/compass/google-maps-extractor)[\n\n\n\n### Instagram Scraper\n\napify/instagram-scraper\n\nScrape and download Instagram posts, profiles, places, hashtags, photos, and comments. Get data from Instagram using one or more Instagram URLs or search queries. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.\n\n\n\nApify\n\n\n\n](https://apify.com/apify/instagram-scraper)[\n\n\n\n### Website Content Crawler\n\napify/website-content-crawler\n\nCrawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.\n\n\n\nApify\n\n\n\n](https://apify.com/apify/website-content-crawler)[\n\n\n\n### Amazon Scraper\n\njunglee/free-amazon-product-scraper\n\nGets you product data from Amazon. Unofficial API. Scrapes and downloads product information without using the Amazon API, including reviews, prices, descriptions, and ASIN.\n\n\n\nJunglee\n\n\n\n](https://apify.com/junglee/free-amazon-product-scraper)[\n\n\n\n### Build your own Actor\n\nyou/new-idea\n\nApify gives you all the tools and documentation you need to build reliable scrapers. Fast.\n\n\n\nYou? 🫵\n\n\n\n](https://apify.com/templates)\n\n[\n\n\n\n### TikTok Data Extractor\n\nclockworks/free-tiktok-scraper\n\nExtract data about videos, users, and channels based on hashtags or scrape full user profiles including posts, total likes, name, nickname, numbers of comments, shares, followers, following, and more.\n\n\n\nClockworks\n\n\n\n](https://apify.com/clockworks/free-tiktok-scraper)\n\n[\n\n\n\n### Google Maps Extractor\n\ncompass/google-maps-extractor\n\nExtract data from hundreds of places fast. Scrape Google Maps by keyword, category, location, URLs & other filters. Get addresses, contact info, opening hours, popular times, prices, menus & more. Export scraped data, run the scraper via API, schedule and monitor runs, or integrate with other tools.\n\n\n\nCompass\n\n\n\n](https://apify.com/compass/google-maps-extractor)\n\n[Browse 3,000+ Actors](https://apify.com/store)\n\nTrusted by global technology leaders\n\nNot just a web scraping API\n---------------------------\n\nEasily integrate \nZapierany appGitHubGoogle SheetsPineconeAirbyteKeboolaGoogle DriveSlackZapier \nwith Actors\n--------------------------------------------------------------------------------------------------------------\n\nBuild reliable web scrapers. Fast.\n----------------------------------\n\n### We love open source\n\nApify works great with both Python and JavaScript, as well as Playwright, Puppeteer, Selenium, Scrapy, and Crawlee - our own web crawling and browser automation library.\n\n[](https://crawlee.dev/)\n\n```\n1import { PuppeteerCrawler, Dataset } from \"crawlee\";\n2\n3const crawler = new PuppeteerCrawler({\n4 async requestHandler({ request, page, enqueueLinks }) {\n5 await Dataset.pushData({\n6 url: request.url,\n7 title: await page.title(),\n8 });\n9 await enqueueLinks();\n10 },\n11});\n12\n13await crawler.run([\"https://crawlee.dev\"]);\n```\n\n\nPublish Actors. Get paid.\n-------------------------\n\n### Reach thousands of new customers\n\nBuilding and running a SaaS is hard. Building an Actor and selling it on Apify Store is 10x easier. Get visitors from day one.\n\n\n\n#### No upfront costs\n\nPublishing your Actor is free of charge—the customers pay for the computing resources. New creators get $500 free platform credits.\n\n#### Rely on Apify infra\n\nActors scale automatically as you gain new users. You don’t need to worry about compute, storage, proxies, or authentication.\n\n#### Billing is on us\n\nHandling payments, taxes, and invoicing is a painful part of running a SaaS. Apify does all that and sends you a net payout every month.", 5 "timestamp": "2025-01-10T07:10:53.476Z" 6 }, 7 { 8 "url": "https://apify.com/actors", 9 "title": "Actors - fast and easy scraping in the cloud · Apify", 10 "markdown": "Actors are serverless cloud programs that run on the Apify platform and do computing jobs. They are called Actors because, like human actors, they perform actions based on a script.\n\n\n\n### Long-running serverless jobs[](#long-running-serverless-jobs)\n\nApify Actors can perform time-consuming jobs that are longer than the lifespan of a single HTTP transaction.\n\n\n\n### Publish your Actor[](#publish-your-actor)\n\nJoin hundreds of developers who share their Actors on Apify Store and earn money from coding.\n\n[Go to Apify Store](/store)\n\n\n\n### Auto-generated user interface[](#auto-generated-user-interface)\n\nActors can easily define a user interface for their input configuration. Take advantage of lower-level features and settings, or run Actors using our API.\n\n[Learn about Input Schema](https://docs.apify.com/academy/deploying-your-code/input-schema)\n\n\n\n\n\nHost code anywhere\n\nEdit your code on our platform, fetch from a Git repository, or push from your machine.\n\n\n\nDocker support\n\nActors run inside Docker containers on Apify servers. Use a custom Dockerfile.\n\n\n\nReady for scale\n\nRun as many Actors as you need. The Apify platform provisions the necessary resources.\n\n\n\nCustom memory and CPU\n\nAssign each Actor any RAM volume needed. CPU share is allocated automatically.\n\n\n\nCommand-line tool\n\nDevelop and test your Actors locally, push them to the Apify platform when you're ready.\n\n\n\nLogging\n\nView and download logs to debug your code and monitor performance on production.\n\n\n\nActorize your Scrapy spiders[](#actorize-your-scrapy-spiders)\n-------------------------------------------------------------\n\nDeploy your Scrapy code to the cloud with just a few commands. Turn your Scrapy projects into Actors, run, schedule, monitor and monetize them.\n\n[Learn more](/run-scrapy-in-cloud)" 11 }, 12 ... 13]
Related Actors
- 📄 Article Content Extractor - Extract clean article content and metadata from any web page with structured output.
- 🔍 Keyword Density Checker - Analyze webpage content for keyword density and frequency with precise calculations.
- 🤖 AI-powered Search - Transform search queries into structured AI-powered summaries with references.
- 📚 arXiv Search Scraper - Extract comprehensive research paper data with detailed metadata.
- 🔬 Nature Search Results Scraper - Extract research article data with comprehensive metadata.
- 📚 Medium Posts Search Scraper - Extract detailed article data from Medium's search results.
- 📚 Substack Posts Scraper - Scrape Substack posts and articles with comprehensive content data.
- 🌐 URL Metadata Crawler - Extract comprehensive metadata from web pages including meta tags and Open Graph data.
- 📝 YouTube Description Extractor - Extract complete descriptions from YouTube videos automatically.
- 📚 WikiHow Article Scraper - Scrape WikiHow articles with detailed step-by-step content.
- 🔍 Google News Scraper - Collect up to 5000 news articles with flexible search options.
- 📚 PubMed Search Scraper - Scrape research papers and academic articles with comprehensive metadata.
- 📚 Goodreads Book Scraper - Extract comprehensive book data and content from Goodreads.
- 📚 Medium User Posts Scraper - Extract detailed post data from Medium user profiles.
- 📚 Substack Publications Scraper - Scrape detailed publication information from Substack.
