AI Web Content Crawler 🤖

Crawl and extract clean, structured content from any website using AI power

Transform messy web pages into clean, structured content with AI-powered crawling and extraction. Built with NVIDIA NIM and advanced deep learning to remove ads, navigation, and clutter while preserving exactly what you need.

🚀 What Makes This Different

Unlike traditional web crawlers that just grab everything, our AI intelligently filters content based on your specific needs. Whether you need blog articles, product details, or technical documentation, you get exactly what matters—nothing more, nothing less.

✨ Key Features

• 🧠 AI-Powered Intelligence: Uses NVIDIA's deepseek-ai/deepseek-v3.1 model for human-level content understanding
• 🎯 Precision Extraction: Specify exactly what content you want and get laser-focused results
• ⚡ Blazing Fast: Process multiple URLs simultaneously with intelligent caching
• 🧹 Clean Output: Removes ads, navigation, popups, and other web clutter automatically
• 📝 Markdown Ready: Perfectly formatted markdown suitable for blogs, documentation, or data analysis
• 🔄 Batch Crawling: Handle hundreds of URLs efficiently with configurable concurrency

🎯 Perfect For

• Content Creators: Crawl and extract research from multiple sources
• Data Analysts: Get clean datasets from web sources
• SEO Specialists: Analyze competitor content structure
• Developers: Build knowledge bases from documentation
• Researchers: Collect academic content for analysis
• Marketers: Crawl product descriptions and reviews

🚀 30-Second Quick Start

For Apify Proxy (Recommended to avoid blocking):

1. Get Your Apify Token: Visit Apify Console Integrations and copy your API token
2. Set Environment Variable:

   $env:APIFY_TOKEN="your_token_here"
   

3. Run the Crawler:

   $apify run --input-file test-input.json
   

Without Proxy (may get blocked on some sites):

1. Paste URLs: Add any website URLs you want to crawl and extract content from
2. Tell AI What You Want: Describe what content to extract (articles, products, documentation, etc.)
3. Get Clean Results: Receive perfectly structured content in markdown format

🛠️ Input Options

Option	Description	Example
Website URLs	Any web page you want to crawl and extract content from	https://example.com/blog/article
Extraction Instructions	Tell the AI what specific content you need	"Extract the main article content and author information"
Crawling Speed	Control how fast to crawl multiple URLs	1-10 concurrent requests
Custom Headers	Add authentication or specific headers for restricted sites	User-Agent, Authorization, etc.
Custom API Key	Optional: Provide your own NVIDIA NIM API key	(Leave empty for built-in service)
Proxy Configuration	Configure proxies to avoid IP blocking	Use Apify Proxy for better reliability

📊 Output Structure

Each crawled page provides:

• Clean Content: Perfectly formatted markdown text
• Page Title: The actual page title
• All Links: Both internal and external links found
• Media Files: Images and videos with their URLs
• Extraction Status: Success/failure with detailed error messages

⚡ Advanced Use Cases

Content Marketing

Crawl competitor blog posts, analyze content structure, and create better versions

Academic Research

Crawl research papers, articles, and documentation for analysis and citation

E-commerce Analysis

Crawl product descriptions, reviews, and specifications from multiple sites

Technical Documentation

Crawl scattered documentation into structured, searchable knowledge bases

News Aggregation

Crawl articles from multiple news sources for sentiment analysis and trends

🎨 Sample Instructions

For Blog Articles:

Crawl and extract the main blog post content, including:
- Article title and subtitle
- Author name and bio
- Publication date
- Main article body
- Related links mentioned in content
Remove navigation, ads, comments, and sidebar content

For Product Pages:

Extract product information including:
- Product name and brand
- Price and currency
- Description
- Specifications
- Customer reviews summary
Ignore navigation, related products, and promotional content

For Technical Documentation:

Extract technical documentation content:
- API endpoints and parameters
- Code examples and snippets
- Configuration instructions
- Step-by-step guides
Preserve code formatting and technical accuracy

💡 Pro Tips

• Be Specific: Detailed instructions yield better results
• Start Small: Test with 2-3 URLs before processing large batches
• Use Categories: Group similar URLs together for consistent extraction
• Monitor Results: Adjust instructions based on initial output quality

🔧 Technical Specs

• AI Model: NVIDIA deepseek-ai/deepseek-v3.1
• Processing: Concurrent URL processing with rate limiting
• Output Format: Markdown with metadata
• Compatibility: Works with any website accessible via HTTP/HTTPS
• Rate Limits: Configurable concurrency (1-10 URLs simultaneously)
• Proxy Support: Full Apify Proxy integration for reliable scraping

🆘 Support & Documentation

Need help getting started? Check out our comprehensive ./DEVELOPMENT.md for technical details, advanced configuration, and troubleshooting tips.

---

Ready to extract clean content from any website? Get started now and transform your web data extraction workflow with AI precision.