Website Content Scraper
Pricing
$10.00 / 1,000 results
Website Content Scraper
Efficient scraper for extracting meaningful text from URLs found in a sitemap. Supports nested sitemaps, processes large datasets, and filters out boilerplate content. Ideal for SEO research, content analysis, and text mining, providing clean and structured text data for further processing.
Pricing
$10.00 / 1,000 results
Rating
0.0
(0)
Developer
Dolp
Actor stats
2
Bookmarked
20
Total users
20
Monthly active users
a year ago
Last modified
Categories
Share
🌐 Sitemap Text Extractor
A web scraping actor that extracts text content from URLs listed in a sitemap.
📝 Overview
This actor fetches all URLs from a given sitemap, including nested sitemaps, and extracts text content from each URL. The extracted text is then saved to an Apify dataset.
✨ Features
- 🔄 Recursive Sitemap Parsing: Handles nested sitemaps to fetch all URLs.
- 📄 Text Extraction: Extracts meaningful text content from web pages.
- 🚫 Static Asset Filtering: Excludes URLs pointing to static assets like images and PDFs.
- ⚠️ Error Handling: Logs warnings for failed page fetches and continues processing other URLs.
📋 Requirements
- ⚙️ Input Configuration: Provide a
sitemap_urlin the actor's input configuration. - 🌍 Publicly Accessible Sitemap: Ensure the sitemap is publicly accessible.
🚀 Usage
-
Input Configuration:
- Provide a
sitemap_urlin the actor's input configuration. - Ensure the sitemap is publicly accessible.
- Provide a
-
Running the Actor:
- Start the actor in Apify.
- The actor will fetch URLs from the sitemap, extract text content, and save the results to a dataset.
📤 Output
The actor saves the extracted text content to an Apify dataset. Each record in the dataset contains:
- 🔗 url: The URL from which the text was extracted.
- 📜 content: The extracted text content.
Example Output
💡 Example
To use this actor, simply provide the URL of the sitemap you want to process. For example, if you want to extract text from all pages listed in a sitemap located at https://example.com/sitemap.xml, you would enter this URL in the input configuration.
🎯 Use Cases
This actor is useful for extracting text data from websites for various purposes, such as:
- 📊 Data Collection: Gathering text content for analysis or processing.
- 🤖 AI Training: Feeding text data to AI models for training or fine-tuning.
- 📝 Content Summarization: Extracting key information from large volumes of text.
🤝 Contributing
Contributions are welcome! Feel free to report issues or suggest improvements.
