Blog / Dated Content Crawler
Pricing
Pay per usage
Blog / Dated Content Crawler
Crawl an entire blog / knowledge base or filter to just the new content. Supporting relevant AI queries by filtering pages by date
Pricing
Pay per usage
Rating
5.0
(2)
Developer
Diarmuid
Actor stats
9
Bookmarked
135
Total users
13
Monthly active users
a month ago
Last modified
Categories
Share
This actor enables you to crawl blog / dated content websites. What this means is that you can filter the content by its publish date and only keep the content that is newer than a date you select. This is very useful for AI applications to avoid training your model or feeding your LLM with old / outdated / irrelevant data.
This is also useful for any application where you want to download data from websites such as documentation, help articles, or your knowledge base.
How it works
- Enter the url(s) (
startUrls) of the pages / site you want to crawl. - Optional: Enter a start date (
startDate) or more likely a "Relative" start date (relativeStartDate) to filter the content by. "Relative" means that you can enter a date like "1 month" or "2 years" and the crawler will calculate the date relative to the current date each time it runs. - Run the crawler
- The crawler will retrieve only the pages that are newer than the start date (
startDate) you entered or will retrieve all the pages if you don't enter a start date.
