News Source Crawler

Turn a news website into a clean, searchable dataset of articles.

Give the Actor a publication homepage or section URL and it finds public articles, extracts the important details, and returns them in a format ready for research, monitoring, reporting, or analysis.

What you can do with it

• Monitor coverage of a company, person, product, or topic.
• Follow competitors and industry publications.
• Build a research-ready archive of articles from a trusted source.
• Find content that mentions specific keywords or phrases.
• Collect article text for trend analysis, media intelligence, or AI workflows.

Key capabilities

• Crawl a whole news source or just one section, such as /business or /technology.
• Find articles even when they are not all visible on the first page.
• Filter results with simple or advanced searches.
• Search article headlines, full text, or both.
• Exclude short teaser pages and keep substantive articles.
• Extract headlines, authors, dates, full text, featured images, keywords, and summaries.
• Choose the article language and control the number of articles returned.
• Use a proxy configuration when a publisher restricts direct access.

Get started

1. Enter the URL of a news website or section.
2. Choose how many articles you want.
3. Optionally add keywords, a minimum article length, or a language.
4. Start the Actor and explore the results in the Dataset tab.

For example, to find recent AI-related content from a technology publication:

{
  "url": "https://example.com/technology",
  "maxArticles": 100,
  "keywordSearch": "AI OR artificial intelligence",
  "minWordCount": 300
}

Search examples

Goal	Search query
Find either topic	AI OR automation
Find articles that mention both terms	climate AND policy
Find a phrase	"electric vehicles"
Exclude a topic	technology NOT cryptocurrency
Combine conditions	AI AND (agents OR automation) NOT crypto

What you get

Each result is one article with fields such as:

{
  "articleTitle": "How AI is changing media research",
  "articleURL": "https://example.com/technology/ai-media-research",
  "articleAuthors": "Jane Smith",
  "articlePublishDate": "2026-07-13T06:30:00+00:00",
  "articleText": "Full article text...",
  "articleWordCount": 842,
  "articleKeywords": "artificial intelligence, media, research",
  "articleSummary": "A concise overview of the article.",
  "articleTopImage": "https://example.com/images/ai-media.jpg"
}

Download your dataset as JSON, CSV, Excel, XML, or other supported formats, or use it in an automated workflow.

Tips for better results

• Use a section URL when you only need one beat, for example /finance or /world.
• Set a larger article limit when using a narrow keyword search, since some discovered articles may not match.
• Keep the default minimum word count to avoid navigation pages and short teasers. Set it to 0 only when short articles matter.
• If a site returns few or no articles, try a proxy configuration and lower the number of concurrent requests.

Frequently asked questions

Does it access paywalled or private content?

No. It extracts publicly accessible article pages only.

Why are some articles missing?

Some publishers limit access, use heavily interactive pages, or do not expose every article publicly. Keyword, length, and source-section filters can also intentionally narrow the result set.

Can I run it regularly?

Yes. Schedule recurring runs to build a continuously updated news feed or monitoring dataset.

Can I use the data in another tool?

Yes. The structured dataset is suitable for spreadsheets, dashboards, automations, analysis tools, and AI workflows.