# News Scraper - Keyword News Search & Monitoring API (`tidytools/news-scraper`) Actor Search 250+ news publishers by keyword in one run: headline, link, time, publisher, description, keywords. Country, language and topic filters, your own sites and feeds, duplicate stories collapsed, only-new alerts. $1 per 1,000 articles. - **URL**: https://apify.com/tidytools/news-scraper.md - **Developed by:** [Yukai Lin](https://apify.com/tidytools) (community) - **Categories:** News, Automation, AI - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per event This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ### What does News Scraper do? **Enter keywords (and optionally countries, languages, topics or your own sites), get one row per news article with headline, link, publisher, time and the publisher's own description and keywords, at $1 per 1,000 articles.** Article page details cost $0.50 per 1,000 and are optional. Filtered-out articles, duplicate stories, articles returned by earlier runs, unreadable sources and invalid inputs are free. No start fee. It **searches the latest news of 250+ publishers by keyword in one run** and returns one clean row per article: headline, link, publisher, publication time, country, language, the searches it matched, plus the description, news keywords, categories, author and image the publisher provides. - ๐Ÿ”Ž **Keyword search with a real query language**: all words must appear, `"exact phrase"`, `-exclude`, `OR`, `prefix*`; several searches per run, each article lists which ones it matched. - ๐ŸŒ **254 publishers in 47 countries and 22 languages** (Reuters-style wires excepted, see Limitations): BBC, CNN, The New York Times, Bloomberg, Forbes, TechCrunch, The Verge, Der Spiegel, Le Monde, El Paรญs, NHK, The Times of India, Straits Times, ไธญๅคฎ็คพ and many more. Filter by country, language and topic (technology, business, finance, crypto, science, health, sports, press releases...). - โž• **Your own sources**: paste any site, section page, RSS / Atom feed or Google News sitemap (trade press, company newsrooms, local papers, blogs); feeds are found automatically. - ๐Ÿงน **Duplicates collapsed**: the same wire story at ten publishers comes back once, earliest copy first, with the others in `alsoPublishedBy` (free). - ๐Ÿ”” **Only new articles** for schedules: articles returned by an earlier run of the same monitor are skipped for free. - ๐Ÿ“ฐ **Optional article details**: authors, section, tags, modified time, canonical link, paywall flag and word count from the article page's own metadata (no article text). - ๐Ÿ“„ **CSV file** for Excel or Google Sheets, and a SOURCES record showing what every publisher returned. #### Why this Actor? - **No search-engine scraping, so no "0 results" runs.** Google News scrapers break when Google changes its pages or blocks them. This Actor reads what publishers publish *for* crawlers and readers: their **Google News sitemaps** (every article of the last ~48 hours with headline, time and keywords) and their **RSS / Atom feeds**. In our tests all 254 publishers were read in every run. - **Fresh and complete per publisher.** A news sitemap lists every article a publisher put out in the last two days, not just the 10-100 items a search page shows. - **Honest crawler.** Every request names itself (`TidyToolsBot`), follows each site's robots.txt and Content-Signal rules, and asks one thing at a time per site. Nothing is bypassed. - **Cheap and predictable**: $1 per 1,000 articles, no start fee. Filtered articles, collapsed duplicates, already-seen articles and unreadable sources are free. #### Real results (3-4 October 2026, run on our machine) - **`OpenAI OR Anthropic` and `"electric vehicle" battery -recall`, English**: 165 English-language publishers read, 42,723 articles checked, 135 matched, **50 returned in 32 seconds for $0.05** (Business Insider, Forbes, TechCrunch, Engadget, Fox News, Yahoo Finance, Decrypt...). - **Germany, Austria, Switzerland: `Merz OR Bundesregierung`, `Klimaschutz`, sorted by relevance, details for the first 15**: 15 publishers, 3,532 articles checked, **40 returned + 15 article details in 6.5 seconds for $0.0475** (WELT, BILD, Der Spiegel, n-tv, FAZ, SZ, Handelsblatt, tagesschau). - **All 254 publishers at once** (two searches, no filters): 67,568 articles checked in 47 seconds. - **Your own sources** (Cloudflare blog, simonwillison.net, an OpenAI RSS link, TechCrunch, plus 4 bad lines), last 7 days: **30 articles for $0.03 in 3.3 seconds**; The Guardian (HTTP 403), a site without a feed, a missing sitemap and a non-URL came back as 4 free error rows. - **Only new**: `Nvidia OR AMD OR Intel` monitor, first run 90 articles ($0.09); second run a minute later **0 articles, $0.00** (90 returned before). ### Use cases - **Media monitoring and PR**: your brand, products, executives or competitors across major publishers, every few hours into Slack, email or a sheet. - **Market and investment research**: news about a company, sector or commodity by country and language. - **AI agents and RAG**: a fresh, structured news feed with links and descriptions; give the links to a page reader when you need the full text. - **Trade press tracking**: add your industry's sites and newsrooms as sources and get only new articles. - **Content and SEO teams**: what the press is writing about a topic right now, with the publishers' own keywords. ### How to use it 1. Type one or more **keywords** (one search per line), e.g. `OpenAI OR Anthropic`, `"interest rates" -mortgage`, `tariff*`. 2. Optional: pick **countries**, **languages** and **topics**, add **your own sources**, and set **published within** (hours) and **maximum articles**. 3. For alerts, turn on **Only new articles** and schedule the Actor (every 3-6 hours is enough: news sitemaps keep about 48 hours). 4. Run. Results are in the dataset (views: Articles, Keywords and duplicates, Article page details, Errors), in `ARTICLES.csv` and in `SOURCES`. #### Input example ```json { "keywords": ["OpenAI OR Anthropic", "\"electric vehicle\" battery -recall"], "languages": ["en"], "maxAgeHours": 48, "maxArticles": 50, "sortBy": "newest" } ``` Monitoring on a schedule: `{"keywords": ["Acme Corp", "\"Jane Doe\" Acme"], "onlyNew": true, "monitorName": "acme-press"}`. Only your sources: `{"sources": ["https://blog.cloudflare.com", "simonwillison.net", "https://openai.com/news/rss.xml"], "onlyMySources": true, "maxAgeHours": 168}`. #### Output example One article from a real run (3 October 2026): ```json { "type": "article", "rank": 3, "title": "Former Anthropic security leader warns AI agents are becoming too autonomous for humans to keep them in check", "url": "https://www.foxnews.com/tech/former-anthropic-security-leader-warns-ai-agents-becoming-too-autonomous-humans-keep-check", "publisher": "Fox News", "sourceDomain": "foxnews.com", "publishedAt": "2026-10-03T17:26:10.000Z", "country": "US", "language": "en", "topics": ["general", "politics"], "matchedQueries": ["OpenAI OR Anthropic"], "matchedInTitle": true, "description": "Jeffrey Ladish, a former Anthropic security team member, warns AI agents are colluding and hacking with no reliable solution to keep them in check.", "keywords": [], "categories": ["Fox News", "fox-news/tech/artificial-intelligence", "fox-news/politics", "fox-news/tech"], "author": null, "imageUrl": "https://a57.foxnews.com/static.foxnews.com/foxnews.com/content/uploads/2026/10/931/523/jeffrey-ladish.jpg?ve=1&tl=1", "foundVia": ["news-sitemap", "feed"], "alsoPublishedBy": [], "success": true, "charged": true, "scrapedAt": "2026-10-03T19:02:11.000Z" } ``` With **Add article page details**, a row also gets (shortened, from the German run): ```json "details": { "canonicalUrl": "https://www.bild.de/politik/clara-von-nathusius-friedrich-merz-wird-keine-wahlen-mehr-gewinnen-6abf9a17863d7086e22737e0", "metaDescription": "CDU-Influencerin Clara von Nathusius kritisiert Merz, Parteistrukturen und die Koalition. ...", "authors": [], "tags": ["CDU", "Merz Friedrich"], "modifiedAt": "2026-10-03T14:20:00.000Z", "isAccessibleForFree": null, "wordCount": null } ``` #### What the fields mean | Field | Meaning | |---|---| | `title`, `url`, `publisher`, `sourceDomain` | Headline, article link (tracking parameters removed), publisher name and the site it was read from. | | `publishedAt` | Publication time (UTC) from the news sitemap or feed; the earliest when both give one. | | `country`, `language`, `topics` | Publisher country and topics; article language (from the news sitemap when it says so, else the publisher's). | | `matchedQueries`, `matchedInTitle` | Which of your searches matched, and whether the match is in the headline. | | `description`, `keywords`, `categories`, `author`, `imageUrl` | What the feed / news sitemap gives: summary (up to 600 characters), the publisher's news keywords, feed categories, author, image. Empty when the publisher does not provide it. | | `foundVia` | `news-sitemap` and/or `feed`. | | `alsoPublishedBy` | Other publishers with the same headline (collapsed duplicates): publisher, link, time. | | `details`, `detailsError` | Optional article page metadata (see above) or why it could not be read (free). | | `isNew` | Only-new mode: always true on returned rows (older ones are skipped). | | `rank`, `success`, `charged`, `errorType`, `error` | Position in the results and the usual TidyTools status fields. | ### How much does it cost? | Event | Price | When | |---|---|---| | Article | **$1.00 / 1,000** ($0.001 each) | one article returned after your filters | | Article details | $0.50 / 1,000 | only with "Add article page details", and only when the page published metadata | **Free:** articles dropped by your filters, duplicate copies of a story, articles returned before in only-new mode, sources that cannot be read, invalid inputs, details of blocked or empty pages. **No start fee.** Silver plans get 10% off, Gold and above 20%. For comparison (Apify Store, checked October 2026): data_xplorer/google-news-scraper-fast $4.00 / 1,000 results, easyapi/google-news-scraper $5.00, automation-lab/google-news-scraper $2.30, crawlerbros/google-news-scraper $2.00. #### Control your cost - **Maximum articles** caps each run; a run checks tens of thousands of articles but you only pay for those returned. - Apify's **maximum charge per run** is respected: the run saves what fits and lists the rest in `SUMMARY.notProcessed`. - The plan (worst case) is printed at the start of the run and stored in `SUMMARY.costPlan`. ### Use with AI agents (MCP) Connect Apify's MCP server (`https://mcp.apify.com?tools=tidytools/news-scraper`) to Claude, Cursor or any MCP client, then ask e.g. "Find today's news about Nvidia export rules in English and German and give me the links." Minimal input: ```json { "keywords": ["Nvidia export"], "languages": ["en", "de"], "maxArticles": 30 } ``` Failed sources and inputs are not charged and carry an `errorType`; `SUMMARY.status` is `SUCCESS`, `PARTIAL_RESULTS`, `NO_RESULTS`, `LIMIT_REACHED` or `FAILED`. ### Limitations - **Coverage is the publisher list plus your sources**, not the whole web like Google News. Some large publishers are not included because their robots.txt closes their feeds and sitemaps to crawlers, or they refused our requests when the list was built (Reuters, AP, The Guardian, USA Today and about 50 others). You can still add any site as a source; if it refuses, you get a free error row. - **Recency**: news sitemaps hold about 48 hours, feeds the latest items. For a history, schedule the Actor with only-new mode; it cannot search past months. - **Headline-level data**: description, keywords and author depend on what each publisher puts in its feed or sitemap (some give only the headline). No article text is returned: it is the publishers' copyrighted content. Use a page reader on the links if you need the text for your own analysis. - Keywords match whole words (no stemming): `tariff` does not match `tariffs`; use `tariff*`. Chinese, Japanese and Thai keywords match as substrings. - Articles whose feed gives no date (or an impossible one) are skipped unless you turn on "Include articles without a date". - Publisher websites can change their feeds; the built-in list is re-checked regularly and `SOURCES` shows any publisher that failed in your run (free). ### FAQ **Does the Actor collect usage data?** The Actor sends one anonymous usage counter per run (input type, item count range, outcome) to improve the tool; no URLs, content or account data. **What happens if my input is wrong?** The run ends normally (it does not fail) with a free row, `errorType: "invalid_input"`, and a message saying what to fix: a search that only excludes words, an unknown topic, "only my sources" with no sources, or country / language codes that match no publisher (countries are ISO codes such as `GB`, languages two-letter codes such as `en`). A source line that is not an address, a site that cannot be reached or one without any feed is a free error row for that line; the other sources still run. Nothing is charged for any of them. A run where no article matches leaves one free `no_results` row naming the filter that dropped the most articles (none in "Only new" runs that simply have nothing new). **Is this a Google News scraper?** No. It does not touch Google. It reads the same publisher feeds and news sitemaps that news aggregators read, so it keeps working when search pages change. **How fresh are the results?** As fresh as the publishers' own sitemaps and feeds, usually minutes after publication. **Can I get the full article text?** No. The Actor returns headlines, links and the metadata publishers make available. Respecting copyright, the article body is not copied. **Why did a publisher return nothing?** See the `SOURCES` record in the run's key-value store: it lists each source, the feeds and sitemaps read, how many articles they had and any error. **Can I add paywalled sites?** Yes, if their feeds or news sitemaps are public (headline, link and time are returned; the article text never is). Article details include `isAccessibleForFree` when the page declares it. Some paywalled publishers refuse article-page requests (HTTP 403): those rows get a `detailsError` and the details are free. **Can I filter to one publisher?** Add its domain under "Your own sources" and turn on "Only my sources" (for a publisher in the built-in list its verified feeds are used), or use countries, languages and topics to narrow the list. Use "Exclude domains" to drop publishers. **Is it legal?** It only reads public feeds and sitemaps that publishers offer for this purpose, follows robots.txt, identifies itself and returns metadata and links, not article text. Check the terms of the sites you add yourself. # Actor input Schema ## `keywords` (type: `array`): What to look for, one search per line. All words of a line must appear (title, description, news keywords or the link), in any order, whole words, case and accents ignored. Use "quotes" for an exact phrase, -word to exclude, OR between alternatives and word\* for a prefix, e.g. Nvidia OR AMD, "interest rates" -mortgage, tariff\*. An article matching any line is returned. Leave empty to get the latest articles of the selected publishers. ## `countries` (type: `array`): Only publishers from these countries (254 publishers in 47 countries). Empty = all. Does not apply to your own sources. ## `languages` (type: `array`): Only articles in these languages (publisher language, or the article's own language when its news sitemap says so). Empty = all. ## `topics` (type: `array`): Only publishers that cover these topics, e.g. technology for TechCrunch, The Verge, heise... Empty = all publishers (general news sites also cover every topic, so for a subject search usually leave this empty and use keywords). ## `sources` (type: `array`): More sites to read, one per line: a home page or domain (its Google News sitemap and RSS / Atom feeds are found automatically), a section page, or a feed or news sitemap address. Trade press, company newsrooms, local papers, blogs. Sources that cannot be read are listed as free error rows. ## `onlyMySources` (type: `boolean`): Read only the sources listed above, not the built-in publisher list. ## `excludeDomains` (type: `array`): Never return articles from these domains (and their subdomains), e.g. dailymail.co.uk. ## `maxAgeHours` (type: `integer`): Only articles published in the last N hours. Google News sitemaps hold about 48 hours; RSS feeds usually hold the latest 10-100 items, sometimes older. For a longer history, schedule the Actor with "Only new articles". ## `maxArticles` (type: `integer`): Upper limit of articles returned (and charged) in this run, after filters and duplicate collapsing. ## `sortBy` (type: `string`): Newest first, or relevance (keyword in the title first, then articles matching more of your searches, then newest). ## `searchIn` (type: `string`): Title, description, news keywords, categories and link words (default), or the headline only (fewer, more precise results). ## `collapseDuplicates` (type: `boolean`): Wire stories (AP, Reuters, AFP...) republished with the same headline are returned once, earliest copy first, with the other publishers in alsoPublishedBy (free). Turn off to get every copy. ## `includeArticleDetails` (type: `boolean`): Also open each returned article page (if its robots.txt allows) and read the metadata it publishes: authors, section, tags, modified time, canonical link, paywall flag (isAccessibleForFree), word count and description. No article text is returned. Extra "Article details" event per article where metadata was found; blocked or empty pages are free. ## `detailsMaxArticles` (type: `integer`): With article details on, read details only for the first N articles of the results (0 = all). ## `onlyNew` (type: `boolean`): For schedules: return only articles that an earlier run of the same monitor did not return (by link and by headline, remembered 14 days). Articles returned before are skipped for free. The first run returns everything. ## `monitorName` (type: `string`): Name of the "only new" memory. Runs with the same name share it; by default it is derived from your keywords and filters, so changing them starts a new memory. ## `includeUndated` (type: `boolean`): Some feeds do not date their items. Off (default): such items are skipped, because their age cannot be checked. ## `csvExport` (type: `boolean`): Save ARTICLES.csv (main columns) in the run's key-value store for Excel or Google Sheets. ## `maxConcurrency` (type: `integer`): How many publishers are read at the same time. Each publisher still gets one request at a time, with a short pause between requests. ## `proxyConfiguration` (type: `object`): Not needed: the Actor reads publishers' public feeds and sitemaps from Apify's network. Set a proxy only if your own sources refuse data-center IPs. Apify proxy usage is billed to your Apify account. ## Actor input object example ```json { "keywords": [ "OpenAI OR Anthropic", "\"electric vehicle\" battery -recall" ], "languages": [ "en" ], "onlyMySources": false, "maxAgeHours": 48, "maxArticles": 50, "sortBy": "newest", "searchIn": "all", "collapseDuplicates": true, "includeArticleDetails": false, "detailsMaxArticles": 100, "onlyNew": false, "includeUndated": false, "csvExport": true, "maxConcurrency": 16 } ``` # Actor output Schema ## `articles` (type: `string`): No description ## `context` (type: `string`): No description ## `details` (type: `string`): No description ## `full` (type: `string`): No description ## `errors` (type: `string`): No description ## `csv` (type: `string`): No description ## `sources` (type: `string`): No description ## `summary` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "keywords": [ "OpenAI OR Anthropic", "\"electric vehicle\" battery -recall" ], "languages": [ "en" ], "maxArticles": 50 }; // Run the Actor and wait for it to finish const run = await client.actor("tidytools/news-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`๐Ÿ’พ Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // ๐Ÿ“š Want to learn more ๐Ÿ“–? Go to โ†’ https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "keywords": [ "OpenAI OR Anthropic", "\"electric vehicle\" battery -recall", ], "languages": ["en"], "maxArticles": 50, } # Run the Actor and wait for it to finish run = client.actor("tidytools/news-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"๐Ÿ’พ Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # ๐Ÿ“š Want to learn more ๐Ÿ“–? Go to โ†’ https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "keywords": [ "OpenAI OR Anthropic", "\\"electric vehicle\\" battery -recall" ], "languages": [ "en" ], "maxArticles": 50 }' | apify call tidytools/news-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,tidytools/news-scraper" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/c8LZvugrqlfaTaBDe/builds/gpzGk8LFpLLwyw1HT/openapi.json