# Anthropic News & Research Scraper (`automation-lab/anthropic-scraper`) Actor Scrapes news articles and research papers from Anthropic's website. Returns title, date, categories, description, image URL, and optionally full article text. - **URL**: https://apify.com/automation-lab/anthropic-scraper.md - **Developed by:** [Stas Persiianenko](https://apify.com/automation-lab) (community) - **Categories:** AI, News - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing Pay per event This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Anthropic News & Research Scraper Scrape news articles and research papers from [Anthropic's website](https://www.anthropic.com). Get titles, publish dates, categories, descriptions, image URLs, and optionally full article text — no API key required. ### What does Anthropic News & Research Scraper do? Anthropic News & Research Scraper extracts all articles from the [Anthropic News](https://www.anthropic.com/news) and [Anthropic Research](https://www.anthropic.com/research) sections. For each article it collects the title, URL, slug, publish date, categories, description excerpt, hero image URL, and optionally the full article body text. The scraper uses standard HTTP requests and Cheerio — no browser or JavaScript execution required. It works reliably because Anthropic's website is server-side rendered. ### Why scrape Anthropic? Anthropic is one of the world's leading AI safety and research companies. Their news and research pages publish significant announcements about AI models (Claude), safety research, interpretability findings, and policy work. Key reasons to track Anthropic content: - **AI model tracking** — Monitor new Claude model releases and capability updates - **Research monitoring** — Follow Anthropic's safety, interpretability, and societal impact research - **Competitive intelligence** — Track AI product launches and strategic partnerships - **News aggregation** — Build feeds or newsletters covering the AI industry - **AI training data** — Create structured datasets of AI research for RAG pipelines and knowledge bases - **Due diligence** — Stay current on AI policy positions and company announcements ### Who is it for? This actor is for anyone who needs structured, machine-readable access to Anthropic's public content: - **AI researchers** tracking the latest Claude model capabilities and benchmarks - **Investors** monitoring Anthropic's product announcements and funding activity - **Journalists and analysts** covering the AI industry - **Newsletter curators** building AI-focused digest content - **Data scientists** building AI company monitoring dashboards - **Policy researchers** studying AI safety and governance publications - **Engineers** staying current with AI API and platform changes - **AI/ML teams** building training datasets or knowledge bases ### How to use Anthropic News & Research Scraper 1. Go to [Anthropic News & Research Scraper](https://apify.com/automation-lab/anthropic-scraper) on Apify Store 2. Choose which content to scrape: news, research, or both 3. Set max results and whether to include full article text 4. Click **Start** and wait for results 5. Download data as JSON, CSV, or Excel ### Input parameters | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `scrapeType` | string | `"all"` | Which section to scrape: `all`, `news`, or `research` | | `maxResults` | integer | `100` | Maximum number of articles to return | | `scrapeContent` | boolean | `false` | Fetch the full article body text for each article | | `maxRequestRetries` | integer | `3` | Retry attempts for failed HTTP requests | #### Input example ```json { "scrapeType": "all", "maxResults": 20, "scrapeContent": false } ```` To also get the full article text: ```json { "scrapeType": "news", "maxResults": 10, "scrapeContent": true } ``` ### Output Each item in the dataset represents one article: | Field | Type | Description | |-------|------|-------------| | `url` | string | Full article URL | | `slug` | string | Article slug (URL path segment) | | `title` | string | Article title | | `categories` | array | Content categories (e.g., `["Product", "Announcements"]`) | | `publishedDate` | string | Publish date as shown on the page (e.g., `"Apr 16, 2026"`) | | `description` | string | Short excerpt or meta description | | `imageUrl` | string | Hero image URL (if available) | | `type` | string | Content type: `news` or `research` | | `content` | string | Full article body text (only when `scrapeContent: true`) | #### Output example ```json { "url": "https://www.anthropic.com/news/claude-opus-4-7", "slug": "claude-opus-4-7", "title": "Introducing Claude Opus 4.7", "categories": ["Product", "Announcements"], "publishedDate": "Apr 16, 2026", "description": "Our latest model, Claude Opus 4.7, is now generally available.", "imageUrl": "https://cdn.sanity.io/images/4zrzovbb/website/96ea2509a90e527642c822303e56296a07bcfce4-1920x1080.png", "type": "news", "content": null } ``` ### API usage You can run this actor programmatically using the [Apify API](https://docs.apify.com/api/v2) or the official client libraries. #### Node.js (ApifyClient) ```javascript const { ApifyClient } = require('apify-client'); const client = new ApifyClient({ token: 'YOUR_API_TOKEN' }); const run = await client.actor('automation-lab/anthropic-scraper').call({ scrapeType: 'all', maxResults: 20, scrapeContent: false, }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); console.log(items); ``` #### Python (ApifyClient) ```python from apify_client import ApifyClient client = ApifyClient("YOUR_API_TOKEN") run = client.actor("automation-lab/anthropic-scraper").call(run_input={ "scrapeType": "all", "maxResults": 20, "scrapeContent": False, }) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) ``` #### cURL ```bash ## Start the actor curl -X POST "https://api.apify.com/v2/acts/automation-lab~anthropic-scraper/runs?token=YOUR_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{"scrapeType":"all","maxResults":20,"scrapeContent":false}' ## Get results (replace RUN_ID and DATASET_ID with actual values) curl "https://api.apify.com/v2/datasets/DATASET_ID/items?token=YOUR_API_TOKEN&format=json" ``` ### Use with MCP (Claude Desktop / AI Agents) This actor is compatible with the [Apify MCP Server](https://apify.com/apify/mcp-server), enabling direct use from Claude Desktop, Claude Code CLI, Cursor, or any MCP-compatible AI agent. #### Claude Code CLI (recommended) Add only this actor as an MCP tool — no extra actors loaded: ```bash claude mcp add --transport http apify-anthropic-scraper \ "https://mcp.apify.com?tools=automation-lab/anthropic-scraper" \ --header "Authorization: Bearer YOUR_API_TOKEN" ``` #### Claude Desktop / Cursor (full Apify MCP server) ```json { "mcpServers": { "apify": { "command": "npx", "args": ["-y", "@apify/mcp-server"], "env": { "APIFY_TOKEN": "YOUR_API_TOKEN" } } } } ``` #### Example prompts Once connected, try these prompts in Claude: - *"Scrape the latest 20 news articles from Anthropic and summarize the top 3 product announcements."* - *"Get all Anthropic research papers from the last month and list the ones related to interpretability."* - *"Fetch Anthropic news with full article text and identify any mentions of new Claude model releases."* ### Integrations Connect Anthropic News & Research Scraper to your existing workflows with Apify's built-in integrations: #### Anthropic News → Google Sheets monitoring dashboard Use the [Google Sheets integration](https://apify.com/apify/google-sheets-import-export) to append every new article to a spreadsheet automatically. Set up a scheduled run (daily or weekly) and each new article lands as a new row with title, date, category, and URL — a living dashboard of Anthropic's publications. #### Anthropic News → Slack alerts Pipe new articles to a Slack channel using the [Apify → Slack integration](https://apify.com/apify/slack-notification). Configure a webhook trigger: whenever the actor finds articles published after your last run, a message is posted to your `#ai-news` channel with the title and URL. Never miss a Claude model announcement again. #### Anthropic Research → RAG knowledge base Run the actor weekly with `scrapeContent: true` and feed the full article text into a vector database (Pinecone, Weaviate, or Qdrant) via [Apify's dataset webhooks](https://docs.apify.com/platform/integrations/webhooks). Build a retrieval-augmented chatbot that answers questions about Anthropic's research using up-to-date content. #### Anthropic News → Make (Integromat) / Zapier Use the [Apify → Make integration](https://apify.com/apify/make-integration) or [Apify → Zapier integration](https://zapier.com/apps/apify/integrations) to fan out new articles to any downstream service: CRM notes, Notion databases, newsletter drafts, or custom webhooks. ### Tips - **Start small for testing** — set `maxResults` to 10 and `scrapeContent: false` on your first run. Confirm the data looks correct before scraping everything. - **Use `scrapeType: "research"` for technical content** — the research section contains peer-reviewed and technical papers (interpretability, alignment, safety). Use `scrapeType: "news"` for product announcements and press releases. - **Enable `scrapeContent` selectively** — fetching full article text roughly doubles the run time and cost. Only enable it when you actually need the body text (e.g., for RAG ingestion or summarization). - **Schedule incremental runs** — Anthropic posts a handful of articles per week. A daily or weekly scheduled run with `maxResults: 20` is more than enough to stay current without over-fetching. - **Filter by category in post-processing** — the `categories` field lets you narrow results downstream. For example, filter for `["Product"]` to watch only model releases, or `["Policy"]` for regulatory content. ### Pricing This actor uses pay-per-event pricing: | Tier | Price per article | |------|------------------| | Free | $0.001 | | Bronze | $0.00087 | | Silver | $0.00067 | | Gold | $0.00052 | | Platinum | $0.00035 | | Diamond | $0.00024 | Plus a flat **$0.005** fee per run (startup cost). Scraping 20 news articles costs approximately **$0.025** on the Free tier. ### Legality and terms of use This actor scrapes publicly available content from Anthropic's website. All scraped content is publicly accessible without login. The actor respects standard HTTP conventions and does not circumvent any access controls. Users are responsible for ensuring their use of scraped data complies with Anthropic's [Terms of Service](https://www.anthropic.com/legal/consumer-terms) and applicable laws. This actor is intended for research, monitoring, and informational purposes. ### FAQ **Does this require an Anthropic API key?** No. The actor scrapes Anthropic's public website — no API key or authentication is needed. **Does it scrape Claude.ai or the Anthropic API?** No. It only scrapes the public marketing website (anthropic.com/news and anthropic.com/research). It does not access Claude.ai, the Anthropic API, or any authenticated endpoints. **How often does Anthropic publish new content?** Anthropic typically publishes a few news articles and research papers per week. The news listing shows the most recent ~10–15 articles. **Can I get the full article text?** Yes — set `scrapeContent: true` in the input. This makes additional HTTP requests per article and increases cost proportionally. **What categories exist in the research section?** Common research categories include: Interpretability, Alignment, Policy, Economic Research, Societal Impacts, and Safety. **Does it handle pagination?** The current version scrapes the articles shown on the main listing pages. Anthropic typically shows the most recent 10–15 articles per section. **Why are some articles missing `description` or `imageUrl`?** Anthropic doesn't always include a description excerpt or hero image for every article — particularly older research papers. When these fields aren't present in the page HTML, the actor returns `null` for those fields. This is expected behavior, not a bug. **The run succeeded but returned fewer articles than my `maxResults` setting — why?** `maxResults` is an upper bound. If Anthropic's listing page contains fewer articles than your limit, the actor returns however many are available. This is normal for fresh installations or when the listing page hasn't been updated recently. **The scraper returned zero results or failed with HTTP errors — what should I do?** This usually means Anthropic temporarily changed their page structure or the listing page is returning a non-200 status. Try these steps: (1) check the run log for specific error messages, (2) run again — transient errors often resolve in a retry, (3) if failures persist for more than a day, [report an issue](https://github.com/apify/apify-sdk-js/issues) so the actor can be updated. **Can I scrape specific articles by URL instead of the full listing?** The actor currently scrapes the listing pages (`/news`, `/research`) and optionally fetches individual article content. Direct URL scraping of arbitrary articles is not supported in the current version. ### Related actors - [Hugging Face Papers Scraper](https://apify.com/automation-lab/huggingface-papers-scraper) — Scrape AI research papers from HuggingFace - [arXiv Scraper](https://apify.com/automation-lab/arxiv-scraper) — Scrape papers from arXiv - [TechCrunch Scraper](https://apify.com/automation-lab/techcrunch-scraper) — Scrape tech news from TechCrunch - [Hacker News Scraper](https://apify.com/automation-lab/hackernews-scraper) — Scrape stories from Hacker News # Actor input Schema ## `scrapeType` (type: `string`): Which section to scrape: news articles, research papers, or both. ## `maxResults` (type: `integer`): Maximum number of articles to return across all sections. ## `scrapeContent` (type: `boolean`): When enabled, fetches each article's full body text. This increases runtime and cost proportionally. ## `maxRequestRetries` (type: `integer`): Number of retry attempts for failed HTTP requests. ## Actor input object example ```json { "scrapeType": "all", "maxResults": 20, "scrapeContent": false, "maxRequestRetries": 3 } ``` # Actor output Schema ## `overview` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "scrapeType": "all", "maxResults": 20, "scrapeContent": false, "maxRequestRetries": 3 }; // Run the Actor and wait for it to finish const run = await client.actor("automation-lab/anthropic-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "scrapeType": "all", "maxResults": 20, "scrapeContent": False, "maxRequestRetries": 3, } # Run the Actor and wait for it to finish run = client.actor("automation-lab/anthropic-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "scrapeType": "all", "maxResults": 20, "scrapeContent": false, "maxRequestRetries": 3 }' | apify call automation-lab/anthropic-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=automation-lab/anthropic-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Anthropic News & Research Scraper", "description": "Scrapes news articles and research papers from Anthropic's website. Returns title, date, categories, description, image URL, and optionally full article text.", "version": "0.1", "x-build-id": "C9xWl0jfmkQFC3Fbg" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/automation-lab~anthropic-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-automation-lab-anthropic-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/automation-lab~anthropic-scraper/runs": { "post": { "operationId": "runs-sync-automation-lab-anthropic-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/automation-lab~anthropic-scraper/run-sync": { "post": { "operationId": "run-sync-automation-lab-anthropic-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "scrapeType": { "title": "Content type", "enum": [ "all", "news", "research" ], "type": "string", "description": "Which section to scrape: news articles, research papers, or both.", "default": "all" }, "maxResults": { "title": "Max results", "minimum": 1, "maximum": 500, "type": "integer", "description": "Maximum number of articles to return across all sections.", "default": 100 }, "scrapeContent": { "title": "Scrape full article text", "type": "boolean", "description": "When enabled, fetches each article's full body text. This increases runtime and cost proportionally.", "default": false }, "maxRequestRetries": { "title": "Max request retries", "minimum": 1, "maximum": 10, "type": "integer", "description": "Number of retry attempts for failed HTTP requests.", "default": 3 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```