# The Verge News Scraper (`real-estate-api/the-verge-news-scraper`) Actor

The Verge News Scraper is a Python tool designed to extract tech headlines, article content, authors, publication dates, and category tags from The Verge. It exports structured news data into clean CSV or JSON formats for media monitoring and analysis.

- **URL**: https://apify.com/real-estate-api/the-verge-news-scraper.md
- **Developed by:** [real-estate-api](https://apify.com/real-estate-api) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.99 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## The Verge News Scraper: Headlines to JSON or CSV

<p align="center">
  <img src="https://imgur.com/hF8vZp4.png" alt="The Verge News Scraper" width="800" />
</p>

The Verge News Scraper collects headlines from The Verge's public Atom feed and saves one dataset row per story. Extract headlines, article links, publication dates, authors, and short summaries for news monitoring, research, and scheduled exports.

Export the dataset as JSON, CSV, or Excel. The feed contains recent stories rather than a complete archive, so schedule recurring runs to keep your collection current and use the `id` field to identify previously processed stories.

### Use Cases

- **News monitoring:** Collect headlines on a schedule for a spreadsheet or database.
- **Keyword tracking:** Filter stories by words in their headlines or summaries.
- **Link export:** Build a list of article URLs without opening individual stories.
- **Research:** Collect author names and publication dates for recent coverage.
- **Change tracking:** Compare story IDs across runs to identify new items.

### Input Parameters

| Parameter | Type | Default | Description |
|---|---|---|---|
| `feedUrls` | Array | The Verge Atom feed | Feed URLs to process. Only the verified The Verge feed is accepted. |
| `keyword` | String | Empty | Optional filter applied to headlines and short summaries. |
| `maxItems` | Integer | `100` | Maximum number of headline rows to save per run. Hard limit: `1000`. |
| `timeoutSecs` | Integer | `300` | Overall run time limit in seconds. |
| `requestTimeoutSecs` | Integer | `30` | Timeout for an individual feed request in seconds. |

### Example Input

```json
{
  "feedUrls": [
    "THE_VERGE_ATOM_FEED_URL"
  ],
  "keyword": "",
  "maxItems": 100,
  "timeoutSecs": 300,
  "requestTimeoutSecs": 30
}
```

### What Data Does This Actor Extract?

Each feed entry becomes a dataset row. The actor extracts the headline, story link, publication timestamp, author, short summary, and stable entry ID. HTML is removed from summary text, and full article bodies are not collected.

#### Example Output

```json
{
  "itemTitle": "Example The Verge Story Headline",
  "link": "ARTICLE_URL",
  "published": "2026-10-11T16:06:43-04:00",
  "author": "Example Author",
  "summary": "A short summary of the story from the feed.",
  "id": "STABLE_STORY_ID",
  "sourceUrl": "THE_VERGE_ATOM_FEED_URL",
  "scrapedAt": "2026-10-12T00:00:00+00:00"
}
```

#### Output Fields

| Field | Type | Description |
|---|---|---|
| `itemTitle` | String | Story headline. |
| `link` | String or null | Article URL provided by the feed. |
| `published` | String or null | Publication timestamp as provided by the feed. |
| `author` | String or null | Author name, when available. |
| `summary` | String or null | Short summary with HTML removed. |
| `id` | String or null | Stable entry identifier used for deduplication. |
| `sourceUrl` | String | Feed URL that produced the row. |
| `scrapedAt` | String | UTC timestamp when the row was created. |
| `error` | String | Error description for a failed input. |
| `input` | String | Invalid or failed input associated with an error row. |

#### Error Handling

An invalid feed URL, an HTTP error, or an HTML response where Atom XML is expected produces an error row. The actor continues processing other configured feed entries when possible.

### How It Works

1. Reads the feed list, optional keyword, item limit, and timeout settings.
2. Validates the configured feed URL against the accepted The Verge feed.
3. Requests the feed using a descriptive User-Agent.
4. Retries HTTP 429 and 5xx responses up to three times, waiting 2, 4, and 8 seconds, or following a suitable `Retry-After` value when provided.
5. Parses Atom entries, removes HTML from text fields, filters by keyword when configured, and removes duplicate IDs.
6. Saves results until the configured `maxItems` limit is reached.

#### Feed Limitations

- The feed contains recent stories and does not provide a complete historical archive.
- There is no next-page cursor for retrieving older entries from this feed.
- Schedule recurring runs approximately 5–15 minutes apart for ongoing monitoring.
- Compare the `id` field across runs to identify stories that have not yet been processed.

### Limits and Terms

This actor stores headlines, links, publication dates, author names, and short summaries supplied by the feed. It does not fetch or store full article bodies.

The applicable source terms should be reviewed before using exported data, particularly for commercial use, republication, attribution, and copyrighted material.

**Pricing:** The actor is free to run. Normal Apify platform usage charges may apply.

### FAQ

#### Does The Verge News Scraper require a login or application key?

No. It reads The Verge's public Atom feed without requiring a login.

#### Can I export results to CSV or JSON?

Yes. Dataset results can be downloaded as JSON, CSV, or Excel.

#### Why are older stories missing?

The feed contains recent entries rather than a complete archive. Schedule repeated runs to build an ongoing collection of headlines.

#### What happens if a feed URL is invalid?

The invalid input is recorded in an error row. Other configured feed entries can still be processed.

#### Does the actor download full articles?

No. It collects feed metadata and short summaries, not full article text.

### Trademark Notice

The Verge and related names, logos, and trademarks belong to their respective owners. Their use in this documentation is solely to identify the source of the data and does not imply affiliation, sponsorship, endorsement, or partnership.

### Disclaimer

This actor is an independent tool and is not affiliated with, endorsed by, sponsored by, or officially associated with The Verge or its owners. Users are responsible for complying with applicable source terms, rate limits, attribution requirements, and licensing restrictions.

### Is Web Scraping Legal?

Collecting information that is already public is generally legal. Scraping is not the same thing as hacking, and there is no single law that bans it. The limits are the usual ones: do not break into accounts, do not take private or confidential data, and be careful with personal data and copyrighted works. A site's terms can also restrict how you access it. Facts such as a public name, price, or rating are not the same thing as republishing someone's article or photo.

This actor collects information from public pages. It does not log in and it does not bypass access controls.

For a longer explanation, see Apify's article [Is web scraping legal?](https://blog.apify.com/is-web-scraping-legal/).

# Actor input Schema

## `feedUrls` (type: `array`):

Public Verge Atom feed URLs. The only verified address is https://www.theverge.com/rss/index.xml. The feed lists the latest headlines, not an archive.

## `keyword` (type: `string`):

Optional case-insensitive filter applied to the headline and short summary. Leave empty to keep every item.

## `maxItems` (type: `integer`):

Maximum number of headline rows to save. Values are clamped between 1 and 1000.

## `timeoutSecs` (type: `integer`):

Overall time limit for the run, in seconds. The actor stops fetching when this deadline is reached.

## `requestTimeoutSecs` (type: `integer`):

Per-request timeout for the feed download, in seconds.

## `proxyConfiguration` (type: `object`):

Optional proxy for feed requests. Default is no proxy. Turn on Apify Proxy only if the feed starts blocking your IP.

## Actor input object example

```json
{
  "feedUrls": [
    "https://www.theverge.com/rss/index.xml"
  ],
  "keyword": "Flock",
  "maxItems": 100,
  "timeoutSecs": 300,
  "requestTimeoutSecs": 30,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

Headline rows saved from the Verge Atom feed.

## `overview` (type: `string`):

Table view of headlines, links, authors, and publish dates.

## `json` (type: `string`):

Dataset items as JSON.

## `csv` (type: `string`):

Dataset items as CSV.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "feedUrls": [
        "https://www.theverge.com/rss/index.xml"
    ],
    "keyword": "",
    "maxItems": 100,
    "timeoutSecs": 300,
    "requestTimeoutSecs": 30,
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("real-estate-api/the-verge-news-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "feedUrls": ["https://www.theverge.com/rss/index.xml"],
    "keyword": "",
    "maxItems": 100,
    "timeoutSecs": 300,
    "requestTimeoutSecs": 30,
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("real-estate-api/the-verge-news-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "feedUrls": [
    "https://www.theverge.com/rss/index.xml"
  ],
  "keyword": "",
  "maxItems": 100,
  "timeoutSecs": 300,
  "requestTimeoutSecs": 30,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call real-estate-api/the-verge-news-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,real-estate-api/the-verge-news-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/YhpFT82w5jZs8Tcvm/builds/pbrK1rHtudL5h1R8a/openapi.json