# Guardian Articles & Liveblogs Scraper (`scrapyx/guardian-articles-scraper`) Actor

Fetches Guardian articles and liveblogs via the Guardian's own Open Platform Content API -- headline, standfirst, full body text, byline, contributor and keyword tags, section and dates -- with search, section, tag and date filtering across 1.9M+ items. Needs your own free Guardian API key.

- **URL**: https://apify.com/scrapyx/guardian-articles-scraper.md
- **Developed by:** [Ibnu Adzim](https://apify.com/scrapyx) (community)
- **Categories:** News
- **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.10 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Guardian Articles & Liveblogs Scraper

Fetches Guardian articles and liveblogs — **with the full body text** — through
the Guardian's own Open Platform Content API.

| | |
| --- | --- |
| **Coverage** | **2,690,015 items** at last check, back to **1899** |
| **Returns** | headline, standfirst, trail text, byline, full body text, word count, contributor / keyword / tone / series tags, section, pillar, dates, thumbnail |
| **Liveblogs** | optional block-by-block update timeline with per-update text, time and contributors |
| **Filtering** | free-text search, section, tag, date range, sort order — all applied **upstream**, not after the fact |
| **Method** | HTTP only, one documented JSON API. No browser, no scraping, no login |

### You need your own API key

Register a **free Developer key** in about 30 seconds at
[open-platform.theguardian.com/access](https://open-platform.theguardian.com/access/).
It arrives by email. Paste it into the `apiKey` input.

| Tier | Cost | Limits | Use |
| --- | --- | --- | --- |
| **Developer** | Free | 60 calls/min, 500 calls/day, includes article text | **Non-commercial only** — student work, research, hackathons, non-profit apps |
| **Commercial** | Paid | Custom | Everything else, including AI/ML training, text and data mining, and any product derived from Guardian content |

**This matters, so it is worth being direct about it.** The Developer tier is
free but explicitly non-commercial. If you are reselling this data, publishing a
paid product built on it, or using it to train models, the Commercial key is the
tier that applies — the Guardian names those use cases on that page. The actor
works identically on either.

### Why the API and not a scraper

The Guardian's `robots.txt` puts `ClaudeBot` and friends under `Disallow: /`,
its header comment rules out LLM/AI and commercial use in prose, and
`theguardian.com/license.xml` is a machine-readable licence permitting AI use
only under a paid subscription. This actor therefore reads their API instead of
their website.

That is also simply better data: ~442 URLs are reachable from the news sitemap,
versus **2.69 million** items here, with real pagination, upstream filtering and
an archive reaching back to 1899.

### Example input

```json
{
  "apiKey": "your-key-here",
  "queries": ["climate change"],
  "sections": ["environment"],
  "contentTypes": ["article", "liveblog"],
  "dateFrom": "2026-01-01",
  "maxItemsPerQuery": 100,
  "includeBody": true
}
```

Filters alone, with no search term, are a complete query:

```json
{
  "apiKey": "your-key-here",
  "tags": ["technology/artificialintelligenceai"],
  "orderBy": "newest",
  "maxItemsPerQuery": 200
}
```

### Output

Every row carries the portfolio envelope — `_input`, `_source`, `_scrapedAt`,
`recordType`. Three record types share the dataset:

- **`ARTICLE`** / **`LIVEBLOG`** — one per item. Same field set; `contentType`
  and `recordType` tell them apart, and a liveblog adds `liveblogUpdates`.
- **`SEARCH_SUMMARY`** — one per query: the API's own `total` and `pages`, how
  many rows came back, how many pages were read, the filters actually applied,
  and **`apiCallsUsed`** so you can see what the run cost against your daily quota.
- **`ERROR`** — one per query that failed, so every input maps to at least one row.

Tags arrive from the API as one flat array mixing contributors, keywords, tone
and series. They are split into `articleContributors`, `articleKeywords`,
`articleTone` and `articleSeries`, with the untouched array kept as `tagsRaw`.

### Watch your quota

500 calls a day is the binding constraint on a Developer key, and this actor is
built around it:

- `maxConcurrency` defaults to **1** — leave it there. The real limit is 60
  calls a minute (the API says so in its own headers, despite the docs saying
  one per second), so concurrency buys nothing and only produces rate-limit
  errors faster.
- `pageSize` is the real lever: at 200 items per call instead of 50, the same
  10,000 items cost 50 calls rather than 200.
- Every run logs your key's **actual** remaining quota, read from the API's
  response headers — not a local guess, so it accounts for calls the same key
  spent in earlier runs today.
- Multiple `sections` or `tags` are combined into **one** upstream call, not one
  call each.
- A run using more than half your daily budget says so in the log.

### Known limits

- **Articles and liveblogs only.** The API also serves video, galleries, audio,
  interactives and crosswords; they are out of scope here by design, and asking
  for one returns a named error rather than failing obscurely.
- **`includeLiveblogUpdates` needs `liveblog` in `contentTypes`** — otherwise
  there are no updates to return, and the summary row says so.
- **Relevance sorting only means something with a search term.**
- **A single query can only reach 38,000 items.** The API refuses any offset
  past that with an HTTP 400, whatever the reported total says. The actor stops
  before the boundary and says so on the summary row. Narrow with a date range
  or a section/tag filter rather than paging deeper.

### Anti-bot

None to speak of — this is a documented developer API, not a scrape. A 401 here
means a key problem, not a block, and the actor never retries one: rotating a
fingerprint cannot fix a rejected credential, and each retry would spend another
call from your daily budget.

# Actor input Schema

## `apiKey` (type: `string`):

Your Guardian Open Platform key. Register a free Developer key in about 30 seconds at https://open-platform.theguardian.com/access/ and it is emailed to you. The Developer tier is FREE but NON-COMMERCIAL (1 call/second, 500 calls/day, includes full article text). Commercial use -- including reselling this data or publishing derived products -- needs a Commercial key from the same page. This actor reads the Guardian's own API rather than scraping theguardian.com, which their robots.txt and license.xml do not permit.

## `queries` (type: `array`):

Optional free-text search. Each term is a separate query with its own SEARCH\_SUMMARY row and its own share of your daily call budget. Leave empty to search by filters alone -- a section, tag or date range with no keyword is a complete query on this API.

## `contentTypes` (type: `array`):

Articles, liveblogs, or both in one run. They share an identical schema on this API -- a liveblog is a content item with type 'liveblog' plus a block-by-block update list -- which is why one actor covers both. The API also serves video, gallery, audio, interactive and crossword; those are out of scope for this actor by design.

## `sections` (type: `array`):

Optional Guardian section IDs, e.g. world, politics, business, technology, sport, environment, football, culture. Several values are combined as OR in ONE upstream call rather than one call each, which matters against a 500-call daily budget.

## `tags` (type: `array`):

Optional Guardian tag IDs, e.g. environment/climate-crisis, technology/artificialintelligenceai, profile/george-monbiot. Tags are the Guardian's richest filter -- every article carries contributor, keyword, tone and series tags. Combined as OR in one call, same as sections.

## `dateFrom` (type: `string`):

Optional lower bound on publication date, inclusive. Applied upstream by the API, not locally, so it narrows the result set rather than filtering rows after the fact.

## `dateTo` (type: `string`):

Optional upper bound on publication date, inclusive.

## `orderBy` (type: `string`):

Newest first, oldest first, or by the API's own relevance score. Relevance is only meaningful when a search term is supplied.

## `includeBody` (type: `boolean`):

Request the complete article body as plain text, plus standfirst, byline and word count. On by default -- full text is the main reason to use this API and the Developer tier includes it. Turning it off makes responses much smaller but costs the same number of API calls.

## `includeLiveblogUpdates` (type: `boolean`):

For liveblogs, also return the block-by-block timeline: each update's title, text, publish time and contributors. Off by default because a long liveblog's update list dwarfs the article payload. Has no effect unless 'liveblog' is in Content types.

## `maxItemsPerQuery` (type: `integer`):

Cap on items returned per search term. Set to 0 for everything the query matches -- but note that at 50 items per call, 10,000 items is 200 of your 500 daily calls.

## `pageSize` (type: `integer`):

How many items each upstream call fetches, up to the API maximum of 200. Higher means fewer calls for the same data, which directly conserves your daily quota; the trade-off is a larger response per call.

## `maxConcurrency` (type: `integer`):

Leave at 1 on a Developer key. That tier allows one call per second, so concurrency buys nothing and only produces rate-limit errors faster. Raise it only on a Commercial key with a higher throttle.

## `minRequestInterval` (type: `number`):

Pacing floor between call starts. The default sits just above the Developer tier's one-call-per-second limit; a start exactly on the second boundary can still land inside the previous second at the server. Lower it only on a Commercial key.

## `proxyConfiguration` (type: `object`):

There is no bot mitigation on this endpoint -- it is the Guardian's own developer API and answers plain JSON to any client with a valid key. Residential is the portfolio default; datacentre would work equally well here.

## Actor input object example

```json
{
  "queries": [
    "climate change"
  ],
  "contentTypes": [
    "article"
  ],
  "sections": [],
  "tags": [],
  "orderBy": "newest",
  "includeBody": true,
  "includeLiveblogUpdates": false,
  "maxItemsPerQuery": 50,
  "pageSize": 50,
  "maxConcurrency": 1,
  "minRequestInterval": 1.1,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ]
  }
}
```

# Actor output Schema

## `items` (type: `string`):

One row per scraped record. See the dataset's default view for field definitions.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "queries": [
        "climate change"
    ],
    "contentTypes": [
        "article"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("scrapyx/guardian-articles-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "queries": ["climate change"],
    "contentTypes": ["article"],
}

# Run the Actor and wait for it to finish
run = client.actor("scrapyx/guardian-articles-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "queries": [
    "climate change"
  ],
  "contentTypes": [
    "article"
  ]
}' |
apify call scrapyx/guardian-articles-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,scrapyx/guardian-articles-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/8XZhWvTYIKDfIBMdk/builds/jZYvD5uDBb2xMV8aT/openapi.json