# Sydney Morning Herald (SMH) News Scraper (`confidential_gnat/smh-news-scraper`) Actor

Scrape news articles from The Sydney Morning Herald (smh.com.au) — headline, author, publish date, section, keywords, images and full public article text, with a paywall flag. Export to JSON, CSV or Excel, or call it as an API.

- **URL**: https://apify.com/confidential\_gnat/smh-news-scraper.md
- **Developed by:** [ActorFlow](https://apify.com/confidential_gnat) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $3.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Sydney Morning Herald (SMH) News Scraper

**Scrape news articles from The Sydney Morning Herald (smh.com.au)** — no login, no code required. This **SMH scraper** extracts the headline, summary, author, publish and update timestamps, section, keywords, images and the publicly served article text from every article it visits, and flags whether the story is premium. Paste a section URL such as `https://www.smh.com.au/business`, press **Start**, and download structured **JSON, CSV or Excel** — or call it as an **API** from Python, JavaScript, Zapier, Make or n8n.

**Target website:** [smh.com.au](https://www.smh.com.au)

### ✨ Features of this SMH article scraper

- **Full article extraction** — headline, summary, author, publish and update timestamps, section, keywords, images and the publicly served article text.
- **Paywall flagging** — each record carries `isPaywalled`, so premium Sydney Morning Herald stories are easy to filter out.
- **Automatic URL detection** — section and article URLs are told apart automatically, so you can mix both in Start URLs.
- **Whole-site mode** — one toggle crawls every major SMH section instead of a hand-picked list.
- **Cross-run caching** — an optional project name skips articles already scraped in an earlier run.
- **Proxy support** — optional Apify proxy configuration (datacenter or residential).
- **No browser required** — built on lightweight HTTP requests, which makes runs fast and cheap.

### 🚀 How to scrape SMH articles in 5 steps

1. [Sign up](https://apify.com/sign-up) for a free Apify account — includes **$5 monthly credit**.
2. Open the actor page and click **Try for free**.
3. Fill in the **Input** fields (at least one start URL is required).
4. Click **Start** and wait for the run to complete.
5. Download results from the **Output** tab in JSON, CSV, or Excel format.

You can also run this actor via the [Apify API](https://docs.apify.com/api/v2) or integrate it directly into your workflows using [Zapier](https://zapier.com/apps/apify), [Make](https://www.make.com/), or [n8n](https://n8n.io/).

### 💰 Pricing

This actor uses **pay-per-result** billing based on the compute units a run consumes.

- New Apify accounts include **$5 of free monthly credit**, which is enough to scrape thousands of SMH articles at no cost.
- Because the scraper uses lightweight HTTP requests rather than a headless browser, it costs significantly less to run than browser-based news scrapers.
- Leaving `proxyConfiguration` disabled (the default) keeps runs at their cheapest — SMH responds reliably without a proxy.

### 🔧 Input configuration

| Field | Type | Required | Default | Description |
| ----- | ---- | -------- | ------- | ----------- |
| `startUrls` | array | ✅ | `[{"url": "https://www.smh.com.au/business"}]` | SMH section or topic listing URLs, or direct article URLs. The type of each URL is detected automatically. |
| `maxItems` | integer | — | `5` | Maximum number of articles to scrape for each start URL. |
| `scrapeWholeSite` | boolean | — | `false` | Ignore Start URLs and crawl the main SMH section pages instead. |
| `cacheProjectName` | string | — | — | Optional. When set, articles already scraped under this project name in a previous run are skipped instead of being scraped again. |
| `proxyConfiguration` | object | — | `{"useApifyProxy": false}` | SMH responds fine without a proxy. Enable datacenter proxies first if you start seeing blocks, and residential only if datacenter still fails. |

**Supported URL types:**

- **Listing pages** — section and topic pages such as `https://www.smh.com.au/business` or `https://www.smh.com.au/politics/federal`.
- **Article pages** — URLs ending in a date and public id, e.g. `https://www.smh.com.au/business/markets/some-slug-20260910-p60w1s.html`.

### 📦 SMH scraper output data

Structured JSON (exportable to CSV/Excel) with one record per article, containing: `url`, `title`, `description`, `authors`, `publisher`, `publishedAt`, `modifiedAt`, `category`, `section`, `isPaywalled`, `keywords`, `images`, `body` and `wordCount`.

The dataset has one view, **Overview**, a compact table of each article's title, section, authors, publish date, paywall status and URL.

**Sample output:**

```json
{
  "url": "https://www.smh.com.au/technology/apple-unveils-its-foldable-iphone-duo-20260909-p60vzj.html",
  "title": "Apple unveils its foldable iPhone Duo",
  "description": "New Apple CEO John Ternus said the Duo has the “largest display ever on an iPhone, yet fits into your pocket”, claiming it will redefine the experience of using a foldable smartphone.",
  "authors": [
    "David Swan"
  ],
  "publisher": "The Sydney Morning Herald",
  "publishedAt": "2026-09-09T18:30:26Z",
  "modifiedAt": "2026-09-09T20:41:13Z",
  "category": "Phones",
  "section": "Home",
  "isPaywalled": true,
  "keywords": [
    "Phones",
    "Just in",
    "Apple",
    "Gadgets",
    "AI"
  ],
  "images": [
    "https://static.ffx.io/images/$zoom_1.3238%2C$multiply_0.7025%2C$ratio_1.777778%2C$width_1059%2C$x_0%2C$y_53/t_crop_custom/q_86%2Cf_auto/072c2959dafe46a3e7b32cdf363153be6476964ca32322a2e396a149e0edb196"
  ],
  "body": "Apple will charge Australians $3499 for its first folding iPhone, unveiled on Thursday morning AEST, making it the most expensive phone the company has released.\n\nThe company’s new chief executive, John Ternus, who took over from Tim Cook on September 1, introduced the latest lineup at the Steve Jobs Theatre.\n\nAfter cycling through other updates to the iPhone, the Apple Watch and AirPods along wit …",
  "wordCount": 941
}
```

### 🐍 How to scrape SMH with Python, JavaScript or the API

Run the actor programmatically with the official Apify clients. Replace `<YOUR_API_TOKEN>` with the token from your [Apify Console](https://console.apify.com/account/integrations).

**Python** (`pip install apify-client`):

```python
from apify_client import ApifyClient

client = ApifyClient("<YOUR_API_TOKEN>")

run = client.actor("confidential_gnat/smh-news-scraper").call(run_input={
    "startUrls": [{"url": "https://www.smh.com.au/business"}],
    "maxItems": 10,
})

for article in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(article["publishedAt"], article["title"])
```

**JavaScript** (`npm install apify-client`):

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_API_TOKEN>' });

const run = await client.actor('confidential_gnat/smh-news-scraper').call({
    startUrls: [{ url: 'https://www.smh.com.au/business' }],
    maxItems: 10,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((article) => console.log(article.publishedAt, article.title));
```

**cURL** — start a run and wait for the dataset:

```bash
curl -X POST "https://api.apify.com/v2/acts/confidential_gnat~smh-news-scraper/run-sync-get-dataset-items?token=<YOUR_API_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://www.smh.com.au/business"}],"maxItems":10}'
```

### 💡 What you can use SMH news data for

- Tracking how a topic, company or public figure is covered over time
- Monitoring Australian market and business coverage
- Building a searchable archive of headlines and summaries
- Competitive and brand monitoring
- Feeding dashboards, internal newsletters or an LLM/RAG pipeline
- Academic and market research on Australian media coverage

Typical users are media monitoring teams, analysts and researchers who need a structured, machine-readable feed of Australian news rather than reading and copying stories by hand — across media & entertainment, market research, financial services and academic research.

### ⚠️ Limitations & known issues

- **Premium articles** — stories marked `isPaywalled: true` are captured only as far as SMH serves them publicly; the actor does not log in to retrieve subscriber-only text.
- **Section pages are not paginated** — an SMH section returns a fixed window of current stories (around 30–35), and requesting `?page=2` returns the same set, so a single section URL cannot reach further back in the archive. Use several section or topic URLs to widen coverage.

### ❓ Frequently asked questions

#### Can I scrape Sydney Morning Herald articles legally?

This actor only collects what The Sydney Morning Herald already serves publicly to an ordinary visitor — it does not log in, supply subscriber credentials, or bypass any access control. Scraping publicly available data is generally considered lawful (see *hiQ Labs v. LinkedIn* as precedent). You remain responsible for complying with SMH's Terms of Service, with copyright in the article text, and with any applicable privacy laws.

#### Does this scraper get paywalled SMH articles?

No. Premium stories are captured only as far as smh.com.au returns them without a subscription, and every record carries an `isPaywalled` flag so you can filter them out.

#### How many articles can I scrape from one SMH section?

An SMH section page serves a fixed window of roughly 30–35 current stories and does not paginate, so one section URL cannot reach deeper into the archive. Add several section or topic URLs to widen coverage.

#### Do I need a proxy to scrape smh.com.au?

No. SMH responds reliably without one, and `proxyConfiguration` defaults to disabled to keep your runs cheap. If you start seeing blocks, enable datacenter proxies first and residential only if datacenter still fails.

#### How do I scrape SMH news with Python?

Install `apify-client`, then call the actor with your API token — see the Python example above. The same run is available in JavaScript and via a single cURL request.

#### Can I run this SMH scraper on a schedule?

Yes. Use Apify [Schedules](https://docs.apify.com/platform/schedules) to run it hourly or daily, and set `cacheProjectName` so articles already scraped in an earlier run are skipped.

#### What output formats are supported?

JSON, CSV, Excel, XML and RSS, either from the **Output** tab or through the Apify API.

### 🔗 Other actors you may find useful

- 📰 **[Detik News AI Scraper](https://apify.com/confidential_gnat/detik-news-scraper)** — Scrapes news articles from Detik.com, including headline, author, publish date, category, images and full article text. Additionally use AI to paraphrase, summarize and get sentiment analysis value.
- 🌐 **[WHOIS Domain Lookup Scraper](https://apify.com/confidential_gnat/whois-com)** — Bulk WHOIS lookups via whois.com. Accepts URLs, bare domains, or whois.com lookup links, and returns registration dates, registrar, name servers, status codes, abuse contacts, registrant/technical contact details, and similar domain suggestions.
- 🧴 **[Incidecoder Scraper](https://apify.com/confidential_gnat/incidecoder-scraper)** — Scrapes skincare product pages from INCIDecoder, extracting full ingredient lists, highlights, key/other ingredient groupings by function, per-ingredient ratings, and detailed explanations.
- 🎟️ **[Gametime Events Website Scraper](https://apify.com/confidential_gnat/gametime-events-scraper)** — Scrapes event, venue, and ticket offer data from Gametime. Supports search, group/team, and single-event URLs, returning schedule, location, performer, and pricing details — including per-offer section, row, seat count, and deal rating — as structured JSON with pagination support.
- ⚖️ **[Austria Ediktsdatei Scraper](https://apify.com/confidential_gnat/austria-ediktsdatei-scraper)** — Scrapes Austrian judicial auction listings (Gerichtliche Versteigerungen) from edikte.justiz.gv.at, including property details, dates, valuations and PDF documents.

### 💬 Support & contact

If you encounter any issues or have questions, please [open an issue](https://apify.com/confidential_gnat/smh-news-scraper/issues/open).

You can also find more of our actors on the [Apify Store](https://apify.com/confidential_gnat).

# Actor input Schema

## `startUrls` (type: `array`):

SMH section or topic listing URLs, or direct article URLs. The type of each URL is detected automatically.

## `maxItems` (type: `integer`):

Maximum number of articles to scrape for each start URL.

## `scrapeWholeSite` (type: `boolean`):

Ignore Start URLs and crawl the main SMH section pages instead.

## `cacheProjectName` (type: `string`):

Optional. When set, articles already scraped under this project name in a previous run are skipped instead of being scraped again.

## `proxyConfiguration` (type: `object`):

SMH responds fine without a proxy. Enable datacenter proxies first if you start seeing blocks, and residential only if datacenter still fails.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://www.smh.com.au/business"
    }
  ],
  "maxItems": 5,
  "scrapeWholeSite": false,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://www.smh.com.au/business"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("confidential_gnat/smh-news-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://www.smh.com.au/business" }] }

# Run the Actor and wait for it to finish
run = client.actor("confidential_gnat/smh-news-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://www.smh.com.au/business"
    }
  ]
}' |
apify call confidential_gnat/smh-news-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,confidential_gnat/smh-news-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/6Pu8Q5yf3udm63v82/builds/7Mcq2AwWv0ZRl6Edg/openapi.json