Sydney Morning Herald (SMH) News Scraper

Scrape news articles from The Sydney Morning Herald (smh.com.au) — no login, no code required. This SMH scraper extracts the headline, summary, author, publish and update timestamps, section, keywords, images and the publicly served article text from every article it visits, and flags whether the story is premium. Paste a section URL such as https://www.smh.com.au/business, press Start, and download structured JSON, CSV or Excel — or call it as an API from Python, JavaScript, Zapier, Make or n8n.

Target website: smh.com.au

✨ Features of this SMH article scraper

• Full article extraction — headline, summary, author, publish and update timestamps, section, keywords, images and the publicly served article text.
• Paywall flagging — each record carries isPaywalled, so premium Sydney Morning Herald stories are easy to filter out.
• Automatic URL detection — section and article URLs are told apart automatically, so you can mix both in Start URLs.
• Whole-site mode — one toggle crawls every major SMH section instead of a hand-picked list.
• Cross-run caching — an optional project name skips articles already scraped in an earlier run.
• Proxy support — optional Apify proxy configuration (datacenter or residential).
• No browser required — built on lightweight HTTP requests, which makes runs fast and cheap.

🚀 How to scrape SMH articles in 5 steps

1. Sign up for a free Apify account — includes $5 monthly credit.
2. Open the actor page and click Try for free.
3. Fill in the Input fields (at least one start URL is required).
4. Click Start and wait for the run to complete.
5. Download results from the Output tab in JSON, CSV, or Excel format.

You can also run this actor via the Apify API or integrate it directly into your workflows using Zapier, Make, or n8n.

💰 Pricing

This actor uses pay-per-result billing based on the compute units a run consumes.

• New Apify accounts include $5 of free monthly credit, which is enough to scrape thousands of SMH articles at no cost.
• Because the scraper uses lightweight HTTP requests rather than a headless browser, it costs significantly less to run than browser-based news scrapers.
• Leaving proxyConfiguration disabled (the default) keeps runs at their cheapest — SMH responds reliably without a proxy.

🔧 Input configuration

Field	Type	Required	Default	Description
startUrls	array	✅	[{"url": "https://www.smh.com.au/business"}]	SMH section or topic listing URLs, or direct article URLs. The type of each URL is detected automatically.
maxItems	integer	—	5	Maximum number of articles to scrape for each start URL.
scrapeWholeSite	boolean	—	false	Ignore Start URLs and crawl the main SMH section pages instead.
cacheProjectName	string	—	—	Optional. When set, articles already scraped under this project name in a previous run are skipped instead of being scraped again.
proxyConfiguration	object	—	{"useApifyProxy": false}	SMH responds fine without a proxy. Enable datacenter proxies first if you start seeing blocks, and residential only if datacenter still fails.

Supported URL types:

• Listing pages — section and topic pages such as https://www.smh.com.au/business or https://www.smh.com.au/politics/federal.
• Article pages — URLs ending in a date and public id, e.g. https://www.smh.com.au/business/markets/some-slug-20260910-p60w1s.html.

📦 SMH scraper output data

Structured JSON (exportable to CSV/Excel) with one record per article, containing: url, title, description, authors, publisher, publishedAt, modifiedAt, category, section, isPaywalled, keywords, images, body and wordCount.

The dataset has one view, Overview, a compact table of each article's title, section, authors, publish date, paywall status and URL.

Sample output:

{
  "url": "https://www.smh.com.au/technology/apple-unveils-its-foldable-iphone-duo-20260909-p60vzj.html",
  "title": "Apple unveils its foldable iPhone Duo",
  "description": "New Apple CEO John Ternus said the Duo has the “largest display ever on an iPhone, yet fits into your pocket”, claiming it will redefine the experience of using a foldable smartphone.",
  "authors": [
    "David Swan"
  ],
  "publisher": "The Sydney Morning Herald",
  "publishedAt": "2026-09-09T18:30:26Z",
  "modifiedAt": "2026-09-09T20:41:13Z",
  "category": "Phones",
  "section": "Home",
  "isPaywalled": true,
  "keywords": [
    "Phones",
    "Just in",
    "Apple",
    "Gadgets",
    "AI"
  ],
  "images": [
    "https://static.ffx.io/images/$zoom_1.3238%2C$multiply_0.7025%2C$ratio_1.777778%2C$width_1059%2C$x_0%2C$y_53/t_crop_custom/q_86%2Cf_auto/072c2959dafe46a3e7b32cdf363153be6476964ca32322a2e396a149e0edb196"
  ],
  "body": "Apple will charge Australians $3499 for its first folding iPhone, unveiled on Thursday morning AEST, making it the most expensive phone the company has released.\n\nThe company’s new chief executive, John Ternus, who took over from Tim Cook on September 1, introduced the latest lineup at the Steve Jobs Theatre.\n\nAfter cycling through other updates to the iPhone, the Apple Watch and AirPods along wit …",
  "wordCount": 941
}

🐍 How to scrape SMH with Python, JavaScript or the API

Run the actor programmatically with the official Apify clients. Replace <YOUR_API_TOKEN> with the token from your Apify Console.

Python (pip install apify-client):

from apify_client import ApifyClient

client = ApifyClient("<YOUR_API_TOKEN>")

run = client.actor("confidential_gnat/smh-news-scraper").call(run_input={
    "startUrls": [{"url": "https://www.smh.com.au/business"}],
    "maxItems": 10,
})

for article in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(article["publishedAt"], article["title"])

JavaScript (npm install apify-client):

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_API_TOKEN>' });

const run = await client.actor('confidential_gnat/smh-news-scraper').call({
    startUrls: [{ url: 'https://www.smh.com.au/business' }],
    maxItems: 10,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((article) => console.log(article.publishedAt, article.title));

cURL — start a run and wait for the dataset:

curl -X POST "https://api.apify.com/v2/acts/confidential_gnat~smh-news-scraper/run-sync-get-dataset-items?token=<YOUR_API_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://www.smh.com.au/business"}],"maxItems":10}'

💡 What you can use SMH news data for

• Tracking how a topic, company or public figure is covered over time
• Monitoring Australian market and business coverage
• Building a searchable archive of headlines and summaries
• Competitive and brand monitoring
• Feeding dashboards, internal newsletters or an LLM/RAG pipeline
• Academic and market research on Australian media coverage

Typical users are media monitoring teams, analysts and researchers who need a structured, machine-readable feed of Australian news rather than reading and copying stories by hand — across media & entertainment, market research, financial services and academic research.

⚠️ Limitations & known issues

• Premium articles — stories marked isPaywalled: true are captured only as far as SMH serves them publicly; the actor does not log in to retrieve subscriber-only text.
• Section pages are not paginated — an SMH section returns a fixed window of current stories (around 30–35), and requesting ?page=2 returns the same set, so a single section URL cannot reach further back in the archive. Use several section or topic URLs to widen coverage.

❓ Frequently asked questions

Can I scrape Sydney Morning Herald articles legally?

This actor only collects what The Sydney Morning Herald already serves publicly to an ordinary visitor — it does not log in, supply subscriber credentials, or bypass any access control. Scraping publicly available data is generally considered lawful (see hiQ Labs v. LinkedIn as precedent). You remain responsible for complying with SMH's Terms of Service, with copyright in the article text, and with any applicable privacy laws.

Does this scraper get paywalled SMH articles?

No. Premium stories are captured only as far as smh.com.au returns them without a subscription, and every record carries an isPaywalled flag so you can filter them out.

How many articles can I scrape from one SMH section?

An SMH section page serves a fixed window of roughly 30–35 current stories and does not paginate, so one section URL cannot reach deeper into the archive. Add several section or topic URLs to widen coverage.

Do I need a proxy to scrape smh.com.au?

No. SMH responds reliably without one, and proxyConfiguration defaults to disabled to keep your runs cheap. If you start seeing blocks, enable datacenter proxies first and residential only if datacenter still fails.

How do I scrape SMH news with Python?

Install apify-client, then call the actor with your API token — see the Python example above. The same run is available in JavaScript and via a single cURL request.

Can I run this SMH scraper on a schedule?

Yes. Use Apify Schedules to run it hourly or daily, and set cacheProjectName so articles already scraped in an earlier run are skipped.

What output formats are supported?

JSON, CSV, Excel, XML and RSS, either from the Output tab or through the Apify API.

🔗 Other actors you may find useful

• 📰 Detik News AI Scraper — Scrapes news articles from Detik.com, including headline, author, publish date, category, images and full article text. Additionally use AI to paraphrase, summarize and get sentiment analysis value.
• 🌐 WHOIS Domain Lookup Scraper — Bulk WHOIS lookups via whois.com. Accepts URLs, bare domains, or whois.com lookup links, and returns registration dates, registrar, name servers, status codes, abuse contacts, registrant/technical contact details, and similar domain suggestions.
• 🧴 Incidecoder Scraper — Scrapes skincare product pages from INCIDecoder, extracting full ingredient lists, highlights, key/other ingredient groupings by function, per-ingredient ratings, and detailed explanations.
• 🎟️ Gametime Events Website Scraper — Scrapes event, venue, and ticket offer data from Gametime. Supports search, group/team, and single-event URLs, returning schedule, location, performer, and pricing details — including per-offer section, row, seat count, and deal rating — as structured JSON with pagination support.
• ⚖️ Austria Ediktsdatei Scraper — Scrapes Austrian judicial auction listings (Gerichtliche Versteigerungen) from edikte.justiz.gv.at, including property details, dates, valuations and PDF documents.

💬 Support & contact

If you encounter any issues or have questions, please open an issue.

You can also find more of our actors on the Apify Store.