Mojeek News Feed Scraper

Collect the current public news feed displayed at https://www.mojeek.com/news in one fast HTTP request. The Actor extracts every unique article that has a public heading link across the hero, related-story, and standard sections, then saves a normalized dataset suitable for APIs, spreadsheets, monitoring, and downstream analysis.

Important behavior

Mojeek's current News page is a feed, not a topic-search endpoint. During live validation, adding q and category-like URL parameters returned the same feed rather than different search results. Version 3 therefore does not claim that it searches Mojeek News or paginates unavailable pages. Optional keyword and category inputs filter the retrieved feed locally. Legacy query and category inputs remain supported as aliases, but they are also local filters.

This design keeps results accurate: a record is never labelled as matching a server-side query that Mojeek did not execute.

Input

Field	Type	Default	Purpose
filterKeyword	string	none	Case-insensitive match against title, snippet, publisher, category, and source line.
categories	string[]	none	Exact, case-insensitive category filters, for example World.
query	string	none	Backward-compatible alias for filterKeyword.
category	string	none	Backward-compatible single-category alias.
maxItems	integer	50	Maximum matching records, from 1 to 100. The live feed may contain fewer.
maxRequestRetries	integer	1	Bounded retries for temporary failures.
navigationTimeoutSecs	integer	30	HTTP request timeout.
requestHandlerTimeoutSecs	integer	60	Extraction timeout.
proxyConfiguration	object	direct	Optional Apify Proxy or custom proxy URLs.

GOOGLE_SERP cannot be used because it is limited to supported Google Search and Shopping requests; it is not a general-purpose proxy for Mojeek.

Example:

{
  "categories": ["World", "Science"],
  "maxItems": 10,
  "proxyConfiguration": { "useApifyProxy": false }
}

Use maxItems: 1 when only the first current article is needed. A filter with no matches completes successfully with an empty dataset.

Output fields

Every record includes a stable URL-derived id, feed and filtered positions, title, canonical article URL, article domain and host, feed section, provider, retrieval method, source flags, run metadata, and ISO-8601 scrape time. When Mojeek displays them, the Actor also includes:

• publisher and category parsed from the visible source line;
• visible publication text and an estimated ISO timestamp for relative values such as “2 hours ago”;
• public snippet text;
• original image URL decoded from Mojeek's image proxy, the public proxy URL, alt text, and dimensions.

Optional values that are absent on the website are omitted. They are not emitted as null, empty strings, empty arrays, or invented placeholders. The Actor intentionally does not create guessed publisher homepages, Google favicons, author names, ad flags, or duplicate URL aliases.

Performance and reliability

Version 3 uses CheerioCrawler instead of launching Playwright. The Actor makes one request with a consistent HTTP profile, validates the status and HTML content type, rejects access-denied and challenge documents, deduplicates by canonical article URL, and buffers records until the request succeeds. This avoids saving partial data after a terminal failure.

Direct access is the default because it is typically the fastest for this public feed. Apify Proxy and user-provided proxy URLs are available when ordinary regional or network routing requires them. Proxy credentials and sensitive headers are never written to logs or datasets. The Actor does not solve or bypass CAPTCHAs; a challenge causes a clear failed run rather than contaminated output.

Running locally

Install the Apify CLI, then run:

npm ci
npm test
apify run --purge --input-file INPUT.json
npm run audit:dataset

The dataset is written under storage/datasets/default. The included audit checks required fields, types, duplicates, null and empty values, URLs, timestamps, challenge content, and sensitive field names.

Using results

Download the default dataset as JSON, CSV, Excel, XML, or another Apify-supported format, or consume it through the dataset API. Scheduled runs can be used to snapshot the current feed over time. Because the source page has no observed feed pagination, each run represents only the articles currently exposed by Mojeek; long-term history should be accumulated by the caller or an Apify schedule.

Respect the target website's terms, robots guidance, copyright, and applicable law. Article URLs and snippets remain the property of their respective publishers.