Goodreads Books Scraper (Listopia)

Scrapes books from Goodreads Listopia lists — community-curated rankings like Best Books Ever, Best Fantasy of the 2020s or any genre/award list. HTTP-only, no browser.

What you get

One row per book, plus a LIST_SUMMARY row per list and an ERROR row for any input that fails — so every input maps to at least one row.

Field	Example
bookId	2767052
title	The Hunger Games (The Hunger Games, #1)
author	Suzanne Collins
avgRating	4.35
ratingsCount	10232595
listRank	1 (position in this list)
listScore	4518690 (Listopia vote score)
bookUrl / coverImage	full URLs

Optional detail pass

includeBookDetails: true fetches each book's page and attaches its JSON-LD Book under bookDetail, adding aggregateRating, numberOfPages, bookFormat, inLanguage, awards and the full author list — plus, under bookDetail.goodreadsExtras, the description and genres, which the JSON-LD does not carry. One extra request per book, so it is off by default.

Input

{
  "listUrls": [
    "https://www.goodreads.com/list/show/1.Best_Books_Ever"
  ],
  "maxItems": 100,
  "includeBookDetails": false
}

Open any Listopia list on Goodreads and paste its URL. Lists get large — Best Books Ever holds ~79,000 books at 100 per page — so maxItems and maxPages are the cost controls.

Why list URLs instead of a search box

Goodreads' robots.txt disallows /search, /work, /review/show and /book/reviews/ for generic crawlers. Listopia (/list/show/…), book pages and author pages are not disallowed, and Goodreads publishes siteindex.list.xml advertising those lists for crawling.

So this actor takes list URLs and refuses a pasted /search URL rather than quietly fetching a surface the site asks crawlers to stay off. Reviews are not scraped at all, for the same reason.

Known limits

Pagination ends by repetition, not emptiness. Past a long list's real end, Goodreads re-serves the last page forever (pages 790–2000 of list 1 all returned the identical 100 books). This actor dedups by book id and stops when a page yields no new ids, reporting paginationClampHit. Short lists do end honestly with an empty page; both paths are handled.

listTotalBooks is the list's declared size, not the number of rows you will get — that depends on maxItems/maxPages and where the clamp begins.

A dead book id answers HTTP 503, not 404. Handled: such a book keeps its list row with bookDetail: null rather than failing the run.

Anti-bot / transport

No WAF, challenge page or JS gate was observed — 8 TLS profiles returned clean 200s cold, without warmup. List pages are large (~770 KB), and two transient read timeouts appeared during deep-pagination probing, so the client keeps a wide profile pool and rotates on retry rather than hammering one profile. Residential proxy is the default, without a country pin.