TVTropes Scraper

Apify Actor for scraping TVTropes.org — the massive wiki of storytelling tropes in media. Uses PlaywrightCrawler to handle Cloudflare protection and Google Custom Search rendering.

Features

• Search TVTropes by keyword (trope names, work titles, characters)
• Extract full trope/work page details: descriptions, examples, related tropes
• Parse examples organized by media type (Film, Literature, VideoGames, etc.)
• Fetch laconic (one-line) summaries for each trope
• Track related trope links with namespace categorization
• Two modes: search results only or full detail scraping

Input

Field	Type	Default	Description
searchQueries	string[]	["Batman"]	Trope or work names to search
maxListings	integer	100	Max pages per query (0=unlimited)
scrapeDetails	boolean	true	Scrape full page details
proxyConfiguration	object	{ useApifyProxy: true }	Proxy settings (residential recommended)

Example Input

{
    "searchQueries": ["Chekhov Gun", "The Matrix", "Batman"],
    "maxListings": 10,
    "scrapeDetails": true,
    "proxyConfiguration": { "useApifyProxy": true }
}

Output

Detail mode (scrapeDetails: true)

{
    "title": "Chekhov's Gun",
    "namespace": "Main",
    "pageName": "ChekhovsGun",
    "url": "https://tvtropes.org/pmwiki/pmwiki.php/Main/ChekhovsGun",
    "description": "Russian writer Anton Chekhov, master of the short story, gave this advice: If it's not essential, leave it out of the narrative...",
    "laconicSummary": "An object that isn't significant until later in the story.",
    "relatedTropes": [
        {
            "title": "Foreshadowing",
            "url": "https://tvtropes.org/pmwiki/pmwiki.php/Main/Foreshadowing",
            "namespace": "Main"
        },
        {
            "title": "Red Herring",
            "url": "https://tvtropes.org/pmwiki/pmwiki.php/Main/RedHerring",
            "namespace": "Main"
        }
    ],
    "relatedTropeCount": 45,
    "examples": [
        {
            "text": "In Shaun of the Dead, the Winchester rifle above the bar...",
            "work": "Shaun of the Dead",
            "workUrl": "https://tvtropes.org/pmwiki/pmwiki.php/Film/ShaunOfTheDead"
        }
    ],
    "exampleCount": 150,
    "folderLabels": ["Animation", "Audio Play", "Comic Books"],
    "searchQuery": "Chekhov Gun",
    "scrapedAt": "2026-04-25T12:00:00.000Z"
}

Search-only mode (scrapeDetails: false)

{
    "title": "Batman (Comic Book)",
    "namespace": "ComicBook",
    "pageName": "Batman",
    "url": "https://tvtropes.org/pmwiki/pmwiki.php/ComicBook/Batman",
    "description": "Batman is one of the three best-known superheroes ever...",
    "searchQuery": "Batman",
    "scrapedAt": "2026-04-25T12:00:00.000Z"
}

TVTropes Namespaces

Pages are organized by namespace which indicates the type of content:

Namespace	Description
Main	Trope definitions
Film	Movies
Series	TV series
Literature	Books and novels
ComicBook	Comic books
VideoGame	Video games
WesternAnimation	Western animated shows
Anime	Japanese animation
Manga	Japanese comics
Music	Music and musicians
Theatre	Stage plays
Franchise	Multi-media franchises
Characters	Character pages
Creator	Writers, directors, actors
Laconic	One-line trope summaries

How It Works

1. Searches TVTropes using Google Custom Search (CSE) embedded on the site
2. Extracts search result links, titles, and snippets
3. Optionally navigates to each result page for full content extraction
4. Parses the #main-article container for descriptions, examples, and trope links
5. Fetches the Laconic version of each page for a one-line summary
6. Examples are extracted from .folder containers (organized by media type)
7. Related trope links are extracted from a.twikilink elements

Notes

• TVTropes uses Cloudflare protection, so PlaywrightCrawler with a real browser is required
• Residential proxies are recommended for reliable access on Apify cloud
• Google CSE returns up to 10 results per page
• Large trope pages (like Chekhov's Gun) can have 100+ examples across many media categories

Quick Start

$apify run --purge

Deploy to Apify

apify login
apify push

---

Run on Apify

This scraper runs on the Apify platform — a full-stack web scraping and automation cloud. Sign up for a free account to get started with 30-day trial of all features.

Try Apify free

Related Scrapers

More scrapers and data tools by lulzasaur:

• AbeBooks Scraper — Rare and used books
• Bonanza Scraper — Online marketplace listings
• Goodreads Scraper — Book ratings and reviews
• IMDb Scraper — Movie and TV show data
• PSA Population Report — Card grading data
• Reverb Scraper — Music gear marketplace
• TCGPlayer Scraper — Trading card prices