# TVTropes Scraper - Tropes, Works & Media Wiki (`lulzasaur/tvtropes-scraper`) Actor Scrape TVTropes.org wiki pages for storytelling tropes and media analysis. Search by keyword, extract trope descriptions, related tropes, examples by media type, and laconic summaries. Perfect for narrative research, media analysis, and creative writing. - **URL**: https://apify.com/lulzasaur/tvtropes-scraper.md - **Developed by:** [lulz bot](https://apify.com/lulzasaur) (community) - **Categories:** E-commerce - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $10.00 / 1,000 results This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## TVTropes Scraper Apify Actor for scraping TVTropes.org — the massive wiki of storytelling tropes in media. Uses PlaywrightCrawler to handle Cloudflare protection and Google Custom Search rendering. ### Features - Search TVTropes by keyword (trope names, work titles, characters) - Extract full trope/work page details: descriptions, examples, related tropes - Parse examples organized by media type (Film, Literature, VideoGames, etc.) - Fetch laconic (one-line) summaries for each trope - Track related trope links with namespace categorization - Two modes: search results only or full detail scraping ### Input | Field | Type | Default | Description | |-------|------|---------|-------------| | `searchQueries` | string[] | `["Batman"]` | Trope or work names to search | | `maxListings` | integer | `100` | Max pages per query (0=unlimited) | | `scrapeDetails` | boolean | `true` | Scrape full page details | | `proxyConfiguration` | object | `{ useApifyProxy: true }` | Proxy settings (residential recommended) | #### Example Input ```json { "searchQueries": ["Chekhov Gun", "The Matrix", "Batman"], "maxListings": 10, "scrapeDetails": true, "proxyConfiguration": { "useApifyProxy": true } } ```` ### Output #### Detail mode (`scrapeDetails: true`) ```json { "title": "Chekhov's Gun", "namespace": "Main", "pageName": "ChekhovsGun", "url": "https://tvtropes.org/pmwiki/pmwiki.php/Main/ChekhovsGun", "description": "Russian writer Anton Chekhov, master of the short story, gave this advice: If it's not essential, leave it out of the narrative...", "laconicSummary": "An object that isn't significant until later in the story.", "relatedTropes": [ { "title": "Foreshadowing", "url": "https://tvtropes.org/pmwiki/pmwiki.php/Main/Foreshadowing", "namespace": "Main" }, { "title": "Red Herring", "url": "https://tvtropes.org/pmwiki/pmwiki.php/Main/RedHerring", "namespace": "Main" } ], "relatedTropeCount": 45, "examples": [ { "text": "In Shaun of the Dead, the Winchester rifle above the bar...", "work": "Shaun of the Dead", "workUrl": "https://tvtropes.org/pmwiki/pmwiki.php/Film/ShaunOfTheDead" } ], "exampleCount": 150, "folderLabels": ["Animation", "Audio Play", "Comic Books"], "searchQuery": "Chekhov Gun", "scrapedAt": "2026-04-25T12:00:00.000Z" } ``` #### Search-only mode (`scrapeDetails: false`) ```json { "title": "Batman (Comic Book)", "namespace": "ComicBook", "pageName": "Batman", "url": "https://tvtropes.org/pmwiki/pmwiki.php/ComicBook/Batman", "description": "Batman is one of the three best-known superheroes ever...", "searchQuery": "Batman", "scrapedAt": "2026-04-25T12:00:00.000Z" } ``` ### TVTropes Namespaces Pages are organized by namespace which indicates the type of content: | Namespace | Description | |-----------|-------------| | `Main` | Trope definitions | | `Film` | Movies | | `Series` | TV series | | `Literature` | Books and novels | | `ComicBook` | Comic books | | `VideoGame` | Video games | | `WesternAnimation` | Western animated shows | | `Anime` | Japanese animation | | `Manga` | Japanese comics | | `Music` | Music and musicians | | `Theatre` | Stage plays | | `Franchise` | Multi-media franchises | | `Characters` | Character pages | | `Creator` | Writers, directors, actors | | `Laconic` | One-line trope summaries | ### How It Works 1. Searches TVTropes using Google Custom Search (CSE) embedded on the site 2. Extracts search result links, titles, and snippets 3. Optionally navigates to each result page for full content extraction 4. Parses the `#main-article` container for descriptions, examples, and trope links 5. Fetches the Laconic version of each page for a one-line summary 6. Examples are extracted from `.folder` containers (organized by media type) 7. Related trope links are extracted from `a.twikilink` elements ### Notes - TVTropes uses Cloudflare protection, so PlaywrightCrawler with a real browser is required - Residential proxies are recommended for reliable access on Apify cloud - Google CSE returns up to 10 results per page - Large trope pages (like Chekhov's Gun) can have 100+ examples across many media categories ### Quick Start ```bash apify run --purge ``` ### Deploy to Apify ```bash apify login apify push ``` *** ### Run on Apify This scraper runs on the [Apify platform](https://apify.com/?fpr=lulzasaur) — a full-stack web scraping and automation cloud. Sign up for a free account to get started with 30-day trial of all features. [Try Apify free](https://apify.com/?fpr=lulzasaur) ### Related Scrapers More scrapers and data tools by [lulzasaur](https://apify.com/lulzasaur): - [AbeBooks Scraper](https://apify.com/lulzasaur/abebooks-scraper) — Rare and used books - [Bonanza Scraper](https://apify.com/lulzasaur/bonanza-scraper) — Online marketplace listings - [Goodreads Scraper](https://apify.com/lulzasaur/goodreads-scraper) — Book ratings and reviews - [IMDb Scraper](https://apify.com/lulzasaur/imdb-scraper) — Movie and TV show data - [PSA Population Report](https://apify.com/lulzasaur/psa-pop-scraper) — Card grading data - [Reverb Scraper](https://apify.com/lulzasaur/reverb-scraper) — Music gear marketplace - [TCGPlayer Scraper](https://apify.com/lulzasaur/tcgplayer-scraper) — Trading card prices # Actor input Schema ## `searchQueries` (type: `array`): Trope or work names to search for on TVTropes (e.g. 'Batman', 'Chekhov Gun', 'The Matrix') ## `maxListings` (type: `integer`): Maximum number of wiki pages to scrape per search query (0 = unlimited) ## `scrapeDetails` (type: `boolean`): Follow search result links and scrape full trope/work pages with descriptions, examples, and related tropes ## `proxyConfiguration` (type: `object`): Proxy settings — residential proxies recommended to bypass Cloudflare protection ## Actor input object example ```json { "searchQueries": [ "Batman" ], "maxListings": 100, "scrapeDetails": true, "proxyConfiguration": { "useApifyProxy": true } } ``` # Actor output Schema ## `results` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "searchQueries": [ "Batman" ] }; // Run the Actor and wait for it to finish const run = await client.actor("lulzasaur/tvtropes-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "searchQueries": ["Batman"] } # Run the Actor and wait for it to finish run = client.actor("lulzasaur/tvtropes-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "searchQueries": [ "Batman" ] }' | apify call lulzasaur/tvtropes-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=lulzasaur/tvtropes-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "TVTropes Scraper - Tropes, Works & Media Wiki", "description": "Scrape TVTropes.org wiki pages for storytelling tropes and media analysis. Search by keyword, extract trope descriptions, related tropes, examples by media type, and laconic summaries. Perfect for narrative research, media analysis, and creative writing.", "version": "1.0", "x-build-id": "qma9RRP8ybRwF96J6" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/lulzasaur~tvtropes-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-lulzasaur-tvtropes-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/lulzasaur~tvtropes-scraper/runs": { "post": { "operationId": "runs-sync-lulzasaur-tvtropes-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/lulzasaur~tvtropes-scraper/run-sync": { "post": { "operationId": "run-sync-lulzasaur-tvtropes-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "searchQueries": { "title": "Search Queries", "type": "array", "description": "Trope or work names to search for on TVTropes (e.g. 'Batman', 'Chekhov Gun', 'The Matrix')", "default": [ "Batman" ], "items": { "type": "string" } }, "maxListings": { "title": "Max Pages to Scrape", "minimum": 0, "type": "integer", "description": "Maximum number of wiki pages to scrape per search query (0 = unlimited)", "default": 100 }, "scrapeDetails": { "title": "Scrape Full Page Details", "type": "boolean", "description": "Follow search result links and scrape full trope/work pages with descriptions, examples, and related tropes", "default": true }, "proxyConfiguration": { "title": "Proxy Configuration", "type": "object", "description": "Proxy settings — residential proxies recommended to bypass Cloudflare protection", "default": { "useApifyProxy": true } } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```