# Fandom Scraper (`khadinakbar/fandom-scraper`) Actor Scrape any Fandom / Wikia wiki: article text, infobox data, categories, images, and search results via the public MediaWiki API. - **URL**: https://apify.com/khadinakbar/fandom-scraper.md - **Developed by:** [Khadin Akbar](https://apify.com/khadinakbar) (community) - **Categories:** AI, Automation, MCP servers - **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $5.00 / 1,000 article scrapeds This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Fandom Scraper Scrape any **Fandom / Wikia wiki** — Harry Potter, Marvel, Minecraft, One Piece, Genshin Impact, Star Wars, Pokémon, Naruto, and the hundreds of thousands of other communities on `fandom.com` — into clean, structured JSON. Built directly on the public **MediaWiki Action API**, so it is HTTP-only, login-free, fast, and reliable. No cookies, no browser, no breakage. The headline feature: the article **infobox** (the structured side-panel of stats — a character's affiliations, an item's manufacturer, a location's region) is parsed into a flat `{ label: value }` object. That is the part fans, datasets, and AI agents actually want, and most generic scrapers throw it away. ### What you get One JSON record per article: | Field | Description | |-------|-------------| | `title`, `url`, `pageId` | Article identity | | `wiki`, `subdomain`, `language` | Which wiki, and its language code | | `summary` | Lead paragraphs as clean plain text | | `infobox` | Portable-infobox data as a flat `{ label: value }` object | | `infoboxImage` | Main infobox image URL | | `categories` | Category names the article belongs to | | `sections` | Heading outline (`level`, `title`, `anchor`) | | `images` | Image filenames used on the page | | `internalLinks`, `externalLinks` | Outgoing links (capped, configurable) | | `wordCount`, `lastEdited`, `lastRevisionId` | Size + freshness (ISO-8601 edit time) | | `contentText`, `contentHtml`, `wikitext` | Full article body — opt-in toggles | By default records are **lean** (summary + infobox + metadata) so they fit comfortably in an AI agent's context window. Flip `includeFullText`, `includeHtml`, or `includeWikitext` when you need the whole body. ### Four modes — auto-detected from your input 1. **Article URLs** — paste exact pages into `startUrls`, e.g. `https://harrypotter.fandom.com/wiki/Wand`. You can mix wikis and languages in a single run. 2. **Search** — set `wiki` (`harrypotter`) + `searchQuery` (`elder wand`); the top matching articles are scraped, up to `maxResults`. 3. **Category** — set `wiki` + `category` (`Wands`); every article in the category is scraped, with automatic pagination. 4. **Titles** — set `wiki` + `titles` (`["Wand", "Elder Wand"]`) when you already know the exact pages. `wiki` accepts either a subdomain (`harrypotter`) or a full host (`harrypotter.fandom.com`). Language wikis work too, via either the subdomain form (`es.naruto`) or a language-path URL (`harrypotter.fandom.com/de/wiki/Zauberstab`). #### Example input ```json { "wiki": "harrypotter", "category": "Wands", "maxResults": 50 } ```` #### Example output (trimmed) ```json { "wiki": "harrypotter.fandom.com", "subdomain": "harrypotter", "language": "en", "title": "Wand", "url": "https://harrypotter.fandom.com/wiki/Wand", "summary": "A wand was a quasi-sentient magical instrument through which a witch or wizard channelled their magical powers…", "infobox": { "Manufacturer": "Ollivanders, Gregorovitch", "Usage": "Casting spells" }, "infoboxImage": "https://static.wikia.nocookie.net/harrypotter/images/5/5c/Wands-Woods.png", "categories": ["Wands"], "sections": [{ "level": 2, "title": "History", "anchor": "History" }], "wordCount": 10124, "lastEdited": "2026-05-24T19:02:03Z" } ``` ### Who uses this - **Dataset & ML teams** building knowledge bases or fine-tuning corpora on fictional universes, games, and media. - **Content & SEO sites** that aggregate game guides, character bios, item stats, and lore. - **Researchers & analysts** studying fan communities, franchises, and worldbuilding. - **AI agents** that need a single, canonical fact source for a game or show, returned as structured JSON. ### Pricing Pay-per-event — you pay only for articles actually delivered: - **Actor start** — $0.00005 per run - **Article scraped** — **$0.005** per article successfully returned A 50-article category run costs about **$0.25**. Failed or missing pages are never charged. ### For AI agents (MCP-ready) Exposed through the Apify MCP server as `apify--fandom-scraper`. The input is narrow (a wiki plus a query or a URL), the output is structured JSON, and the per-article cost is predictable — exactly the shape an agent wants. Use it to pull canonical facts about characters, items, locations, and events across any fan wiki. This is **not** a general web crawler. It speaks only to Fandom / Wikia (MediaWiki) wikis. For other sites, use the matching scraper in the store. ### FAQ **Does it need login or cookies?** No. It reads only public wiki content through the official MediaWiki API. **Which wikis work?** Any wiki on `fandom.com` (formerly Wikia), in any language. **How current is the data?** Live — each run hits the wiki in real time. `lastEdited` tells you when the article itself last changed. **What if a page doesn't exist?** It is skipped (and not charged). If a run's pages are *all* invalid, the run fails cleanly with a clear message. **Can I get the full article text?** Yes — set `includeFullText` for clean plain text, `includeHtml` for rendered HTML, or `includeWikitext` for the raw MediaWiki source. **How do I keep records small for an LLM?** Leave the content toggles off (the default) and lower `maxLinks`. ### Notes & legal - Reads only public wiki pages via the official MediaWiki Action API; redirects are followed automatically. - Fandom content is community-authored and licensed under **CC-BY-SA** — attribute the source wiki when you republish. - Use responsibly and within [Fandom's Terms of Use](https://www.fandom.com/terms-of-use). This tool retrieves publicly available pages only and performs no login or circumvention. # Actor input Schema ## `startUrls` (type: `array`): Exact Fandom/Wikia article URLs to scrape, e.g. 'https://harrypotter.fandom.com/wiki/Wand'. Works across any language path. When set, this takes priority over the wiki + search/category/titles modes. Leave empty to use a search, category, or title-list query instead. ## `wiki` (type: `string`): The Fandom wiki to query in search/category/titles mode. Accepts a subdomain like 'harrypotter' or a full host like 'harrypotter.fandom.com'. Ignored when 'Fandom article URLs' is provided. NOT a full article URL — use the URLs field for those. ## `searchQuery` (type: `string`): Free-text full-text search run on the chosen wiki (e.g. 'elder wand'). The top matching articles are scraped, up to 'Max results'. Requires the 'Wiki' field. Leave empty if using URLs, titles, or category mode. ## `category` (type: `string`): Scrape every article in this wiki category, e.g. 'Wands' or 'Category:Wands' (the 'Category:' prefix is optional). Members are paginated up to 'Max results'. Requires the 'Wiki' field. NOT a search query. ## `titles` (type: `array`): Explicit list of article titles on the chosen wiki to scrape, e.g. \['Wand', 'Elder Wand']. Requires the 'Wiki' field. Use this when you already know the exact pages; use 'Search query' for discovery. ## `maxResults` (type: `integer`): Maximum number of articles to scrape in search or category mode. Bounds cost and runtime. Defaults to 50. Ignored in URLs and titles mode, which scrape exactly what you list. ## `includeFullText` (type: `boolean`): When true, each record includes the full article body as clean plain text ('contentText'), not just the summary. Increases record size. Defaults to false for lean, agent-friendly output. ## `includeHtml` (type: `boolean`): When true, each record includes the article's rendered HTML body ('contentHtml'). Large. Defaults to false. ## `includeWikitext` (type: `boolean`): When true, each record includes the raw MediaWiki wikitext source ('wikitext'). Useful for template/markup analysis. Defaults to false. ## `maxLinks` (type: `integer`): Cap on how many internal and external links are included per record, keeping output lean. Defaults to 100. Set to 0 to omit links entirely. ## `proxyConfiguration` (type: `object`): Proxy settings. Defaults to Apify Proxy (datacenter), which is sufficient for the MediaWiki API. Switch to residential only if you hit rate limits on very large jobs. ## Actor input object example ```json { "startUrls": [ "https://harrypotter.fandom.com/wiki/Wand" ], "wiki": "harrypotter", "searchQuery": "elder wand", "category": "Wands", "maxResults": 50, "includeFullText": false, "includeHtml": false, "includeWikitext": false, "maxLinks": 100, "proxyConfiguration": { "useApifyProxy": true } } ``` # Actor output Schema ## `articles` (type: `string`): One record per Fandom article scraped. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ "https://harrypotter.fandom.com/wiki/Wand" ], "maxResults": 50 }; // Run the Actor and wait for it to finish const run = await client.actor("khadinakbar/fandom-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": ["https://harrypotter.fandom.com/wiki/Wand"], "maxResults": 50, } # Run the Actor and wait for it to finish run = client.actor("khadinakbar/fandom-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ "https://harrypotter.fandom.com/wiki/Wand" ], "maxResults": 50 }' | apify call khadinakbar/fandom-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=khadinakbar/fandom-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Fandom Scraper", "description": "Scrape any Fandom / Wikia wiki: article text, infobox data, categories, images, and search results via the public MediaWiki API.", "version": "1.0", "x-build-id": "ZWwpbHWTFKKH7UlSn" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/khadinakbar~fandom-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-khadinakbar-fandom-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/khadinakbar~fandom-scraper/runs": { "post": { "operationId": "runs-sync-khadinakbar-fandom-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/khadinakbar~fandom-scraper/run-sync": { "post": { "operationId": "run-sync-khadinakbar-fandom-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "startUrls": { "title": "Fandom article URLs", "type": "array", "description": "Exact Fandom/Wikia article URLs to scrape, e.g. 'https://harrypotter.fandom.com/wiki/Wand'. Works across any language path. When set, this takes priority over the wiki + search/category/titles modes. Leave empty to use a search, category, or title-list query instead.", "items": { "type": "string" } }, "wiki": { "title": "Wiki (subdomain or host)", "type": "string", "description": "The Fandom wiki to query in search/category/titles mode. Accepts a subdomain like 'harrypotter' or a full host like 'harrypotter.fandom.com'. Ignored when 'Fandom article URLs' is provided. NOT a full article URL — use the URLs field for those." }, "searchQuery": { "title": "Search query", "type": "string", "description": "Free-text full-text search run on the chosen wiki (e.g. 'elder wand'). The top matching articles are scraped, up to 'Max results'. Requires the 'Wiki' field. Leave empty if using URLs, titles, or category mode." }, "category": { "title": "Category", "type": "string", "description": "Scrape every article in this wiki category, e.g. 'Wands' or 'Category:Wands' (the 'Category:' prefix is optional). Members are paginated up to 'Max results'. Requires the 'Wiki' field. NOT a search query." }, "titles": { "title": "Page titles", "type": "array", "description": "Explicit list of article titles on the chosen wiki to scrape, e.g. ['Wand', 'Elder Wand']. Requires the 'Wiki' field. Use this when you already know the exact pages; use 'Search query' for discovery.", "items": { "type": "string" } }, "maxResults": { "title": "Max results", "minimum": 1, "maximum": 5000, "type": "integer", "description": "Maximum number of articles to scrape in search or category mode. Bounds cost and runtime. Defaults to 50. Ignored in URLs and titles mode, which scrape exactly what you list.", "default": 50 }, "includeFullText": { "title": "Include full plain text", "type": "boolean", "description": "When true, each record includes the full article body as clean plain text ('contentText'), not just the summary. Increases record size. Defaults to false for lean, agent-friendly output.", "default": false }, "includeHtml": { "title": "Include rendered HTML", "type": "boolean", "description": "When true, each record includes the article's rendered HTML body ('contentHtml'). Large. Defaults to false.", "default": false }, "includeWikitext": { "title": "Include raw wikitext", "type": "boolean", "description": "When true, each record includes the raw MediaWiki wikitext source ('wikitext'). Useful for template/markup analysis. Defaults to false.", "default": false }, "maxLinks": { "title": "Max links per article", "minimum": 0, "maximum": 5000, "type": "integer", "description": "Cap on how many internal and external links are included per record, keeping output lean. Defaults to 100. Set to 0 to omit links entirely.", "default": 100 }, "proxyConfiguration": { "title": "Proxy configuration", "type": "object", "description": "Proxy settings. Defaults to Apify Proxy (datacenter), which is sufficient for the MediaWiki API. Switch to residential only if you hit rate limits on very large jobs.", "default": { "useApifyProxy": true } } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```