# Python Scraper (`sovanza.inc/python-scraper`) Actor Python Scraper extracts web page data using Requests and BeautifulSoup. It collects titles, meta tags, headings, links, images, Open Graph data, text snippets, and custom CSS selector fields, with exports to JSON, CSV, Excel, XML, or HTML. - **URL**: https://apify.com/sovanza.inc/python-scraper.md - **Developed by:** [Sovanza](https://apify.com/sovanza.inc) (community) - **Categories:** Developer tools, SEO tools - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, NaN bookmarks - **User rating**: 5.00 out of 5 stars ## Pricing from $2.00 / 1,000 scraped pages This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ### Python Scraper – Extract Web Page Data with Requests & BeautifulSoup Scrape any public web page at scale using **Python** (Requests + BeautifulSoup). Extract titles, meta tags, headings, links, Open Graph data, and custom CSS fields — then export results to **JSON, CSV, Excel, XML, or HTML** via Apify dataset and key-value store. ### What is Python Scraper and How Does It Work? Python Scraper is a flexible **general-purpose web scraping** actor on Apify. It fetches each URL you provide, parses HTML with BeautifulSoup, and returns structured data per page. It is designed for: - Developers building custom scrapers from a Python template - Data analysts collecting page metadata at scale - SEO and content teams auditing titles, descriptions, and headings - Automation engineers feeding pipelines with JSON/CSV exports Each run processes your **start URLs** sequentially, pushes one dataset item per URL, and writes a **combined export file** to the default key-value store in the format you choose. ### Why Use This Python Scraper? Use this actor to: - Scrape multiple URLs in one Apify run without writing boilerplate - Extract standard page fields (title, meta, H1, links, images) out of the box - Add **custom CSS selectors** for any extra text fields you need - Export all results as **JSON, CSV, Excel, XML, or HTML** for downstream tools - Integrate with Apify API, schedules, and webhooks for recurring jobs ➡️ Lightweight and fast — no browser required; ideal for static HTML and simple sites. ### What Data Does Python Scraper Extract? This actor outputs **one dataset item per URL**, including (when available on the page): **Core page data** - `url` — input URL - `finalUrl` — URL after redirects - `status` — HTTP status code - `contentType`, `charset`, `contentLength` - `title` — document title - `metaDescription`, `metaKeywords`, `robots` - `canonicalUrl`, `language` - `h1` — primary heading - `headings` — optional `h1` / `h2` / `h3` lists - `text` — visible text snippet (up to 4,000 characters) **Social & media** - `openGraph` — Open Graph meta properties - `twitterCard` — Twitter card meta properties - `images` — image URLs and alt text (up to 25) **Links (optional)** - `links` — extracted hrefs (when `includeLinks` is true) - `linkCount` — number of links - `firstLinkText`, `firstLinkUrl` — first anchor on the page **Custom fields** - Any keys you define in **`selectors`** (CSS selector → field name) **Errors** - `error` — message when a URL fails to fetch or parse **Export (key-value store)** - Combined file: `{exportKey}.json` | `.csv` | `.xlsx` | `.xml` | `.html` (based on `exportFormat`) ➡️ Dataset rows are structured and exportable in JSON, CSV, or Excel via Apify. Optional raw `html` in export when `includeHtml` is enabled. ### Features - **Multi-URL scraping** — batch many `startUrls` in a single run - **Custom CSS selectors** — map field names to selectors (supports `::text` suffix) - **Rich metadata** — title, meta, canonical, Open Graph, Twitter card - **Link extraction** — optional full link list per page - **Multiple export formats** — JSON, CSV, Excel, XML, HTML to KV store - **Configurable HTTP** — method, headers, timeout - **Clean output** — empty fields omitted from dataset items - **Automation-ready** — Apify API, schedules, webhooks ### How to Use Python Scraper on Apify #### Using the Actor 1. **Go to Python Scraper** on the Apify platform. 2. **Input Configuration**: - Add one or more **start URLs** (public pages you are allowed to scrape). - Optionally set **CSS selectors** for extra fields. - Choose **export format** and **export key** for the combined KV file. 3. **Run the Actor** — Each URL produces one dataset row; export file is written to the key-value store. 4. **Access Your Results** — **Dataset** tab for per-URL items; **Key-value store** for the combined export; use API links from the **Output** schema. 5. **Schedule** (optional) — Recurring runs for monitoring or refresh workflows. #### Input Configuration The actor accepts the following parameters: ```json { "startUrls": [ "https://example.com", "https://news.ycombinator.com/" ], "method": "GET", "headers": { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Python Scraper (Apify)" }, "timeoutSecs": 60, "selectors": { "headline": "h1::text", "tagline": "p.lead::text" }, "includeLinks": true, "includeHtml": false, "exportFormat": "json", "exportKey": "EXPORT" } ```` | Field | Description | |--------|-------------| | `startUrls` | **Required.** URLs to scrape (one per line in Console or JSON array). | | `method` | HTTP method: `GET` or `POST` (default `GET`). | | `headers` | Optional request headers (JSON object). | | `timeoutSecs` | Per-request timeout in seconds (default `60`, max `300`). | | `selectors` | Optional `{ "fieldName": "css selector" }` — first matching text per field. | | `includeLinks` | If true, extract and include page links (default `true`). | | `includeHtml` | If true, include raw HTML in export (can be large; default `false`). | | `exportFormat` | `json`, `csv`, `excel`, `xml`, or `html` — combined KV export format. | | `exportKey` | KV store key prefix (default `EXPORT`); file extension added automatically. | ### Output **Dataset** — one item per URL (overview table in Console): | Field | Description | |--------|-------------| | `url` | Input URL | | `finalUrl` | Final URL after redirects | | `status` | HTTP status code | | `title` | Page title | | `metaDescription` | Meta description | | `h1` | Primary H1 text | | `firstLinkText` / `firstLinkUrl` | First link on page | | `linkCount` | Number of links (if `includeLinks`) | | `contentType` | Response content type | | `openGraph` / `twitterCard` | Social meta objects | | `images` | Image list with `src` and optional `alt` | | `headings` | Nested h1/h2/h3 when multiple headings exist | | `error` | Error message on failed URLs | | *custom* | Fields from your `selectors` map | **Key-value store** — combined export at `{exportKey}.{ext}` (e.g. `EXPORT.json`). Example dataset item (illustrative): ```json { "url": "https://example.com", "finalUrl": "https://example.com/", "status": 200, "title": "Example Domain", "metaDescription": "Example description", "h1": "Example Domain", "text": "This domain is for use in documentation examples...", "openGraph": { "title": "Example", "type": "website" }, "linkCount": 1, "contentType": "text/html" } ``` Use the actor **Output** schema in Apify Console for direct API links to dataset items and export files. ### How the Scraper Works 1. **Requests** fetches each `startUrl` with your headers and timeout. 2. **BeautifulSoup (lxml)** parses HTML and extracts structured fields. 3. **Custom selectors** run per field name you define in input. 4. Each result is **pushed to the default dataset** (empty values omitted). 5. All results are **serialized** to the key-value store in the chosen `exportFormat`. ### Reliability & Best Practices - Respect **robots.txt**, site terms, and rate limits for target sites. - Use realistic **User-Agent** and headers for sites that block bots. - Increase `timeoutSecs` for slow pages. - Disable `includeHtml` unless you need raw HTML (large files). - For JavaScript-heavy SPAs, consider a Playwright-based actor instead. ### Performance - No browser overhead — fast for static HTML. - Batch many URLs in one run; sequential fetching keeps memory predictable. - Large `includeHtml` exports increase KV store size. ### Use Cases - SEO audits (titles, meta, H1, canonical) - Content monitoring and change detection (scheduled runs) - Lead / directory page extraction with custom selectors - Research datasets and internal analytics pipelines - Template for building custom Python scrapers on Apify ### Integrations & API - Full API via Apify (`apify-client` Python / Node.js) - Zapier, Make, Google Sheets via dataset export - Webhooks and scheduled runs - Output schema links for dataset and KV export URLs ### FAQ #### Can I scrape any website with this actor? Only pages you are **legally permitted** to access and process. You must comply with each site’s terms, robots.txt, and applicable privacy laws. This tool does not bypass paywalls or authentication by default. #### Does this actor run JavaScript? No. It uses HTTP + HTML parsing. Dynamic sites that render content only in the browser may need a Playwright/Puppeteer actor. #### How do custom CSS selectors work? Add a JSON object `selectors` where keys are output field names and values are CSS selectors. Use `::text` suffix if needed; the actor normalizes selectors and extracts the first match’s text. #### Where is the full export file? In the run’s **default key-value store**, named `{exportKey}.{extension}` (e.g. `EXPORT.csv`). Dataset items are always written per URL regardless of export format. #### What happens if one URL fails? That URL gets a dataset row with `error`; other URLs still process. The combined export includes both successful and failed rows. #### Can I automate runs on a schedule? Yes. Use Apify schedules and pass updated `startUrls` via API or integrations. ### SEO Keywords (high-intent) python web scraper\ python scraper apify\ beautifulsoup scraper\ requests web scraping\ html scraper api\ extract page metadata\ seo scraper tool\ web scraping template python\ scrape urls to json csv\ apify python actor ### Why Choose This Actor? - Popular Python stack (Requests + BeautifulSoup) - Per-URL dataset + combined multi-format export - Custom CSS selectors without code changes - Clear overview table via dataset schema - Ideal starter template for Apify Python projects ### Limitations - **No JavaScript rendering** — static HTML only. - **No built-in proxy** — add headers or use Apify Proxy at platform level if needed for your deployment. - **Sequential requests** — no built-in concurrency per URL. - Some sites block datacenter IPs or require cookies — not included by default. ### Running locally 1. `pip install -r requirements.txt` 2. Create `INPUT.json` in the actor folder, e.g.: ```json { "startUrls": ["https://example.com"], "exportFormat": "json", "exportKey": "EXPORT" } ``` 3. Run with Apify CLI: `apify run` or execute `main.py` in an Apify-compatible environment. ### Deploy to Apify Use `INPUT_SCHEMA.json`, `OUTPUT_SCHEMA.json`, `.actor/dataset_schema.json`, and the provided `Dockerfile`. Push with Apify CLI or connect the Git repository. ### Get Started Add your URLs, define optional selectors, pick an export format, and start extracting structured web data with Python on Apify. 🚀 # Actor input Schema ## `startUrls` (type: `array`): One or more URLs to scrape (one per line). ## `method` (type: `string`): HTTP method used for requests. ## `headers` (type: `object`): Optional headers for each request (e.g. Authorization, Cookie, API keys). Values are stored as a secret input. Leave empty to use the default User-Agent. ## `timeoutSecs` (type: `integer`): Per-request timeout in seconds. ## `selectors` (type: `object`): Optional mapping of fieldName -> CSS selector. The actor will extract the first matching text for each selector. ## `includeLinks` (type: `boolean`): Extract and include page links (hrefs). ## `includeHtml` (type: `boolean`): If enabled, the export file will contain a 'html' field with the raw HTML (can be large). ## `exportFormat` (type: `string`): Export all extracted results to the key-value store. ## `exportKey` (type: `string`): Key where the export will be saved in the default key-value store. ## Actor input object example ```json { "startUrls": [ "https://example.com" ], "method": "GET", "timeoutSecs": 60, "selectors": {}, "includeLinks": true, "includeHtml": false, "exportFormat": "json", "exportKey": "EXPORT" } ``` # Actor output Schema ## `pages` (type: `string`): One dataset row per URL with title, meta tags, headings, links, and custom CSS selector fields. ## `exportJson` (type: `string`): Combined results when exportFormat is json (default). Filename: {exportKey}.json ## `exportCsv` (type: `string`): Combined results when exportFormat is csv. Filename: {exportKey}.csv ## `exportExcel` (type: `string`): Combined results when exportFormat is excel. Filename: {exportKey}.xlsx ## `exportXml` (type: `string`): Combined results when exportFormat is xml. Filename: {exportKey}.xml ## `exportHtml` (type: `string`): Combined results when exportFormat is html. Filename: {exportKey}.html # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ "https://example.com" ] }; // Run the Actor and wait for it to finish const run = await client.actor("sovanza.inc/python-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": ["https://example.com"] } # Run the Actor and wait for it to finish run = client.actor("sovanza.inc/python-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ "https://example.com" ] }' | apify call sovanza.inc/python-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=sovanza.inc/python-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Python Scraper", "description": "Python Scraper extracts web page data using Requests and BeautifulSoup. It collects titles, meta tags, headings, links, images, Open Graph data, text snippets, and custom CSS selector fields, with exports to JSON, CSV, Excel, XML, or HTML.", "version": "0.0", "x-build-id": "RtbDiggRBbF7fgP8A" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/sovanza.inc~python-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-sovanza.inc-python-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/sovanza.inc~python-scraper/runs": { "post": { "operationId": "runs-sync-sovanza.inc-python-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/sovanza.inc~python-scraper/run-sync": { "post": { "operationId": "run-sync-sovanza.inc-python-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "startUrls" ], "properties": { "startUrls": { "title": "Start URLs", "type": "array", "description": "One or more URLs to scrape (one per line).", "items": { "type": "string" } }, "method": { "title": "HTTP method", "enum": [ "GET", "POST" ], "type": "string", "description": "HTTP method used for requests.", "default": "GET" }, "headers": { "title": "Request headers", "type": "object", "description": "Optional headers for each request (e.g. Authorization, Cookie, API keys). Values are stored as a secret input. Leave empty to use the default User-Agent." }, "timeoutSecs": { "title": "Timeout (seconds)", "minimum": 1, "maximum": 300, "type": "integer", "description": "Per-request timeout in seconds.", "default": 60 }, "selectors": { "title": "CSS selectors (optional)", "type": "object", "description": "Optional mapping of fieldName -> CSS selector. The actor will extract the first matching text for each selector.", "default": {} }, "includeLinks": { "title": "Extract links", "type": "boolean", "description": "Extract and include page links (hrefs).", "default": true }, "includeHtml": { "title": "Include raw HTML in export", "type": "boolean", "description": "If enabled, the export file will contain a 'html' field with the raw HTML (can be large).", "default": false }, "exportFormat": { "title": "Export format", "enum": [ "json", "csv", "excel", "xml", "html" ], "type": "string", "description": "Export all extracted results to the key-value store.", "default": "json" }, "exportKey": { "title": "Export key (KV store)", "type": "string", "description": "Key where the export will be saved in the default key-value store.", "default": "EXPORT" } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```