# ConsumerAffairs Reviews Scraper (`b3264/consumeraffairs-reviews-scraper`) Actor Scrape company reviews and ratings from ConsumerAffairs by company URL. Get review text, star ratings, dates, reviewer info, and company responses. NOTE: Anti-bot protection may block datacenter IPs - residential proxies or VPN recommended. - **URL**: https://apify.com/b3264/consumeraffairs-reviews-scraper.md - **Developed by:** [Brandon Hamm](https://apify.com/b3264) (community) - **Categories:** Marketing - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.50 / 1,000 results This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. - **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use). - **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`. - **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`. - **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`. - **Any other language** — the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## ConsumerAffairs Reviews Scraper [![Apify Actor](https://img.shields.io/badge/Apify-Actor-blue)](https://apify.com) [![Python](https://img.shields.io/badge/Python-3.12-yellow)](https://www.python.org) [![Playwright](https://img.shields.io/badge/Playwright-Powered-green)](https://playwright.dev) Scrape **company reviews and ratings** from [ConsumerAffairs.com](https://www.consumeraffairs.com) by company URL. Extract review text, star ratings, dates, reviewer names, verified buyer badges, company responses, and location data — all without an API key. > **⚠️ Proxy Note**: ConsumerAffairs uses anti-bot protection that may block datacenter IPs. If you get 0 results, try running with residential proxies or locally with a VPN. ### Features - 🔍 **Search by URL** — pass any ConsumerAffairs company page URL - ⭐ **Full review data** — rating, title, body, date, reviewer, location - ✅ **Verified buyer detection** — identifies verified purchaser reviews - 💬 **Company responses** — captures official company replies to reviews - 🏢 **Company metadata** — overall rating, total review count, category - 📄 **Pagination** — automatically follows pagination across all review pages - 🔄 **Multi-layout parsing** — handles HTML, JSON-LD schema, and API-intercepted data - 🛡️ **Anti-bot bypass** — uses Apify residential proxy + Cloudflare challenge detection - 🚫 **No API key required** — fully client-side scraping ### Input | Field | Type | Required | Default | Description | |-------|------|----------|---------|-------------| | `companyUrls` | array\ | ✅ | — | ConsumerAffairs company URLs (e.g., `https://www.consumeraffairs.com/travel/united.html`) | | `maxReviews` | integer | ❌ | `200` | Maximum reviews to scrape per company | | `includeCompanyMetadata` | boolean | ❌ | `true` | Extract overall rating, total reviews, category | | `useProxy` | boolean | ❌ | `true` | Use Apify residential proxy for anti-bot bypass | #### Example Input ```json { "companyUrls": [ "https://www.consumeraffairs.com/travel/united.html", "https://www.consumeraffairs.com/finance/wells-fargo.html" ], "maxReviews": 500, "includeCompanyMetadata": true, "useProxy": true } ``` ### Output Each review is pushed to the Apify dataset as a JSON object: | Field | Type | Description | |-------|------|-------------| | `reviewId` | string | Unique review identifier (from CA or content hash) | | `sourceUrl` | string | URL the review was scraped from | | `rating` | integer/float | Star rating (1–5) | | `reviewer` | string | Reviewer display name | | `date` | string | Review date (as shown on page) | | `reviewTitle` | string | Review headline/summary | | `reviewText` | string | Full review body (max 5,000 chars) | | `verifiedBuyer` | boolean | Whether the reviewer is a verified buyer | | `companyResponse` | string | Official company response if present | | `location` | string | Reviewer location if available | | `companyName` | string | Company name (if metadata enabled) | | `overallRating` | float | Company's overall rating (if metadata enabled) | | `totalReviews` | integer | Company's total review count (if metadata enabled) | | `category` | string | Company category (if metadata enabled) | #### Example Output ```json { "reviewId": "abc123def456", "sourceUrl": "https://www.consumeraffairs.com/travel/united.html?page=1", "rating": 2, "reviewer": "John D.", "date": "August 1, 2024", "reviewTitle": "Cancelled flight, no refund", "reviewText": "My flight was cancelled and I still haven't received a refund after 3 months...", "verifiedBuyer": true, "companyResponse": "We apologize for the inconvenience. Our team is looking into this...", "location": "Chicago, IL", "companyName": "United Airlines", "overallRating": 1.4, "totalReviews": 12847, "category": "travel" } ``` ### Use Cases #### 🏢 Brand Reputation Monitoring Track what consumers are saying about your company across ConsumerAffairs. Monitor review volume, average ratings, and sentiment trends over time. #### 📊 Competitor Analysis Scrape competitor review pages to benchmark customer satisfaction, identify common complaints, and find gaps in the market. #### 📈 Market Research Aggregate consumer feedback across an entire industry category (insurance, finance, telecom, travel). Identify recurring issues and pain points. #### 💬 Customer Experience Analytics Extract review text for NLP sentiment analysis, topic modeling, or feed into LLM pipelines for automated insight generation. #### ⚖️ Compliance & Dispute Tracking Monitor verified buyer complaints and company responses for regulatory compliance or legal evidence gathering. ### How It Works 1. **Playwright renders** each company page with full JavaScript execution 2. **Lazy-loading triggered** via scroll simulation to load all reviews 3. **API interception** captures any XHR/fetch JSON responses containing review data 4. **Multi-strategy parsing** extracts reviews from HTML containers, JSON-LD schema, or intercepted API payloads 5. **Cloudflare detection** identifies challenge pages and retries automatically 6. **Pagination** follows "Next" links, numeric page links, or constructs `?page=N` URLs 7. **Deduplication** ensures the same review never appears twice across pages or sources ### Technical Details - **Runtime:** Python 3.12 + Playwright (Chromium) - **Proxy:** Apify residential proxy (configurable) - **Parsing:** BeautifulSoup + lxml for HTML; native JSON for API/JSON-LD - **Anti-bot:** Cloudflare challenge detection with retry logic - **Dedup:** SHA-256 content hashing for reviews without explicit IDs ### Related Actors - [Google Maps Reviews Scraper](https://apify.com/store/google-maps-reviews) — scrape Google Maps reviews - [Trustpilot Reviews Scraper](https://apify.com/store/trustpilot-reviews) — extract Trustpilot company reviews - [Yelp Reviews Scraper](https://apify.com/store/yelp-reviews) — scrape Yelp business reviews - [Amazon Reviews Scraper](https://apify.com/store/amazon-reviews) — get Amazon product reviews - [Glassdoor Reviews Scraper](https://apify.com/store/glassdoor-reviews) — scrape employer reviews ### Limitations - ConsumerAffairs may rate-limit or block IPs; using Apify proxy is strongly recommended - Some pages may trigger Cloudflare challenges — the actor retries but cannot guarantee bypass - Review text is capped at 5,000 characters per review - Maximum 10,000 reviews per company (configurable via input) ### Support If you encounter issues or have feature requests, please open an issue on the actor's Apify store page. *** **No API key required. No login required. Just provide company URLs and start scraping.** # Actor input Schema ## `companyUrls` (type: `array`): List of ConsumerAffairs company page URLs (e.g., 'https://www.consumeraffairs.com/travel/united.html') ## `maxReviews` (type: `integer`): Maximum number of reviews to scrape per company ## `includeCompanyMetadata` (type: `boolean`): Also extract overall rating, total reviews count, and category ## `useProxy` (type: `boolean`): Use Apify residential proxy to bypass anti-bot protection (recommended) ## Actor input object example ```json { "maxReviews": 200, "includeCompanyMetadata": true, "useProxy": true } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = {}; // Run the Actor and wait for it to finish const run = await client.actor("b3264/consumeraffairs-reviews-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = {} # Run the Actor and wait for it to finish run = client.actor("b3264/consumeraffairs-reviews-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{}' | apify call b3264/consumeraffairs-reviews-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,b3264/consumeraffairs-reviews-scraper" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/TcJyvapjKpm24osjT/builds/8fjNGdPb3jnMkpLqH/openapi.json