# Changelog of German Imprint Scraper (`dominic-quaiser/imprint-contact-scraper`) Actor

- **URL**: https://apify.com/dominic-quaiser/imprint-contact-scraper/changelog.md
- **Full Actor documentation**: https://apify.com/dominic-quaiser/imprint-contact-scraper.md

## Changelog

### \[v0.12.3-beta] — 2026-05-06

#### Fixed

- **Playwright browser health monitoring**: Browser is now forcibly relaunched when either 5 consecutive goto failures occur or 100 pages have been served since last launch.
- **Proxy error fast-fail**: HTTP fetches that fail with a proxy tunnel error (`ProxyError`) now skip the Playwright fallback entirely, since the proxy is bound at browser launch and would fail identically.
- **Pseudo-URL link discovery**: Imprint link detection no longer returns `javascript:`, `mailto:`, `tel:`, and similar non-fetchable hrefs.
- **Persistent error early exit**: Fetch failures flagged as non-transient (proxy tunnel, SSL/cert errors, DNS failure, oversized payload) no longer burn remaining retry budget; the URL is abandoned immediately.
- **NER retry storm**: Reduced per-call retries from 5 to 3 and circuit-breaker threshold from 4 to 3 consecutive failures, so a degraded NER API opens the breaker faster and stops stacking 60 s waits.
- **15-minute process hang on exit**: `asyncio.run()` blocks on `shutdown_default_executor()`, waiting forever for regex threads that can't be cancelled. Replaced with a manual event loop and `os._exit()` in the finally block so the process exits immediately after all durable cleanup completes.

#### Changed

- **HTML payload cap**: Both HTTP and Playwright fetchers now refuse HTML bodies larger than 2 MB (`MAX_HTML_BYTES`), preventing catastrophic regex backtracking and the OOM failures it caused.
- **Playwright goto timeout**: Reduced from 120 s to 60 s (`PLAYWRIGHT_GOTO_TIMEOUT_MS`) to reclaim time budget on hung navigations.
- **Concurrency**: Reduced from 4 to 2 concurrent URLs (`MAX_CONCURRENT_URLS`) to lower peak memory pressure.

### \[v0.12.2-beta] — 2026-04-23

#### Fixed

- **Playwright cascade failure**: A single site whose `page.close()` was slow could trigger `_force_cleanup()`, closing the shared Browser and killing all other in-flight concurrent fetches. Those fetches then hit the same path, causing repeated browser relaunches and cascading `Slot timeout (360s)` errors. Each fetch now runs in its own `BrowserContext`; a slow teardown orphans only that page/context and never touches the shared Browser. The Browser is replaced only when `Browser.is_connected()` returns `False`, serialized under the existing lock.
- **`_force_cleanup()` race condition**: `_force_cleanup()` was called without holding `_browser_lock`, allowing concurrent double-closes and races with a relaunch in `_get_browser()`. All callers now hold the lock.
- **Slow `page.close()` on `beforeunload`-heavy sites**: `page.close()` is now called with `run_before_unload=False` to skip site-level teardown handlers — the most common cause of the previous 5-second hangs. Cleanup timeout raised from 5 s to 15 s as an additional buffer.

### \[v0.12.1-beta] — 2026-04-02

#### Changed

- **Constants changes**: Increased concurrency.
- **Fallback to Playwright**: Added retry if no imprint page was found doing the HTTP request.

### \[v0.12.0-beta] — 2026-03-07

#### Changed

- **Proxy configuration for headless browser**: Full proxy integration completed. When a proxy is configured via `proxyConfiguration` input or `SCRAPER_PROXY_URL` environment variable, the Playwright headless browser now correctly authenticates with the proxy using credentials extracted from the proxy URL. Supports proxies with or without embedded credentials (e.g., `http://user:pass@proxy.example.com:8000` or `http://proxy.example.com:8000`).

#### Fixed

- **Playwright proxy authentication**: Playwright requires credentials to be passed separately as `username` and `password` fields, not embedded in the `server` URL. This fixes timeout issues when using Apify proxies with headless browser requests.

### \[v0.11.0-beta] — 2026-02-27

#### Added

- **Fallback NER API support**: Optional secondary NER API for decision maker extraction resilience. If configured via `NER_FALLBACK_API_URL` and `NER_FALLBACK_API_KEY` environment variables, the scraper will automatically retry against the fallback API if the primary API fails. Each API endpoint gets up to 5 retry attempts with exponential backoff. Health checks validate availability at startup.
- **Legal form identification in output**: The `company_name` output now includes an optional `legal_form` object with `short` (e.g., "GmbH") and `full` (e.g., "Gesellschaft mit beschränkter Haftung") keys when a legal form is detected. Supports all German legal forms from the official XUnternehmen Rechtsformen codelist, compound forms (e.g., "GmbH & Co. KG"), and common foreign forms (Ltd., B.V., S.A., etc.).

### \[v0.10.0-beta] — 2026-02-22

#### Added

- **Dataset schema**: Defined `dataset_schema.json` documenting all 13 output fields with titles, descriptions, and examples. Includes support for optional error records and metadata.
- **Output schema**: Defined `output_schema.json` documenting actor output with links to JSON/CSV/Excel exports and 5 specialized dataset views:
  - **Overview**: All extracted fields for complete picture
  - **Contact Information**: Company name, emails, phone/fax numbers — ready for outreach
  - **Company Details**: Legal information (name, address, register number, VAT ID)
  - **Decision Makers**: Extracted responsible persons per company
  - **Social Media**: Social media profile links found on imprint pages
- **Cost manager**: Centralized `CostManager` class that wraps `Actor.charge()`, loads tier-specific pricing from `pay_per_event.json`, and independently tracks cumulative costs for proactive budget enforcement.

#### Changed

- **Charging interface**: Replaced dual `Actor.charge()` + tracking calls with single `CostManager.charge_event()` async method at all 5 charge points (website-processed, headless-browser, decision-maker-extracted, dataset-item, actor-start).
- **Cost limit handling**: When `ACTOR_MAX_TOTAL_CHARGE_USD` is exceeded, skipped URLs remain pending (not marked as failed) so they can be retried on resurrection without re-processing failed URLs.
- **Removed**: Deleted unused `pricing_manager.py` — functionality fully subsumed by `CostManager`.

### \[v0.9.3-beta] — 2026-02-19

#### Added

- **Proxy configuration module**: Standalone proxy setup handler that creates and manages Apify proxy configurations based on actor input.
- **Proxy support in actor configuration**: New `proxy` property in `ActorConfig` to expose raw proxy input from the actor's configuration.

#### Changed

- **HTTP and Playwright fetchers**: Both now accept `proxy_config` parameter to use Apify-managed proxies or fall back to `SCRAPER_PROXY_URL` environment variable.
- **Page fetcher initialization**: Updated to accept and forward `proxy_config` to underlying HTTP and Playwright fetcher instances.
- **Main actor flow**: Proxy configuration is now set up during component initialization (step 4) before creating the page fetcher, ensuring proxy settings are available from the first request.
- **Logging improvements**:
  - `LoggingManager` now automatically detects runtime environment (Apify vs. local) and uses appropriate logger (Actor.log on Apify, stdlib Logger locally).
  - `setup_logging(debug_mode)` method added for proper initialization of log level at startup.
  - All log messages across modules now include consistent module prefixes: `[Main]`, `[URLProcessor]`, `[PageFetcher]`, `[HTTPFetcher]`, `[Playwright]`, `[StateManager]`, `[NERHealthChecker]`, `[ResultHandler]`, `[ResultSetter]`.
  - Local logging uses human-readable format: `YYYY-MM-DD HH:MM:SS [LEVEL    ] message`.

### \[v0.9.2-beta] — 2026-02-09

#### Added

- **Concurrent URL processing**: The scraper now processes multiple URLs simultaneously starting with 2 concurrent URLs.
- **Per-URL extraction summary**: Each completed URL now logs a single summary line listing all successfully extracted fields (e.g., `Finished: http://example.com | company_name, business_address, emails`).

#### Changed

- **Logging cleanup for concurrency**: Per-field extraction logs (e.g., "Extracted 1 phone numbers") downgraded from INFO to DEBUG to prevent confusing interleaved output during concurrent processing. All extraction debug logs now include the source URL prefix for traceability.
- **State manager**: `current_url` field changed to `current_urls` (set) to track multiple in-flight URLs, with backward-compatible migration for older saved states.

### \[v0.9.1-beta] — 2026-01-25

#### Added

- **Email obfuscation decoding**: Enhanced email extraction with support for modern anti-scraping techniques. Now decodes Cloudflare email protection (XOR cipher), removes zero-width Unicode characters, and handles CSS-based obfuscation (RTL text reversal).

#### Fixed

- **Social media extraction**: Fixed critical bug where social media links in footer sections were not being detected.
- **Fax number extraction**: Fixed issue where fax numbers in long paragraphs were not being extracted.

### \[v0.9.0-beta] — 2026-01-22

#### Changed

- **Company name extraction**: Enhanced extraction logic with official German legal forms codelist integration. Improved text cleaning to better handle formatting artifacts, illegal characters, and various HTML rendering issues commonly found in imprint pages.

### \[v0.8.0-beta] — 2025-12-24

#### Added

- **VAT ID extraction**: New `extract_vat_id` module with ISO/IEC 7064 MOD 11,10 checksum validation. Extracts German VAT IDs (Umsatzsteuer-Identifikationsnummer) from imprint pages with context-aware scoring and support for various formatting styles (e.g., `DE 123 456 789`, `DE-123-456-789`).

### \[v0.7.0-beta] — 2025-12-22

#### Added

- **Fax number extraction**: New `extract_fax_numbers` module with intelligent pattern matching, context validation, and priority scoring. Extracts up to 10 ranked fax numbers from German imprint pages with support for international formats and address proximity detection.

### \[v0.6.1-beta] — 2025-11-21

#### Added

- **Discount Tier Pricing System**: Implemented comprehensive pricing manager with support for four discount tiers (FREE, BRONZE, SILVER, GOLD).
  - Automatically detects user's discount tier from `APIFY_ACTOR_PRICING_TIER` environment variable.
  - Dynamic pricing adjustments based on tier with detailed debug logging.
  - New `PricingManager` class in `src/utilities/pricing_manager.py`.

#### Changed

- **Pricing Integration**: Replaced `charging_manager` with `pricing_manager` throughout the codebase for discount tier support.

#### Removed

- **Automatic Billing Events**: Removed manual charging for events now handled automatically by Apify:
  - `actor-start`: Now charged automatically by Apify platform.
  - `successful-result`: Now charged automatically by Apify platform.
- Removed unused `charge_event` import from `result_handler.py`.

### \[v0.6.0-beta] — 2025-11-15

#### Changed

- **Headless Browser Configuration**: Replaced the binary `usePlaywright` toggle with a three-mode `headlessBrowser` option offering granular control over fetching strategy:
  - `headlessBrowserOn`: Always use browser (most reliable for JavaScript-heavy sites)
  - `headlessBrowserAuto`: Automatic mode with HTTP first, browser fallback (default, recommended)
  - `headlessBrowserOff`: HTTP only, no browser (fastest, but may fail on dynamic sites)
- **Backward Compatibility**: The deprecated `usePlaywright=true` setting now acts as an override, forcing `headlessBrowserOn` mode when explicitly set.

#### Removed

- **Optional Error Output**: Removed the optional error output that pushes URLs that failed to extract data into the dataset with their error message.

### \[v0.5.5-beta] — 2025-11-08

#### Changed

- **NER API Integration Update**: Migrated to new model-specific endpoint `/extract-names/german` for improved accuracy.
- **Base URL Configuration**: `NER_API_URL` environment variable now expects base URL only (e.g., `https://ner-api.domain.net`), endpoint path is automatically appended.
- **API Response Format**: Updated to support new response structure with `persons` and `raw_entities` fields.

#### Added

- Created `.env.example` template file for environment configuration.
- Added `ENV_SETUP.md` with comprehensive documentation for NER API setup, URL construction, and troubleshooting.

#### Fixed

- Improved confidence score extraction from `raw_entities` field with fallback to default value (0.8) when scores are missing.
- Enhanced URL handling to automatically strip trailing slashes from base URLs.

### \[v0.5.4-beta] — 2025-09-12

#### Added

- Added cost limit checking to automatically stop processing when the user-configured 'Maximum cost per run' is reached.

### \[v0.5.3-beta] — 2025-09-06

#### Changed

- Asynchronous Handling: The main extract method now creates and runs all extraction tasks concurrently using `asyncio.gather.`
- Small improvements in the company name extraction.

### \[v0.5.2-beta] — 2025-09-01

#### Changed

- The phone number and email output is now limited to 10 results.

### \[v0.5.1-beta] — 2025-08-30

#### Changed

- Added additional keywords for decision maker extraction.

### \[v0.5.0-beta] — 2025-08-28

#### Added

- Migration support: when the server is migrated on Apify's side, the Actor now persists state across runs using `Actor.set_value()` and `Actor.get_value()`.
- The time the website was finished scraping (`scraped_at`) can now be found under the metadata output.

#### Changed

- Slightly improved the decision maker extraction for better accuracy.
- Moved `imprint_url` output from the `metaData` to the standard output.

#### Fixed

- Bug in company name extraction that occasionally returned incorrect values.

### \[v0.4.0-beta] — 2025-08-27

This is a major update, marking the transition from alpha to the first beta release! The actor has been completely rewritten from the ground up to be more powerful, reliable, and flexible.

#### Added

- **Dual Fetching Technology**: The actor can now use a fast HTTP-based method for simple sites and automatically fall back to a powerful headless browser (Playwright) for modern, JavaScript-heavy websites. This dramatically increases the success rate of finding and scraping imprint pages.
- **Selective Data Extraction**: You now have full control over what data you want. A new input field `fieldsToExtract` allows you to choose the exact information you need (e.g., only company name and email).
- **Enhanced Configuration**: New input options like `metaData` and `errorOutput` have been added to give you more insight and control over the scraping process.
- **Proxy Support**: A proxy server provided by Apify can now be set in the input configuration.

#### Changed

- **Reliability Overhaul**: The entire codebase has been refactored. This results in better stability and significantly more accurate data extraction.
- **Smarter Scraping Logic**: The algorithms for identifying and parsing data have been completely reworked, leading to higher quality results across a wider variety of websites.
- **ML-Powered Decision Maker Extraction**: The logic for identifying decision-makers has been upgraded from simple keyword matching to a sophisticated NER (Named Entity Recognition) model, resulting in much higher accuracy.
- **Redesigned Input**: The actor's input configuration has been updated to be more intuitive and powerful, replacing the previous simple toggles with more granular controls.
- **Improved Output Structure**: The output JSON is now more cleanly structured and provides additional context, such as confidence scores for certain data points.

### \[v0.3.0-alpha] — 2025-07-17

#### Added:

- Handelsregister number and court extraction from imprint pages.
- Graceful shutdown handling with signal handlers (SIGINT, SIGTERM).
- Health check system for monitoring actor responsiveness.
- Semaphore-based concurrency control to limit simultaneous requests.
- Enhanced HTTP client timeout configuration.

#### Fixed:

- Critical bug where actor would hang indefinitely when URL processing timeout was reached.

#### Changed:

- Enhanced logging for better debugging and monitoring.

### \[v0.2.3-alpha] — 2025-06-24

#### Added:

- Timeout to automatically skip URLs that take too long to process.
- Added URL validation to filter out malformed URLs.
- Error loggings for unsuccessfully processed URLs can now be included it the output.

### \[v0.2.2-alpha] — 2025-05-02

#### Changed:

- Extracted Python directory for looking up German postal codes and cities.
- Emails are now sorted based on an algorithm that determents their relevance.

### \[v0.2.1-alpha] — 2025-05-02

#### Changed:

- Improvements to the extraction of addresses and emails.

#### Fixed:

- Doing the email extraction the script didn't properly filter Unicode encoded characters.

### \[v0.2.0-alpha] — 2025-04-24

#### Added:

- Search for social media links.

#### Changed:

- Improved performance of the decision maker extraction.

### \[v0.1.1-alpha] — 2025-04-17

#### Changed:

- Default settings: Decision Makers Search is now set as activated (`true`) in the default input settings.

#### Removed:

- Input `max_dept` option removed, since changes by the end user is not required for this actor's functionality.

#### Fixed:

- Decision maker search functionality is now working properly.

### \[v0.1.0-alpha] — 2025-04-14

#### Added:

- Initial release of the German Imprint Scraper.
- Extracts Company Name, Address, Phone, Email from Imprint pages.
- Optional extraction of Decision Makers.