# Changelog of Webhook Debugger, Logger & API Mocking Suite (`ar27111994/webhook-debugger-logger`) Actor

- **URL**: https://apify.com/ar27111994/webhook-debugger-logger/changelog.md
- **Full Actor documentation**: https://apify.com/ar27111994/webhook-debugger-logger.md

## Changelog

All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

### \[3.0.5] - 2026-04-21

#### Fixed (3.0.5)

- **Latency Metrics**: Make `processingTime` consistently represent server-side processing time only, excluding any configured `responseDelayMs` simulation from persisted log data.
- **Performance**: Cache compiled JSON Schema validators so repeated requests and stable hot-reload configurations do not trigger unnecessary recompilation on the webhook path.
- **Lifecycle**: Reset webhook-manager singleton state during test teardown and recreate the sync-service limiter after shutdown so repeated initialize/stop/start flows remain stable.
- **Lifecycle**: Cache the active `SyncService` limiter stop promise so disconnect cleanup can be retried safely after a partial stop failure, and so a later `start()` drains stale limiter cleanup before creating a replacement scheduler.
- **Shutdown Ordering**: Drain the HTTP listener before stopping `SyncService` and closing DuckDB so in-flight requests and readiness probes do not race read-model teardown.
- **DuckDB Lifecycle**: Drain both pooled and in-use DuckDB connections before resetting the singleton so repeated DB teardown and rebuild flows do not leave stale handles behind.
- **DuckDB Reset Coordination**: Keep the reset gate for new DuckDB callers while allowing already-queued writes and transactions to drain first, preventing reset deadlocks in the serialized write path.
- **DuckDB Init Invalidation**: Prevent an in-flight DuckDB initializer from republishing a stale singleton after `resetDbInstance()` clears teardown state, and allow clean reinitialization even when that late initializer ultimately fails.
- **Contracts & Docs**: Align dataset, output, OpenAPI, README, architecture, and API reference documentation with the finalized `processingTime` semantics and response-delay behavior.
- **Worker Cleanup**: Remove the custom script executor's `void` cleanup chaining and keep message/error/exit settlement deterministic even when worker listener cleanup throws.
- **Tests**: Add regression coverage for latency semantics, validator-cache reuse and memoized schema cache keys, DuckDB reset coordination with active reads plus queued write/transaction drain paths, malformed JSON sanitation persistence, restart-safe integration harness cleanup, shutdown-only sync error suppression, and spawned-process close-path resilience.
- **Tests**: Add focused regression coverage for `SyncService` stop/start retry paths and custom-script worker cleanup failure handling, and drive the touched files to 100% focused unit coverage before rerunning the stress suite.
- **Tests**: Make `setupTestApp()` fail fast when `node:fs/promises.mkdtemp()` is mocked incorrectly, and keep in-process integration suites on a real temp-directory implementation so `APIFY_LOCAL_STORAGE_DIR` isolation stays intact.
- **Tooling**: Remove the unused `cross-env` dependency and keep `npm run test:stress` on a direct Node + Jest invocation with `--expose-gc`, which works after a native Windows dependency install regenerates the local toolchain.

### \[3.0.4] - 2026-04-18

#### Fixed (3.0.4)

- **Apify**: Relax the dataset storage schema for `body` and `responseBody` so webhook events can be stored when request or response payloads are captured as any JSON-compatible values instead of only strings and objects.
- **Tests**: Add a regression test that keeps the Actor dataset schema aligned with the webhook payload shapes persisted by the runtime.

### \[3.0.3] - 2026-04-18

#### Fixed (3.0.3)

- **Documentation**: Convert README asset and document links to absolute GitHub URLs so the Apify Store listing renders images and documentation links correctly outside the repository context.

### \[3.0.2] - 2026-04-17

#### Fixed (3.0.2)

- **Apify**: Restore valid Actor input schema metadata by adding the missing nested alert channel descriptions required by Apify schema validation.
- **Apify**: Reorder `signatureVerificationSecret` so it appears directly above `signatureVerification` in the Actor input UI while preserving the same backward-compatible runtime mapping.
- **CI/CD**: Validate the Actor input schema with `apify validate-schema` alongside the web server schema so Apify build-time schema errors are caught before release.
- **CI/CD**: Split npm validation from npm publishing so `id-token: write` is granted only during release publishing, and keep Docker PR validation tags limited to the single synthetic PR tag.
- **CI/CD**: Normalize known flaky external link handling in the link check workflow by matching stable URL origin-and-path keys instead of full query-string URLs.
- **Tooling**: Stop hardcoding the Apify CLI package version inside the validation script by invoking the project-installed `apify` binary instead.
- **Tooling**: Add `@apify/input_secrets` as an explicit development dependency and align sync-version test fixtures with the production HTTP status key style.

### \[3.0.1] - 2026-04-15

#### Fixed (3.0.1)

- **CI/CD**: Make the release publish workflow use an explicit Jest invocation that force-exits after the suite completes.
- **CI/CD**: Make the release-only npm and Docker workflows report successful pull request checks without publishing artifacts, so Dependabot PRs do not remain stuck waiting on required checks.
- **CI/CD**: Make the required `Link Check` workflow run on every pull request so dependency-only PRs no longer remain stuck in an expected state when path filters skip the job.
- **Apify**: Move the webhook signing secret to a top-level `signatureVerificationSecret` input with `isSecret: true`, while keeping runtime compatibility with older nested secret values.
- **Forwarding**: Implement strict Content-Length parsing and enhance fallback logic in ForwardingService.
- **GitHub/NPM**: Update CI/CD, `package.json` and `package-lock.json` to use the latest versions of various dependencies.

### \[3.0.0] - 2026-04-02

#### Added (3.0.0)

- **Standby Web Server Contract**: Added `.actor/web_server_schema.json` and wired it through `.actor/actor.json` with `usesStandbyMode: true`, turning the Actor into a documented long-lived web service.
- **Expanded HTTP Surface**: Added or formalized routes for dashboard, runtime info, log queries, log detail, payload retrieval, replay, streaming, health, readiness, and system metrics.
- **Webhook Signature Verification**: Added provider-aware verification for Stripe, Shopify, GitHub, Slack, and custom HMAC integrations.
- **Standalone Self-Hosting Track**: Added a `runtime-standalone` target in the main `Dockerfile` for running the product outside the default Apify image path.
- **Environment Bootstrapping**: Added `.env.example` and project-level `.env` loading support for local CLI and self-hosted workflows.
- **Documentation Set**: Added `docs/api-reference.md`, `docs/architecture.md`, multiple operational playbooks, roadmap notes, and expanded publication/release guidance.

#### Improved (3.0.0)

- **Architecture**: Refactored the runtime into a clearer modular monolith split across `routes`, `middleware`, `services`, `repositories`, `consts`, and `utils`.
- **DuckDB Integration**: Migrated to `@duckdb/node-api` with cached instance management, pooled connections, and serialized writes for better stability under load.
- **Log Querying**: Upgraded `/logs` with richer filters, range parsing, sort controls, signature filters, and cursor-based pagination.
- **Hot Reloading**: Improved runtime config refresh with both Apify key-value-store polling and local filesystem watching.
- **Operator Controls**: Expanded input schema support for replay retries/timeouts, memory overrides, redaction, alerting, forwarding, verification, and response simulation.
- **Quality Tooling**: Added coverage matrix tooling, schema validation scripts, version synchronization helpers, and stronger CI/release automation.

#### Security (3.0.0)

- **Access Control**: Hardened management and ingress authentication behavior when `authKey` is configured.
- **Traffic Protection**: Split rate limiting between management endpoints and per-webhook ingestion flows.
- **Forwarding Safety**: Added recursion detection to block self-referential forwarding loops.
- **Data Protection**: Strengthened sensitive-header and body-path redaction options for captured traffic.

#### Testing (3.0.0)

- **Test Pyramid Restructure**: Reorganized tests into explicit `unit`, `integration`, and `e2e` suites.
- **Harness Depth**: Added reusable helpers for DB lifecycle management, process harnessing, payload fixtures, middleware testing, signature assertions, and mock orchestration.
- **Coverage Depth**: Added broad route, repository, middleware, service, utility, and script coverage for the new modular runtime.

### \[2.8.7] - 2026-01-15

#### Fixed (2.8.7)

- Removed redundant `dataset_csv` and `dataset_json` views from output schema (native "Storage" tab should be used for downloads).
- **Documentation**: Updated Stripe Playbook with correct official IP range links.

### \[2.8.6] - 2026-01-15

#### Fixed (2.8.6)

- Removed `clean` parameter from output schema views to resolve Apify Console type validation error.

### \[2.8.5] - 2026-01-14

#### Fixed (2.8.5)

- **Documentation**: Fixed bugs in `output_schema.json` views (parameter types and path formatting).
- **Documentation**: Updated download links to use `clean=true`.

### \[2.8.4] - 2026-01-14

#### Improved (2.8.4)

- **Documentation**: Consolidated "Self-Hosting" sections in `README.md` into a single, comprehensive guide.
- **Documentation**: Highlighted the "Activity-Based Retention" policy with a prominent alert block for better user visibility.

### \[2.8.3] - 2026-01-13

#### Fixed (2.8.3)

- **CLI Compatibility**: The Actor now correctly respects the `INPUT` environment variable when running via `npx`, overriding any local `INPUT.json` artifacts. This restores full stateless CLI functionality (e.g., `INPUT='{...}' npx ...`).
- **NPM Publishing**: Fixed an authentication issue in the CI/CD pipeline by correctly balancing OIDC provenance with legacy `setup-node` requirements.

#### Improved (2.8.3)

- **Startup UX**: Clarified startup logs to distinguish between "Initializing" (0 -> N webhooks) and "Scaling Up" (N -> M webhooks).
- **Log Noise**: Silenced the "Refreshed retention" log for insignificant updates (< 5 mins), preventing console spam during quick restarts.

### \[2.8.2] - 2026-01-13

#### Added (2.8.2)

- **UX**: Automatic `INPUT.json` creation for `npx` users (Zero-Conf Hot-Reload).

#### Security (2.8.2)

- **NPM**: Switched to OIDC Trusted Publishing for verified package provenance.

#### Improved (2.8.2)

- **Type Safety**: Removed all `@ts-ignore` directives in favor of proper JSDoc casting.
- **Documentation**: Clarified `npx` usage and fixed broken anchor links.

### \[2.8.1] - 2026-01-11

#### Fixed (2.8.1)

- **UI**: Fixed a regression where `{{VERSION}}` placeholders in `index.html` were not being fully replaced, causing raw template strings to be visible.

#### Improved (2.8.1)

- **UX**: The `/log-stream` SSE endpoint now sends an immediate `: connected` comment upon connection, preventing browsers from showing a "loading" state indefinitely while waiting for the first log event.

#### DevOps (2.8.1)

- **Docker Verification**: Added a comprehensive `local_docker_testing.md` guide and a new `verify-docker` CI/CD job. This ensures production-only bugs (like SSE compression issues) are caught in the container environment before deployment.

### \[2.8.0] - 2026-01-11

#### Added (2.8.0)

- **Robust Paginated Replay Search**: Implemented "Deep Search" for the `/replay` endpoint. It now defaults to checking the 1000 most recent items (fast path) but automatically paginates through older history if the target event is not found, preventing 404s for valid older events while avoiding OOM crashes.
- **SSRF Protection**: Added a shared `src/utils/ssrf.js` utility with DNS resolution and IP range validation. Applied this protection to both HTTP Forwarding and Replay APIs to prevent internal network scanning.
- **Community Standards**: Added `CODE_OF_CONDUCT.md`, `CONTRIBUTING.md`, `SECURITY.md`, `LICENSE`, and GitHub Issue Templates to meet open-source best practices.

#### Improved (2.8.0)

- **Configuration Refactoring**: Centralized input validation and default value logic (including `maxPayloadSize` and `rateLimitPerMinute`) into `src/utils/config.js` (`coerceRuntimeOptions`). This ensures consistent behavior between initial startup and hot-reloading.
- **Test Suite overhaul**:
  - Achieved **>90%** Statement/Line coverage and **>80%** Branch coverage (144 tests total).
  - Enforced strict type checking (removed `@ts-nocheck`) and standardized Jest mocks across all test files.
  - Added specific test suites for SSRF protection, Config Validation, and Edge Cases (`coverage_gaps.test.js`).
- **Security Hardening**:
  - **Header Stripping**: Automatically strips hop-by-hop headers and sensitive headers (Authorization, Cookie) during forwarding.
  - **Input Sanitization**: Enhanced error handling to prevent information leakage and sanitized `req.query` inputs.
- **Code Quality**: Applied Prettier formatting project-wide (trailing commas, consistent indentation) and resolved all ESLint/TypeScript errors.

#### Fixed (2.8.0)

- **Replay Memory Safety**: Prevented `dataset.getData()` from loading the entire dataset into memory by enforcing pagination limit (1000 items/page).
- **Hot-Reload Stability**: Fixed potential crashes during script re-compilation and ensured stale schemas are cleared on failure.
- **API Robustness**:
  - Added `req.forcedStatus` coercion validation.
  - Fixed handling of "all retries exhausted" in replay logic to prevent `undefined` errors.
  - Ensures correct handling of disconnected clients in SSE (`/log-stream`).

#### Refactored (2.7.2)

- **Code Reusability**: Centralized system constants and type definitions into `src/consts.js` and `src/typedefs.js` for better maintainability.
- **Test Infrastructure**: Standardized mock generation (`createDatasetMock`) and utility helpers to reduce test boilerplate and improve stability.

### \[2.7.1] - 2026-01-08

#### Improved (2.7.1)

- **Quality Assurance**: Achieved >90% test coverage for Lines/Statements and implemented strict type checking across the entire test suite.
- **Reliability**: Hardened forwarding retry logic and platform limit handling with comprehensive new test cases.
- **Maintainability**: Removed all `// @ts-nocheck` directives and standardized Jest mocks for long-term stability.

### \[2.7.0] - 2025-12-31

#### Added (2.7.0)

- **Dynamic Infrastructure Scaling & Hot-Reloading**:
  - Implemented **Actor Hot-Reloading**: Configuration changes (Auth Keys, Allowed IPs, Scripts, Schemas) now apply in real-time via `Actor.on('input')` without Actor restarts.
  - Implemented **urlCount Reconciliation**: The Actor now automatically generates missing webhooks on restart (or hot-reload) if the count is increased, preserving existing IDs.
  - Implemented **Retention Synchronization**: Existing webhooks are now automatically extended if the `retentionHours` setting is increased.
- **Enterprise-Grade Rate Limiting**:
  - Implemented **LRU (Least Recently Used)** eviction strategy for superior client protection under load.
  - Added **strict IP validation** for proxy headers (`X-Forwarded-For`/`X-Real-IP`) to prevent spoofing and malformed data propagation.
  - Added robust validation for `limit`, `windowMs`, and `maxEntries` parameters.
- **Security & Privacy Hardening**:
  - **Enhanced Privacy (PII Masking)**: Integrated an IP masking helper to obfuscate sensitive client data in logs (e.g., `192.168.1.****`).
  - **XSS Protection**: Implemented `escapeHtml` sanitization for all user-controlled output in the "Locked" auth screen and error pages.
  - **Universal Auth UI**: Created a unified landing page with hardened auth detection and zero-leak link generation.
- **Enterprise Integration Suite**:
  - **Forwarding Security**: Automatically strip sensitive headers (`Authorization`, `Cookie`, etc.) during real-time forwarding and replay.
  - **Forwarding Controls**: Added `forwardHeaders` toggle for granular control over header transmission.
  - **Documentation Playbooks**: Added new guides for `Revenue Recovery`, `Low-Code Bridge`, and `Legacy Migration`.
  - **Example Saturation**: Added 3+ comprehensive end-to-end input/output examples to satisfying Apify Quality Score requirements.

#### Improved (2.7.0)

- **Auth Security**: Eliminated all `authKey` leakage by removing automatic query parameter propagation in management links and API responses.
- **Replay Accuracy**: Prioritized nanoid `id` over timestamps in event lookup to eliminate collisions during high-concurrency replays.
- **Architectural Reliability**:
  - **Background Pruning**: Moved hit cleanup to a non-blocking background interval (60s).
  - **Middleware Orchestration**: Guaranteed immediate response delivery before racing background tasks against a 10s timeout.
- **Comprehensive Verification**: Expanded test suite to **85 tests** (17 files), achieving full coverage of security, reliability, dynamic scaling, hot-reloading, and edge case scenarios with zero memory/timer leaks.
- **Repository Health**: Removed heavy binary assets and updated `.gitignore` for a leaner, faster repository.

### \[2.6.0] - 2025-12-27

#### Added (2.6.0)

- **Management Rate Limiting**: Implemented a memory-efficient rate limiter for `/info`, `/logs`, and `/replay` endpoints to prevent brute-force attacks on API keys.
- **Sensitive Data Masking**: Added opt-in masking for sensitive headers (Authorization, Cookie, etc.) in captured logs to enhance user privacy.
- **Resource Offloading**: Dataset schemas and processing have been optimized for better platform performance.
- **Detailed Log Views**: Added "Full Payloads" view to the Apify Dataset for easier inspection of headers and bodies in the console.

#### Improved (2.6.0)

- **SSE Scalability**: Refactored Server-Sent Events to use a high-performance global heartbeat mechanism, significantly reducing memory overhead per concurrent listener.
- **Input Schema Quality**: Added detailed tooltips, grouping, and prefill examples for all v2.0+ features.
- **Documentation**: Major README overhaul with new troubleshooting guides, professional usage examples, and performance metrics.

### \[2.5.0] - 2025-12-26

- **Standby Mode Enabled**: Formally added `"usesStandbyMode": true` to `actor.json` for superior performance and persistence.
- **QA Success Logic**: The Actor now yields an immediate "Server Ready" result to the dataset on startup. This ensures compliance with Apify's automated QA tests (which require a result within 5 minutes).
- **Test & Exit**: Added a hidden `testAndExit` input to allow automated health checks to complete and exit cleanly.
- **Readiness Probes**: Implemented explicit handling for Apify's `x-apify-container-server-readiness-probe` header in the root endpoint.

#### Fixed (2.5.0)

- Resolved "Under maintenance" flag by ensuring the Actor does not timeout during automated platform tests.
- Improved version consistency across all project manifest files.

### \[2.4.2] - 2025-12-22

#### Added (2.4.2)

- **Stress Testing**: Added a comprehensive stress test suite to verify the system's stability under high load.
- **Documentation**: Added missing Pricing, FAQ, Support, Security & Permissions, and Privacy sections to `README.md`.
- **Schema Quality**: Populated `dataset_schema.json` with concrete example values for all fields.
- **Reliability**: Implemented retry logic with exponential backoff (3 attempts) for both HTTP Forwarding and the `/replay` API.

#### Fixed (2.4.2)

- **Stress Testing**: Fixed a memory leak in the stress test suite.
- **Stress Testing**: Fixed a timeout issue in the stress test suite.

### \[2.4.1] - 2025-12-22

#### Fixed (2.4.1)

- **ESM Compatibility**: Fixed `eventsource` import in `demo_cli.js` to support latest named exports.
- **Version Sync**: Synchronized project version across all manifests.

### \[2.4.0] - 2025-12-22

#### Added (2.4.0)

- **Comprehensive Test Suite**: 15+ Automated tests covering unit, integration, and E2E scenarios.
- **Testing Framework**: Integrated Jest and Supertest with full ESM/VM support.
- **Architectural Polish**: Refactored `main.js` to decouple the Express app from the server listener for professional testability.

### \[2.3.1] - 2025-12-22

#### Fixed (2.3.1)

- **Edge Case Hardening**: Added input sanitization and hard-caps for `/logs` API.
- **Safety**: Added object validation for custom response headers to prevent runtime crashes.
- **Maintenance**: Minor documentation cleanup.

### \[2.3.0] - 2025-12-22

#### Added (2.3.0)

- **v2.2 Comprehensive Robustness Update**:
  - Global Express error handling to catch malformed bodies and unhandled exceptions.
  - Hardened state persistence with try-catch and validation logic.
  - Standardized timeouts (10s) for HTTP Forwarding and Replay APIs.
  - Improved error reporting for Replay API (distinguishes between timeouts and target rejections).
  - Resilient initialization to survive storage-layer transients.

### \[2.1.1] - 2025-12-22

#### Fixed (2.1.1)

- **Hotfix**: Added missing `editor` fields to `input_schema.json` to resolve Apify platform validation errors.

### \[2.1.0] - 2025-12-22

#### Added (2.1.0)

- **v2.1 Custom Scripting**: Allow users to provide JavaScript snippets for advanced data transformation before logging.
- Prepared submission for the $1M Challenge with a unified enterprise feature set.

### \[2.0.0] - 2025-12-21

#### Added (2.0.0)

- **v2.0 Enterprise Features**:
  - **Security**: CIDR IP Whitelisting and API Key/Bearer Auth support.
  - **Mocking**: Custom response bodies, status codes, and headers.
  - **Simulation**: Configurable response delay (latency simulation up to 10s).
  - **Workflows**: Real-time HTTP Request Forwarding (pipe webhooks to other APIs).
  - **Replay API**: Endpoint to resend captured events to any destination.
  - **Validation**: JSON Schema validation with professional error reporting.
- New "Enterprise" sections in input schema for easier configuration.

#### Improved (2.0.0)

- Middleware refactored into a high-performance pipeline.
- Upgraded dependencies: `ajv` for validation and `ip-range-check` for security.

### \[1.1.0] - 2025-12-21

#### Added (1.1.0)

- Narrated walkthrough video integrated into README for Quality Score boost.
- 5+ comprehensive FAQs added to documentation.
- CSV Output Format preview table in README.
- Example values added to Dataset Schema for improved platform documentation.
- Developer Support Guarantee (24h response time) added.

#### Improved (1.1.0)

- Input Schema sections and tooltips polished for better UX.
- \[CRITICAL] Safety checks added to logging middleware to prevent data loss.
- GitHub repository synchronized with finalized production assets.

### \[1.0.0] - 2025-12-20

#### Added (1.0.0)

- Initial release of Webhook Debugger & Logger.
- Standby mode support for sub-10ms response times.
- Real-time event streaming via Server-Sent Events (SSE).
- Dynamic webhook URL generation (1-10 IDs per run).
- /logs API with advanced filtering (method, status, ID).
- Response status code override via `?__status=XXX`.
- Robust body parsing for JSON, XML, and URL-encoded form data.
- Payload size limit (configurable, default 10MB).
- Detailed integration guides for Zapier and Make.
- Table view configuration for Apify Dataset.
- CSV/JSON export links in Output tab.
- PPE (Pay-per-Event) pricing support.