# Changelog of Website Content Crawler for LLM's (`salesblaster-ai/website-content-crawler`) Actor

- **URL**: https://apify.com/salesblaster-ai/website-content-crawler/changelog.md
- **Full Actor documentation**: https://apify.com/salesblaster-ai/website-content-crawler.md

## Changelog

All notable changes to the LLM-Optimized Website Content Crawler will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

### \[2.0.0] - 2026-01-09

#### Breaking Changes

##### Input Schema

- **Changed**: `startUrls` (array) replaced with single `startUrl` (string)
  - Old: `{ "startUrls": [{ "url": "https://example.com" }] }`
  - New: `{ "startUrl": "https://example.com" }`
- **Renamed**: `maxRequestsPerCrawl` → `maxPages` (clearer naming)
- **Removed**: `maxRequestsPerUrl` (no longer applicable for single-site crawling)
- **Added**: New configuration options:
  - `sitemapStrategy`: Control sitemap discovery (AUTO, SITEMAP\_FIRST, CRAWL\_LINKS)
  - `includePaths`: Whitelist URL path prefixes
  - `excludePaths`: Blacklist URL path prefixes (default excludes admin/login/cart)
  - `excludeUrlRegex`: Regex pattern to exclude URLs (default excludes media/binary files)
  - `chunkingOptions`: Configure chunk size and overlap
  - `extractContacts`: Enable/disable contact extraction

##### Output Format

- **Changed from full-page records to semantic chunks:**
  - Old output: One record per page with full `markdownContent` field
  - New output: Multiple records per page (chunks) with the following schema:
    ```json
    {
      "site": "example.com",
      "url": "https://example.com/about",
      "title": "About Us",
      "chunkIndex": 0,
      "chunkCount": 3,
      "headingPath": "About > Team > Leadership",
      "markdown": "# Team\n\n...",
      "contentType": "marketing",
      "quality": { "score": 85, "textLength": 1500, "linkDensity": 0.03 },
      "crawledAt": "2026-01-09T12:00:00Z",
      "datasetName": "default"
    }
    ```
- **Removed fields**: `metaDescription`, `linkCount`
- **Added fields**: `chunkIndex`, `chunkCount`, `headingPath`, `contentType`, `quality`, `site`

##### Contact Data Storage

- Contacts now stored in KeyValue store under `OUTPUT` key (not in dataset)
- Format:
  ```json
  {
    "summary": {
      "totalEmails": 5,
      "totalPhones": 3,
      "totalSocialLinks": 8,
      "socialBreakdown": { "linkedin": 3, "twitter": 2, "facebook": 3 }
    },
    "contacts": {
      "emails": ["contact@example.com"],
      "phones": ["+14155552671"],
      "social": [{ "platform": "linkedin", "url": "..." }]
    },
    "crawlStats": {
      "pagesVisited": 20,
      "pagesSkipped": 0,
      "errors": 0
    },
    "generatedAt": "2026-01-09T12:00:00Z"
  }
  ```

#### Added

##### LLM-Optimized Content Extraction

- **Mozilla Readability integration**: Extracts main content, removing nav/footer/cookie banners
- **Fallback selectors**: Uses heuristics (`main`, `article`, `[role="main"]`) if Readability fails
- **Markdown post-processing**: Removes boilerplate patterns (cookie policy, copyright, social share buttons)
- **Quality scoring**: Assigns quality score (0-100) based on text length, link density, and structure
- **Content type classification**: Categorizes pages as `blog`, `docs`, `legal`, `product`, `marketing`, or `other`

##### Semantic Chunking

- **Heading-based chunking**: Splits content by heading hierarchy (h1-h6)
- **Configurable chunk size**: Default 2000 chars with 200 char overlap
- **Heading path tracking**: Maintains breadcrumb trail (e.g., "Home > About > Team")
- **Smart overlap**: Tries to break at sentence/paragraph boundaries

##### Contact Extraction

- **Email extraction**: Regex-based with false positive filtering (excludes example.com, image extensions)
- **Phone number extraction**: From visible text and `tel:` links
- **Phone normalization**: E.164 format using libphonenumber-js
- **Social link extraction**: Detects and categorizes LinkedIn, Instagram, Facebook, X/Twitter, TikTok, YouTube, GitHub
- **Deduplication**: Automatic deduplication across all crawled pages

##### Sitemap Support

- **Three strategies**:
  - `AUTO`: Try sitemap first, fall back to link crawling
  - `SITEMAP_FIRST`: Fail if no sitemap found (strict mode)
  - `CRAWL_LINKS`: Ignore sitemaps, only follow links
- **Sitemap index support**: Recursively parses sitemap indexes
- **Path filtering**: Applies includePaths/excludePaths to sitemap URLs

##### URL Filtering

- **Path-based filtering**: Include/exclude by URL path prefix
- **Regex filtering**: Exclude URLs matching regex pattern
- **Default exclusions**: Automatically skips common non-content paths (admin, login, wp-admin, cart, checkout)
- **Default file exclusions**: Skips binary/media files (PDF, images, videos, CSS, JS)

#### Changed

##### Crawl Behavior

- **From multi-domain to single-site**: Focus on one domain per crawl
- **Conservative concurrency**: Default 5 concurrent requests (was 1000)
- **Simple page budget**: Uses `maxPages` limit instead of per-domain counters
- **Same-site enforcement**: Uses eTLD+1 (registrable domain) comparison via tldts
- **Removed KV store domain counters**: No longer needed for single-site crawling

##### Architecture

- **New module structure**: Organized into lib/ with url, sitemap, extraction, chunking, summary modules
- **Explicit dependencies**: All dependencies now listed in package.json (apify, crawlee, playwright, @mozilla/readability, jsdom, etc.)
- **Type-safe input**: Zod schema validation with helpful error messages
- **CJS build**: Kept CommonJS for Apify compatibility

##### Docker

- **Multi-stage build**: Builder stage for TypeScript compilation, slim production image
- **Removed root Dockerfile**: Only .actor/Dockerfile is used (Apify convention)
- **Explicit dependency installation**: No reliance on base image for dependencies

#### Removed

- **Legacy INPUT\_SCHEMA.json**: Deleted to avoid confusion (use .actor/input\_schema.json)
- **Per-domain KV counters**: No longer needed for single-site crawling
- **Multi-domain support**: Simplified to single-site per run
- **Full body HTML extraction**: Now extracts only main content

#### Migration Guide

If you were using v1.x:

1. **Update input format**:
   ```javascript
   // OLD (v1.x)
   {
     "startUrls": [{ "url": "https://example.com" }],
     "maxRequestsPerCrawl": 50000,
     "maxRequestsPerUrl": 5,
     "datasetName": "my-dataset"
   }

   // NEW (v2.0)
   {
     "startUrl": "https://example.com",
     "maxPages": 20,
     "datasetName": "my-dataset"
   }
   ```

2. **Update downstream consumers**:
   - Change from processing one record per page to multiple chunks per page
   - Access contact data from KV store `OUTPUT` key (not dataset)
   - Adapt to new fields: `chunkIndex`, `chunkCount`, `headingPath`, `quality`, `contentType`

3. **Review filtering options**:
   - Set `includePaths` if you only want specific sections (e.g., `/blog`, `/docs`)
   - Adjust `excludePaths` if defaults don't match your needs
   - Use `excludeUrlRegex` for advanced filtering

4. **Configure chunking** (optional):
   - Adjust `chunkingOptions.maxChars` for your LLM's context window
   - Increase `chunkingOptions.overlapChars` for better context preservation

5. **Disable contacts extraction** (optional):
   - Set `extractContacts: false` if not needed

#### Dependencies

New dependencies added:

- `@mozilla/readability`: ^0.5.0 (content extraction)
- `jsdom`: ^24.0.0 (DOM parsing for Readability)
- `libphonenumber-js`: ^1.11.0 (phone normalization)
- `tldts`: ^6.1.0 (domain extraction)
- `xml2js`: ^0.6.2 (sitemap parsing)
- `zod`: ^3.24.0 (input validation)

Existing dependencies (now explicit):

- `apify`: ^3.2.0
- `crawlee`: ^3.11.0
- `playwright`: ^1.40.0
- `node-html-markdown`: ^1.3.0

#### Technical Details

##### Module Structure

```
src/
├── main.ts                    # Actor entry point
├── routes.ts                  # Playwright router
├── types.ts                   # Shared types
└── lib/
    ├── url/                   # URL utilities (normalize, filter, domain)
    ├── sitemap/               # Sitemap discovery and parsing
    ├── extraction/            # Content + contact extraction, quality scoring
    ├── chunking/              # Markdown chunking by headings
    └── summary/               # Contact aggregation
```

##### Performance

- Default crawl settings: 5 concurrency, 20 pages max
- Memory efficient: Chunks are streamed to dataset, not held in memory
- Conservative session pool: maxPoolSize = concurrency \* 2

##### Quality Assurance

- Input validation with Zod (catches errors early)
- Error handling: Graceful degradation, saves error records to dataset
- Logging: Structured logs with context for debugging

***

### \[1.0.0] - 2025-01-01

Initial multi-domain crawler implementation.

- Multi-site crawling with per-domain limits
- Full body HTML → Markdown conversion
- Dataset output (one record per page)
- Aggressive concurrency (1000 workers)