# Website Freshness Detector - Last Update Signals + Stale Flag (`santamaria-automations/website-freshness-detector`) Actor Probe any website for signals of when it was last updated: sitemap lastmod, Last-Modified header, footer copyright year, and RSS/Atom feed dates. Returns a stale\_flag verdict (active / slow / stale / dead) so agencies can prioritize rebuild pitches. Pay-per-result. - **URL**: https://apify.com/santamaria-automations/website-freshness-detector.md - **Developed by:** [NanoScrape](https://apify.com/santamaria-automations) (community) - **Categories:** Developer tools - **Stats:** 1 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $3.00 / 1,000 url probeds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Website Freshness Detector **How recently was this site actually touched? One HTTP probe, one verdict.** Point this actor at any website URL and it returns a structured freshness snapshot: when the sitemap was last updated, what the Last-Modified header says, the footer copyright year, the newest post in the RSS/Atom feed, and a stale_flag verdict that rolls all four signals into a single label. Pure Go HTTP with a real Chrome 124 TLS fingerprint. No browser, no JavaScript rendering. Fast enough to scan a lead list of thousands in minutes. ### What it does For each URL it probes up to four independent freshness signals: 1. **sitemap.xml lastmod.** Fetches `/sitemap.xml`, `/sitemap_index.xml`, `/wp-sitemap.xml`, plus any `Sitemap:` directive from robots.txt. Parses XML and returns the max `` across all URL entries. Handles sitemap-index files by recursing one level into the first 3 child sitemaps. 2. **Homepage Last-Modified header.** Reads the `Last-Modified` HTTP response header on the homepage. 3. **Footer copyright year.** Scans the last 4000 bytes of the homepage HTML for a copyright anchor and picks the highest 4-digit year nearby. Handles `©`, `©`, `(c)`, `Copyright 2024`, and ranges like `2018 - 2024`. 4. **RSS/Atom feed pubDate.** Discovers feeds from `` tags plus well-known paths (`/feed`, `/rss`, `/rss.xml`, `/feed.xml`, `/atom.xml`, `/index.xml`) and extracts the newest `` (RSS) or `` (Atom). The verdict is computed against the most-recent signal across all four. ### Use cases - **Web-design agency lead prioritization.** Filter your Google Maps or lead-list export for `stale_flag = "stale"` and pitch a rebuild. The concrete "your site's sitemap says it was last updated in 2019" line lands harder than a generic "your site looks old" cold email. - **Competitive analysis.** Snapshot competitor sites monthly to see who is actively publishing and who has gone dormant. - **Content marketing audits.** Identify blogs that stopped publishing so you can pitch a content-refresh or ghostwriting engagement. - **Portfolio monitoring.** Nightly snapshot of a domain portfolio to catch abandoned properties before they get delisted from search. - **M&A due diligence.** Combine with the Compliance Scanner and Tech Stack Detector for a fast website-health snapshot on target companies. ### When to use it Pair with **Broken Link Checker** and **SEO Metadata Extractor** to build a complete "website health" dossier per lead. Freshness answers "when was it last touched?", broken links answer "is it maintained?", SEO metadata answers "is it findable?". ### Input | Parameter | Type | Default | Description | |---|---|---|---| | `urls` | array | **required** | Website homepage URLs to scan. Bare domains accepted. | | `companies` | array | `[]` | Alternative paired input: `[{"company_id":"...","website_url":"..."}]` for CRM join-back. | | `probeTimeoutMs` | integer | `5000` | HTTP timeout per fetch. Cap 15000. | | `respectRobotsTxt` | boolean | `true` | Honor the site's robots.txt on sitemap/feed probes. | | `concurrency` | integer | `5` | Number of URLs scanned in parallel. | | `proxyConfiguration` | object | Apify datacenter | Optional proxy. Datacenter is fine for freshness probes. | ### Output shape One row per input URL: ```json { "url": "https://example.com/", "final_url": "https://www.example.com/", "status_code": 200, "sitemap_url": "https://www.example.com/sitemap.xml", "sitemap_lastmod": "2024-11-14", "homepage_last_modified": "2024-11-14", "footer_copyright_year": 2024, "latest_feed_pubdate": "2024-11-01", "most_recent_signal": "2024-11-14", "most_recent_source": "sitemap", "stale_flag": "active", "months_since_last_signal": 3.2, "signals_found": 4, "signal_note": null, "probed_at": "2026-07-21T09:00:00Z", "error": null } ```` Missing signals serialize as `null`. When the homepage fetch fails: ```json { "url": "https://broken.example/", "status_code": 0, "sitemap_lastmod": null, "homepage_last_modified": null, "footer_copyright_year": null, "latest_feed_pubdate": null, "most_recent_signal": null, "stale_flag": "unknown", "signals_found": 0, "probed_at": "2026-07-21T09:00:00Z", "error": "request timed out" } ``` #### Stale flag threshold table | flag | Meaning | Age of most recent signal | |---|---|---| | `active` | Recently updated | Within 6 months | | `stale` | Rebuild candidate | 6 to 18 months | | `abandoned` | Long-dormant | Over 18 months | | `unknown` | No usable signal, or only a footer copyright year was available | See `signal_note` for the reason | #### Signal precedence Four signals are ranked from strongest to weakest: **sitemap > homepage\_header > feed > footer\_copyright\_year**. Whichever signal has the most-recent date wins `most_recent_signal` and drives `stale_flag`. `footer_copyright_year` is the weakest signal on purpose. Footer years are frequently forgotten (a live 2024 site can still say `© 2019`), and they only carry year granularity (normalised to `YYYY-01-01`). If a probe finds only a footer year and nothing else, `stale_flag` is set to `unknown` and `signal_note` explains why — better to skip the row than to mislabel a fresh site as `abandoned` because someone forgot to bump the footer. The `most_recent_source` field tells you which signal drove the verdict so you can weigh it in your outreach. A `stale_flag = "stale"` driven by `sitemap` or `feed` is high-conviction — those dates almost always reflect real publishing activity. ### Pricing - **$0.001** per run start - **$0.003** per URL probed Failed probes are billed too. They still consumed a run slot and returned a `stale_flag = "unknown"` row so you can retry the URL later. **1000 URLs = $3.00.** ### Related Actors - [Broken Link Checker](https://apify.com/santamaria-automations/broken-link-checker). Crawl a site for 404s, redirects, SSL errors, and dead outbound links. - [Compliance Scanner](https://apify.com/santamaria-automations/compliance-scanner). Detect Impressum, privacy policy, and cookie consent for GDPR/Abmahnung risk scoring. - [SEO Metadata Extractor](https://apify.com/santamaria-automations/seo-metadata-extractor). Pull titles, meta tags, Open Graph, Twitter Cards, and JSON-LD schema. - [Website Tech Stack Detector](https://apify.com/santamaria-automations/website-tech-detector). Identify the CMS, frameworks, analytics, and hosting behind any site. - [SSL Certificate Inspector](https://apify.com/santamaria-automations/ssl-certificate-inspector). Inspect TLS certificates and TLS versions. ### Issues & Feedback Found a signal we should detect, or a site type where the verdict feels wrong? Please open an issue on the actor page. # Actor input Schema ## `urls` (type: `array`): List of website homepage URLs to probe. Bare domains (example.com) or full URLs are accepted. Each URL produces one result row. ## `companies` (type: `array`): Alternative to urls. Pass an array of {company\_id, website\_url} pairs when you want the company\_id echoed back on the output row for CRM join-back. Both urls and companies may be supplied together. ## `probeTimeoutMs` (type: `integer`): HTTP timeout applied to every fetch (homepage + sitemap + feed probes). Slow origin servers may need a higher value. ## `respectRobotsTxt` (type: `boolean`): Honor the site's robots.txt when probing sitemap and feed paths. Homepage is always fetched. Disable only when auditing a site you own or operate. ## `concurrency` (type: `integer`): Number of URLs probed in parallel. Freshness probes are lightweight but the default of 5 keeps load on each target site polite. ## `proxyConfiguration` (type: `object`): Optional Apify proxy settings. Datacenter proxy is fine for freshness probes. Enable if your run is hitting rate limits. ## `preFetchedByURL` (type: `object`): Map of URL -> pre-fetched homepage result (with response headers for Last-Modified). When set, this actor uses the provided HTML/headers for its homepage signals instead of fetching. Sitemap and RSS probes still run normally. Used by the google-maps-scraper orchestrator. Direct users should leave this blank. ## Actor input object example ```json { "urls": [ "https://vercel.com/" ], "companies": [], "probeTimeoutMs": 5000, "respectRobotsTxt": true, "concurrency": 5, "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": [ "BUYPROXIES94952" ] } } ``` # Actor output Schema ## `results` (type: `string`): Dataset with one row per URL. Fields: sitemap\_lastmod, homepage\_last\_modified, footer\_copyright\_year, latest\_feed\_pubdate, most\_recent\_signal, stale\_flag. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "urls": [ "https://vercel.com/" ], "companies": [] }; // Run the Actor and wait for it to finish const run = await client.actor("santamaria-automations/website-freshness-detector").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "urls": ["https://vercel.com/"], "companies": [], } # Run the Actor and wait for it to finish run = client.actor("santamaria-automations/website-freshness-detector").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "urls": [ "https://vercel.com/" ], "companies": [] }' | apify call santamaria-automations/website-freshness-detector --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=santamaria-automations/website-freshness-detector", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Website Freshness Detector - Last Update Signals + Stale Flag", "description": "Probe any website for signals of when it was last updated: sitemap lastmod, Last-Modified header, footer copyright year, and RSS/Atom feed dates. Returns a stale_flag verdict (active / slow / stale / dead) so agencies can prioritize rebuild pitches. Pay-per-result.", "version": "1.0", "x-build-id": "KhyBN8OAVeSjuVEfz" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/santamaria-automations~website-freshness-detector/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-santamaria-automations-website-freshness-detector", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/santamaria-automations~website-freshness-detector/runs": { "post": { "operationId": "runs-sync-santamaria-automations-website-freshness-detector", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/santamaria-automations~website-freshness-detector/run-sync": { "post": { "operationId": "run-sync-santamaria-automations-website-freshness-detector", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "urls" ], "properties": { "urls": { "title": "Website URLs", "type": "array", "description": "List of website homepage URLs to probe. Bare domains (example.com) or full URLs are accepted. Each URL produces one result row.", "items": { "type": "string" } }, "companies": { "title": "Companies (paired input)", "type": "array", "description": "Alternative to urls. Pass an array of {company_id, website_url} pairs when you want the company_id echoed back on the output row for CRM join-back. Both urls and companies may be supplied together." }, "probeTimeoutMs": { "title": "Probe Timeout (ms)", "minimum": 1000, "maximum": 15000, "type": "integer", "description": "HTTP timeout applied to every fetch (homepage + sitemap + feed probes). Slow origin servers may need a higher value.", "default": 5000 }, "respectRobotsTxt": { "title": "Respect robots.txt", "type": "boolean", "description": "Honor the site's robots.txt when probing sitemap and feed paths. Homepage is always fetched. Disable only when auditing a site you own or operate.", "default": true }, "concurrency": { "title": "Concurrency", "minimum": 1, "maximum": 20, "type": "integer", "description": "Number of URLs probed in parallel. Freshness probes are lightweight but the default of 5 keeps load on each target site polite.", "default": 5 }, "proxyConfiguration": { "title": "Proxy Configuration", "type": "object", "description": "Optional Apify proxy settings. Datacenter proxy is fine for freshness probes. Enable if your run is hitting rate limits.", "default": { "useApifyProxy": true, "apifyProxyGroups": [ "BUYPROXIES94952" ] } }, "preFetchedByURL": { "title": "Pre-fetched HTML by URL (internal, orchestrator use only)", "type": "object", "description": "Map of URL -> pre-fetched homepage result (with response headers for Last-Modified). When set, this actor uses the provided HTML/headers for its homepage signals instead of fetching. Sitemap and RSS probes still run normally. Used by the google-maps-scraper orchestrator. Direct users should leave this blank." } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```