# Web Hygiene MCP - Site Checks, URL Status & Feeds for AI Agents (`bruco3/web-hygiene-mcp`) Actor

MCP server for AI agents: robots.txt/llms.txt/sitemap/feed overview of any site, URL status with redirect chains, bulk link checks, feed reading. Every answer from live requests to the site, bounded to seconds. Pay per tool call and per result.

- **URL**: https://apify.com/bruco3/web-hygiene-mcp.md
- **Developed by:** [Luca Pietrini](https://apify.com/bruco3) (community)
- **Categories:** MCP servers, Agents, Developer tools
- **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $5.00 / 1,000 tool calls

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Web Hygiene MCP — live website checks for AI agents

An MCP server that gives Claude, ChatGPT, Cursor and any MCP-capable agent **seven tools that answer from the live web**: what a site publishes, whether URLs actually work, what its feeds contain, and whether the sources a text cites really say what is claimed. Every answer is the result of real HTTP requests made at call time — never a cache, never the model's memory — and every tool is bounded so it returns in seconds, mid-conversation.

Built on the same engine as the **Sitemap URL Extractor**, **Broken Link Checker** and **Feed Monitor** Actors; the tools point to them when a job is too big for a chat turn.

### Tools

| Tool | What the agent gets | Typical prompt |
|---|---|---|
| `site_overview(site)` | robots.txt (sitemaps declared, AI crawlers addressed, blanket disallow), `llms.txt` presence, sitemaps found with a URL count sample, feeds found | *"Before we cite anything from example.com, what does the site publish and allow?"* |
| `list_site_urls(site_or_sitemap, max_urls, include_regex, exclude_regex)` | URLs from the site's sitemaps with `lastmod`, filtered, up to 2,000 | *"List the blog posts on example.com published this year"* |
| `check_url(url)` | Status, full redirect chain, final URL, response time, content type | *"Does this link still work? Where does it redirect?"* |
| `check_links(urls \| site_or_sitemap, max_urls, problems_only)` | Bulk status check: 404s, 5xx, timeouts, SSL errors, redirect chains, slow pages | *"Check every link in this document"* · *"Any broken pages on our site?"* |
| `discover_feeds(site)` | The site's RSS/Atom/JSON feeds | *"Does this blog have a feed?"* |
| `read_feed(feed_or_site, max_items, include_content)` | Normalised items: title, link, date in UTC, author, plain-text summary, categories, enclosures | *"What did they publish this week?"* |
| `verify_citations(text \| urls \| citations, check_archive, max_citations)` | Per cited URL: verdict (`verified`, `reachable`, `mismatch`, `not-found`, `redirected-home`, `paywalled`, `blocked`, `unreachable`), confidence, reasons, the page's real title/date/author, quote match evidence, Internet Archive copy for dead links | *"Check the sources in the answer you just gave me"* · *"Is this quote really on that page?"* |

All tools accept a bare domain and discover the rest; all return structured JSON plus a `nextSteps` / `fullSiteTool` / `batchTool` hint naming the Actor to use for whole-site or whole-document work.

### Connect

Streamable HTTP endpoint (Standby mode — always on, no run to start):

```
https://bruco3--web-hygiene-mcp.apify.actor/mcp
Authorization: Bearer <YOUR_APIFY_API_TOKEN>
```

Claude Desktop / Cursor style configuration:

```json
{
  "mcpServers": {
    "web-hygiene": {
      "url": "https://bruco3--web-hygiene-mcp.apify.actor/mcp",
      "headers": { "Authorization": "Bearer YOUR_APIFY_API_TOKEN" }
    }
  }
}
```

Or add it through the [Apify MCP server](https://mcp.apify.com), which exposes any Actor as a tool.

### Try it — three prompts

1. *"Check the sources in the answer you just gave me: does each link exist, and is the quote really on the page?"* → `verify_citations`
2. *"Before we cite anything from example.com, what does the site publish and allow — robots.txt, llms.txt, sitemaps, feeds?"* → `site_overview`
3. *"Check every link in this README and tell me which are broken or redirected."* → `check_links`

**Claude Code users**: install the ready-made plugin (two skills + this server, asks for your Apify token once): `/plugin marketplace add Luca-003/web-hygiene-claude-plugin` then `/plugin install web-hygiene@luca-003`.

### Why an agent needs this

Language models are confident about links and sites they have never fetched. That is how hallucinated citations, dead links in generated content and crawls of forbidden paths happen. These tools give the agent a cheap, deterministic way to **look before it claims**:

- verify that a URL exists and where it really leads before including it in an answer;
- confirm that a cited page really carries the quoted passage, title and date — the check that turns "the model cited something" into "the citation holds";
- check the site's robots.txt and `llms.txt` before fetching or quoting;
- read what a site actually publishes (sitemap, feed) instead of guessing page URLs;
- validate every link in a document it just wrote.

### Pricing

Pay per event, no subscription, no idle cost to you:

| Event | When |
|---|---|
| Tool call | every invocation of any tool |
| URL returned | each URL in a `list_site_urls` answer |
| URL checked | each URL probed by `check_url` / `check_links` |
| Feed checked / item returned | `read_feed` |
| Citation checked | each URL fetched and verified by `verify_citations` |

A `check_url` costs a fraction of a cent; a 100-URL `check_links` a few cents; verifying the 10 sources of an answer about two cents. Set a maximum charge on the Standby Actor to cap spending.

### Limits (by design)

- `list_site_urls` ≤ 2,000 URLs, `check_links` ≤ 300 URLs, `read_feed` ≤ 100 items, `verify_citations` ≤ 50 citations, `site_overview` samples ≤ 5,000 URLs. Whole-site and whole-document jobs belong to the Actors: [Sitemap URL Extractor](https://apify.com/bruco3/sitemap-url-extractor), [Broken Link Checker](https://apify.com/bruco3/broken-link-checker), [Feed Monitor](https://apify.com/bruco3/feed-monitor), [Citation & Link Verifier](https://apify.com/bruco3/citation-verifier).
- Never more than 8 concurrent requests per host. Bodies are never downloaded for checks (HEAD, or GET closed after the headers); `verify_citations` reads cited pages up to 1.5 MB each.
- No JavaScript rendering and no crawling of pages: the tools read what sites publish for machines — sitemaps, feeds, robots.txt — and probe or read the URLs you name.

### Legal

The tools request only public files sites publish for automated readers (sitemaps, feeds, robots.txt, llms.txt) and probe public URLs with a clear `User-Agent`. No personal data is collected or stored.

### Feedback

A site the tools misread, or a tool you are missing? Open an issue on the **Issues** tab.

### Changelog

- **0.2** — `verify_citations`: live verification of cited sources (quote, title, date, author; soft-404s; archive copies).
- **0.1** — initial release: six tools, Streamable HTTP, Standby mode, pay-per-event.

# Actor input Schema

## Actor input object example

```json
{}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("bruco3/web-hygiene-mcp").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("bruco3/web-hygiene-mcp").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call bruco3/web-hygiene-mcp --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,bruco3/web-hygiene-mcp"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/pIG3zAx1WzjxT9fmM/builds/L7fQqV4Wwjiy1uSVH/openapi.json