📡 SignalRadar: Change & Entity Intelligence Platform

SignalRadar is an enterprise-grade Change Intelligence platform that tracks, parses, and highlights key updates (pricing grids, changelogs, career boards, news) across target company websites. It is built using a decoupled architecture combining a high-performance Apify Actor (scraping and diffing engine) and a stateless Model Context Protocol (MCP) Server (access and query gateway).

Designed for sales teams, investors, and AI agents, SignalRadar optimizes data extraction costs and LLM context windows using cache-first routing, semantic token-saving output options, and automatic change noise reduction.

---

🏗️ Platform Architecture

graph TD
    subgraph AI Client / IDE
        Cursor[Cursor / Windsurf / Custom Agent]
    end

    subgraph Apify Cloud Platform
        Actor[SignalRadar Actor / MCP Server]
        KV[Key-Value Snapshot Store]
        Dataset[Runs Dataset]
    end

    subgraph Competitor Web Targets
        Site1[Pricing Pages]
        Site2[Greenhouse / Job Boards]
        Site3[RSS Changelogs]
    end

    Cursor <-->|Stdio (CLI Bridge) / SSE| Actor
    Actor <-->|ETag / 304 Check| KV
    Actor -->|Append Briefs| Dataset
    Actor <-->|HTTP / Playwright| Site1
    Actor <-->|Direct API Fetch| Site2
    Actor <-->|RSS Parsing| Site3

---

🚀 Key Features

• HTTP-First Scraping Strategy: Utilizes lightweight Axios/Cheerio default adapters to bypass heavy browser processes, keeping container memory <100MB for 90% of requests. Fallbacks to headless Playwright browsers occur only when dynamic JS or captchas require it.
• Cache-First Latency Budget: Serves rapid historical queries under 200 milliseconds from pre-scraped Apify stores, preventing downstream AI client timeouts.
• Token-Aware Output Formatters: Supports three distinct output modes (brief, detailed, visual-link) allowing AI agents to request condensed updates and save up to 75% in context token costs.
• Semantic Schema.org JSON-LD: Outputs change records structured under standard JSON-LD schemas (@type: "Organization", @type: "UpdateAction", PriceSpecification) for plug-and-play AI integrations.
• Scroll-to-Text Evidence Links: Generates exact highlight URLs using Google's Scroll-to-Text Fragment parameters (e.g. #:~:text=) pointing directly to changes on the live web target.
• Public Dashboard & Status Badges: Generates visual EJS HTML timeline dashboards (dashboard.html) and live SVG health badges (e.g. stripe-pricing-green.svg) stored with public read access.
• Noise-Free Diff Engine: Strips boilerplate banners, GDPR overlays, headers, and analytics tracking scripts. Leverages Levenshtein distance calculations to mute trivial typo adjustments (<3% paragraph edits).
• Wayback Machine Fallback: Automatically requests archived snapshots from the Internet Archive Wayback Machine if live targets enforce aggressive Cloudflare blocks on a new target.

---

💎 Advanced Features & Capabilities

SignalRadar is designed to deliver maximum business value with minimum compute and proxy overhead. Here is a detailed look at the advanced capabilities built into the platform:

📈 1. Intelligent Pricing Intelligence

• Structured Pricing Table Parsing: Automatically extracts and maps raw pricing grids into normalized JSON key-value pairs (e.g. {"Free": 0, "Pro": 15}).
• Hydration Script Parsing: Directly extracts React hydration state JSON (from <script id="__NEXT_DATA__"> or Nuxt state window objects) of single-page apps, getting 100% accurate price points without running expensive browser engines.
• Price Change Scoring: Evaluates changes deterministically, flagging modified plans and generating relevance alerts.

💼 2. Career & Job Board Tracking

• Native API Interception: Detects if a company uses major job boards like Greenhouse or Lever. It bypasses the website altogether, fetching the job data directly from Greenhouse/Lever JSON REST endpoints.
• Frictionless Scrapes: Bypasses anti-bot challenges and browser overhead, providing lightweight, structured additions and removals of career posts.

🔄 3. Changelog & Product Feed Monitoring

• RSS/Sitemap Priority: Automatically checks for RSS feeds or XML sitemap updates before scraping html pages. This extracts changelog posts in milliseconds with zero false positives.
• Wayback Machine Fallback: If a target site implements severe Cloudflare CAPTCHAs and has no scraping history, the Actor queries the Internet Archive Wayback Machine to extract the latest snapshot.

📢 4. Noise-Filtering & Diff Muting

• Dynamic Tag Stripping: Automatically strips boilerplate HTML elements (cookie consent alerts, site navigation bars, footers, ad iframe nodes, tracking scripts) before computing diffs.
• Typo Muting: Calculates the Levenshtein distance of modifications. If the edit distance is less than 3% of the paragraph size, the update is muted to prevent alert spam from simple typo fixes.
• Redesign Consolidation: If more than 80% of a page's text changes, the Actor groups all updates into a single WEBSITE_REDESIGN_OR_REBRAND signal instead of triggering dozens of noise alerts.

🚀 5. Performance & Resource Optimization

• HTTP Conditional Requests: Stores ETag and Last-Modified headers from previous scrapes. If subsequent fetches return 304 Not Modified, it immediately skips parsing and diff calculations, saving up to 90% in proxy bandwidth.
• Request Ad-Blocking: Intercepts requests during browser fallbacks, blocking ads and tracking endpoints (Segment, Google Analytics, Sentry, Mixpanel) to shave off another 30% of bandwidth.
• OOM Preventative Memory Auto-Tuning: Detects container allocation RAM. If below 512MB, it automatically disables Playwright browser fallbacks and routes queries to Wayback Machine backups, preventing container crashes.

---

🤖 AI Client & Agent Integration

You can connect your local AI development environments (like Cursor or Windsurf), desktop clients (like OpenClaw), or autonomous agent frameworks (like Hermes) directly to the SignalRadar Actor running on Apify.

A. Connecting Cursor / Windsurf IDE

To connect Cursor or Windsurf directly to your cloud-hosted Apify Actor, configure them to use the Apify CLI as a local Stdio-to-Cloud proxy bridge.

1. Open Cursor/Windsurf and navigate to Settings > Models > MCP.
2. Click + Add New MCP Server.
3. Configure the following fields:
   • Name: SignalRadar
   • Type: command
   • Command: npx @apify/cli actor:run-mcp -a rexreus/SignalRadar -t <YOUR_APIFY_TOKEN>

---

B. Connecting Hermes & OpenClaw

For agent frameworks like Hermes or desktop interfaces like OpenClaw, you can integrate SignalRadar using either the Stdio bridge configuration or the SSE endpoint.

1. Stdio Configuration (mcp-servers.json / config.json)

Add the following to your agent framework's MCP config file:

{
  "mcpServers": {
    "signal-radar": {
      "command": "npx",
      "args": [
        "-y",
        "@apify/cli",
        "actor:run-mcp",
        "-a",
        "rexreus/SignalRadar",
        "-t",
        "<YOUR_APIFY_TOKEN>"
      ]
    }
  }
}

2. SSE Configuration (Remote Web Server Client)

If your framework supports remote SSE servers, start the Actor run on Apify and connect using the container SSE endpoint:

{
  "mcpServers": {
    "signal-radar": {
      "url": "https://<your-actor-run-id>.runs.apify.net/sse",
      "headers": {
        "Authorization": "Bearer <YOUR_APIFY_TOKEN>"
      }
    }
  }
}

---

📋 Concrete AI Agent Task Examples

Once SignalRadar is connected to your agent (Hermes, OpenClaw, or Cursor), you can assign it tasks using natural language. The agent will automatically select the appropriate tools to fulfill the request.

Task Example 1: Track Competitor Pricing Changes (Relevance Score > 7)

• User Prompt: "Check if Stripe has changed their pricing plans in the last 7 days. I only care about high-relevance modifications with a change score of 7 or higher. Summarize the changes briefly."
• Agent Execution Path:
  1. Calls get_recent_changes with arguments: {"domain": "stripe.com", "category": "pricing", "minScore": 7, "outputFormat": "brief"}.
  2. Synthesizes the response for the user: "Yes, Stripe modified their Pro plan from $10 to $15. You can view the highlight evidence here: https://stripe.com/pricing#:~:text=Pro%20plan%20is%20now%20%2415."

Task Example 2: Monitor Career Openings for AI Roles

• User Prompt: "Check if Anthropic has posted any new Senior Machine Learning Engineer jobs on their careers board recently."
• Agent Execution Path:
  1. Calls get_recent_changes with arguments: {"domain": "anthropic.com", "category": "jobs", "outputFormat": "brief"}.
  2. Compares the results and returns details of newly posted roles.

Task Example 3: Full Company Intelligence Audit & Score

• User Prompt: "Run a full business change intelligence audit for openai.com. I want to see their calculated opportunity and risk scores, along with explanations for the scores."
• Agent Execution Path:
  1. Calls run_brief with arguments: {"domain": "openai.com"} (starts background scrape on Apify).
  2. Calls wait_for_brief to await completion.
  3. Calls get_signal_score with arguments: {"domain": "openai.com"}.
  4. Returns the risk breakdown, opportunity ranking, and visual dashboard link (visualBriefUrl).

💡 Saving Token Budget (outputFormat)

When communicating with AI agents, choose the appropriate outputFormat argument to control cost vs. detail:

1. brief (Default): Returns highly condensed, semantic sentence outlines of updates. Excellent for keeping LLM prompts lightweight.

   {
     "changes": [
       "[Pricing] Changed Pro Plan from $10 to $15 (Score: 8/10) - Evidence: https://stripe.com/pricing#:~:text=Pro%20plan%20is%20now%20%2415"
     ]
   }
   

2. visual-link: Returns only the dashboard URL link. Reduces context size to near-zero.

   {
     "visualBriefUrl": "https://api.apify.com/v2/key-value-stores/store-id/records/dashboard-stripe.html",
     "message": "Detailed changes can be viewed visually on the SignalRadar public web dashboard."
   }
   

3. detailed: Returns the complete Schema.org JSON-LD object containing detailed diff text maps. Use this only when you need programmatic access to raw text segments.

---

🛠️ MCP Tools Reference

SignalRadar exposes the following tools to connected clients:

Tool Name	Parameters	Description
list_sources	None	Returns the list of active source adapters (e.g. website, pricing, jobs, changelog, news).
create_watchlist	domains (array of strings, required) adapters (array of strings, optional) webhookUrl (string, optional)	Registers a new company or set of domains to monitor. Normalizes URLs to apex domains.
run_brief	domain (string, required) maxAgeSeconds (integer, optional)	Triggers a background scraping execution on the Apify platform for the entity. Returns a runId.
get_recent_changes	domain (string, required) category (string, optional) minScore (integer, optional) outputFormat (string, optional)	Retrieves detected changes from cache. Returns instantly (<200ms).
get_evidence_pack	changeId (string, required)	Returns source HTML quotes, text diff blocks, and Scroll-to-Text fragment links for a specific change.
get_signal_score	domain (string, required)	Returns computed risk and opportunity scores alongside qualitative business explanations.
wait_for_brief	runId (string, required)	Blocks and polls the Apify Actor status every 2 seconds for up to 15 seconds to return the completed brief.
search_watchlists	query (string, required)	Performs a text search on registered watchlists, saving token budget.
diagnostic_health_check	None	Evaluates Apify credentials validity, checks remaining platform credits, and tests proxy connectivity.

---

⚙️ Technical Highlights

⚡ Conditional Fetching & ETag Support

During Axios page fetching, the scraper persists and sends If-None-Match (ETag) and If-Modified-Since (Last-Modified) headers. If target servers return a 304 Not Modified status, SignalRadar immediately skips the diffing, parsing, and data-writing cycles, saving up to 90% in proxy bandwidth costs.

🛡️ CAPTCHA & Security Mitigation

• Spoofs TLS fingerprints (ALPN settings, ciphers matching Chrome/Safari) inside raw HTTP requests.
• PROXY-resolved DNS: Performs hostname lookups on the proxy server via CONNECT tunnels, escaping data center DNS blacklists.
• Intercepts and blocks analytics trackers, double-click ads, and video resources during browser fallbacks to reduce proxy bandwidth by another ~30%.

---

📄 License

This project is licensed under the MIT License. See the LICENSE file for details.