CVE Scraper

CVE Scraper

Apify Actor that scrapes vulnerability records from cve.org for a given search query. Built on Scrapy, talks directly to CVE.org's search API (no HTML parsing), and paginates through all matching results (up to 10k per query, the API's own ceiling).

How it works

The Actor takes a cve.org search-results URL, pulls the query parameter out of it, and replays that query against https://www.cve.org/restapiv1/search in pages of 100, following pagination until resultsTotal is exhausted. Each result is flattened into one output row — see field list below.

Input

{
    "startUrls": [
        {
            "url": "https://www.cve.org/CVERecord/SearchResults?query=scrapy",
            "method": "GET"
        }
    ]
}

startUrls — one or more cve.org search-results URLs (as produced by the site's own search box). Only the query query-string parameter is used; the rest of the URL is ignored. A startUrls entry without a query parameter is skipped.

Output row example

{
  "cve_id": "CVE-2024-3574",
  "url": "https://www.cve.org/CVERecord?id=CVE-2024-3574",
  "date_published": "2024-04-16T00:00:15.109Z",
  "date_reserved": "2024-04-10T09:54:50.274Z",
  "date_updated": "2024-08-01T20:12:08.239Z",
  "cur_state": "PUBLISHED",
  "title": "Authorization Header Leak During Cross-Domain Redirect in scrapy/scrapy",
  "description": "In scrapy version 2.10.1, an issue was identified where the Authorization header, containing credentials for server authentication, is leaked to a third-party site during a cross-domain redirect. This vulnerability arises from the failure to remove the Authorization header when redirecting across domains. The exposure of the Authorization header to unauthorized actors could potentially allow for account hijacking.",
  "vendor": "scrapy",
  "product": "scrapy/scrapy",
  "versions_affected": [
    "unspecified"
  ],
  "attack_complexity": null,
  "attack_vector": null,
  "availability_impact": null,
  "cvss_base_score": null,
  "base_severity": null,
  "confidentiality_impact": null,
  "integrity_impact": null,
  "privs_required": null,
  "user_interaction": null,
  "vector_string": null,
  "advisory": "49974321-2718-43e3-a152-62b16eed72a9",
  "ref_urls": [
    "https://huntr.com/bounties/49974321-2718-43e3-a152-62b16eed72a9",
    "https://github.com/scrapy/scrapy/commit/5bcb8fd5019c72d05c4a96da78a7fcb6ecb55b75"
  ],
  "ref_ids": []
}

Field	Description
cve_id	CVE identifier, e.g. CVE-2024-3574
url	Link to the record on cve.org
date_published / date_reserved / date_updated	Lifecycle timestamps
cur_state	PUBLISHED, RESERVED, etc.
title	Vulnerability title (may be null for older/reserved records)
description	English-language description, if present
vendor / product	Affected vendor and product (last affected entry wins if there are several)
versions_affected	Affected version strings
attack_complexity, attack_vector, availability_impact, cvss_base_score, base_severity, confidentiality_impact, integrity_impact, privs_required, user_interaction, vector_string	CVSS v4.0 metrics, only populated when the record ships a cvssV4_0 metric
advisory	Advisory ID from the CNA source
ref_urls / ref_ids	Reference links and their short IDs

Fields not present on a given record come back as null or [] — CVE.org's schema varies a lot by CNA and by record age, this is expected, not a bug.

Stability notes

• Retries up to 5 times on 429/5xx/timeouts, with AutoThrottle backing off under rate limiting.
• Malformed or partial entries (missing _source, bad resultsTotal, non-JSON response) are logged and skipped rather than crashing the run.
• Default run memory is 1024 MB (typical peak usage is well under that); timeout 1 hour.

Local development

pip install -r requirements.txt
apify run --purge

Requires an apify CLI login and Docker for apify push / image builds. Source lives under src/, spider logic in src/spiders/cve.py.

Cyber recon toolkit

Part of the Cyber recon toolkit — OSINT, attack-surface recon, and vulnerability-intel data collection for security research:

• Bugcrowd Scraper — Scrapes public Bugcrowd data: disclosed submissions, program directory, leaderboard.
• HackerOne Scraper — Scrapes public HackerOne data: Hacktivity disclosed reports, program directory, researcher leaderboard.
• Intigriti Scraper — Scrapes public Intigriti data: bug bounty/VDP program directory, researcher leaderboard and profiles.
• TLSX - TLS Data Gathering and Analysis — Fast, configurable TLS grabber for TLS-based data collection and analysis.
• BGP.HE.NET Recon Scraper — Scrapes ASN, prefix, IP, and DNS recon data from bgp.he.net.
• gau - Get All URLs — Fetch known URLs from Wayback Machine, Common Crawl, AlienVault OTX, and URLScan.
• Git Email Extractor — Shallow-clones git repos and extracts contributor emails from commit history.
• Wappalyzer Technology Lookup — Detect the technology stack behind any website using Wappalyzer.
• Meta Tags Scraper — Web page metadata scraper (meta tags, Open Graph, Twitter Card).
• Universal Contact Extractor — Extract phone numbers, emails and social media URLs from web pages.
• Web Text Extractor — Extract clean text or Markdown content from web pages.

Did you find this useful?

⭐ Rate this actor on Apify! Your feedback helps other users find it and helps us keep improving it.