CVE Scraper

CVE Scraper

Apify Actor that scrapes vulnerability records from cve.org for a given search query. Built on Scrapy, talks directly to CVE.org's search API (no HTML parsing), and paginates through all matching results (up to 10k per query, the API's own ceiling).

How it works

The Actor takes a cve.org search-results URL, pulls the query parameter out of it, and replays that query against https://www.cve.org/restapiv1/search in pages of 100, following pagination until resultsTotal is exhausted. Each result is flattened into one output row — see field list below.

Input

{
    "startUrls": [
        {
            "url": "https://www.cve.org/CVERecord/SearchResults?query=scrapy",
            "method": "GET"
        }
    ]
}

startUrls — one or more cve.org search-results URLs (as produced by the site's own search box). Only the query query-string parameter is used; the rest of the URL is ignored. A startUrls entry without a query parameter is skipped.

Output row example

{
  "cve_id": "CVE-2024-3574",
  "url": "https://www.cve.org/CVERecord?id=CVE-2024-3574",
  "date_published": "2024-04-16T00:00:15.109Z",
  "date_reserved": "2024-04-10T09:54:50.274Z",
  "date_updated": "2024-08-01T20:12:08.239Z",
  "cur_state": "PUBLISHED",
  "title": "Authorization Header Leak During Cross-Domain Redirect in scrapy/scrapy",
  "description": "In scrapy version 2.10.1, an issue was identified where the Authorization header, containing credentials for server authentication, is leaked to a third-party site during a cross-domain redirect. This vulnerability arises from the failure to remove the Authorization header when redirecting across domains. The exposure of the Authorization header to unauthorized actors could potentially allow for account hijacking.",
  "vendor": "scrapy",
  "product": "scrapy/scrapy",
  "versions_affected": [
    "unspecified"
  ],
  "attack_complexity": null,
  "attack_vector": null,
  "availability_impact": null,
  "cvss_base_score": null,
  "base_severity": null,
  "confidentiality_impact": null,
  "integrity_impact": null,
  "privs_required": null,
  "user_interaction": null,
  "vector_string": null,
  "advisory": "49974321-2718-43e3-a152-62b16eed72a9",
  "ref_urls": [
    "https://huntr.com/bounties/49974321-2718-43e3-a152-62b16eed72a9",
    "https://github.com/scrapy/scrapy/commit/5bcb8fd5019c72d05c4a96da78a7fcb6ecb55b75"
  ],
  "ref_ids": []
}

Field	Description
cve_id	CVE identifier, e.g. CVE-2024-3574
url	Link to the record on cve.org
date_published / date_reserved / date_updated	Lifecycle timestamps
cur_state	PUBLISHED, RESERVED, etc.
title	Vulnerability title (may be null for older/reserved records)
description	English-language description, if present
vendor / product	Affected vendor and product (last affected entry wins if there are several)
versions_affected	Affected version strings
attack_complexity, attack_vector, availability_impact, cvss_base_score, base_severity, confidentiality_impact, integrity_impact, privs_required, user_interaction, vector_string	CVSS v4.0 metrics, only populated when the record ships a cvssV4_0 metric
advisory	Advisory ID from the CNA source
ref_urls / ref_ids	Reference links and their short IDs

Fields not present on a given record come back as null or [] — CVE.org's schema varies a lot by CNA and by record age, this is expected, not a bug.

Stability notes

• Retries up to 5 times on 429/5xx/timeouts, with AutoThrottle backing off under rate limiting.
• Malformed or partial entries (missing _source, bad resultsTotal, non-JSON response) are logged and skipped rather than crashing the run.
• Default run memory is 1024 MB (typical peak usage is well under that); timeout 1 hour.

Local development

pip install -r requirements.txt
apify run --purge

Requires an apify CLI login and Docker for apify push / image builds. Source lives under src/, spider logic in src/spiders/cve.py.