Pricing

from $3.00 / 1,000 snapshot markdowns

Try for free

Go to Apify Store

Wayback CDX → Markdown for RAG

Try for free

Query the Wayback Machine CDX index, fetch selected snapshot HTML, and extract clean Markdown with trafilatura for RAG pipelines. Optional heading-aware chunks. Competitors often stop at CDX URL lists — this Actor returns page body Markdown.

Pricing

from $3.00 / 1,000 snapshot markdowns

Rating

0.0

(0)

Developer

新世紀書僮

Actor stats

Bookmarked

Total users

Monthly active users

15 hours ago

Last modified

Categories

Developer tools

Automation

You can access the Wayback CDX → Markdown for RAG programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

{
  "openapi": "3.0.1",
  "info": {
    "version": "0.1",
    "x-build-id": "VabxHpDc7MD4jGrVt"
  },
  "servers": [
    {
      "url": "https://api.apify.com/v2"
    }
  ],
  "paths": {
    "/acts/ingenious_quip_bxq~wayback-cdx-markdown/run-sync-get-dataset-items": {
      "post": {
        "operationId": "run-sync-get-dataset-items-ingenious_quip_bxq-wayback-cdx-markdown",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK"
          }
        }
      }
    },
    "/acts/ingenious_quip_bxq~wayback-cdx-markdown/runs": {
      "post": {
        "operationId": "runs-sync-ingenious_quip_bxq-wayback-cdx-markdown",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor and returns information about the initiated run in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK",
            "content": {
              "application/json": {
                "schema": {
                  "$ref": "#/components/schemas/runsResponseSchema"
                }
              }
            }
          }
        }
      }
    },
    "/acts/ingenious_quip_bxq~wayback-cdx-markdown/run-sync": {
      "post": {
        "operationId": "run-sync-ingenious_quip_bxq-wayback-cdx-markdown",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK"
          }
        }
      }
    }
  },
  "components": {
    "schemas": {
      "inputSchema": {
        "type": "object",
        "properties": {
          "urls": {
            "title": "Page URLs",
            "type": "array",
            "description": "Exact page URLs to look up in the Wayback CDX index (e.g. https://example.com/about). Paste a list or upload a text file.",
            "items": {
              "type": "object",
              "required": [
                "url"
              ],
              "properties": {
                "url": {
                  "type": "string",
                  "title": "URL of a web page",
                  "format": "uri"
                }
              }
            }
          },
          "domains": {
            "title": "Domains (prefix / wildcard)",
            "type": "array",
            "description": "Optional. Bare domains or hostnames (e.g. example.com). Queried as `example.com/*` so CDX returns archived paths under that host. Use Max snapshots carefully — large sites have huge CDX result sets.",
            "items": {
              "type": "string"
            }
          },
          "fromDate": {
            "title": "From date (Wayback)",
            "type": "string",
            "description": "Earliest snapshot timestamp prefix (YYYY, YYYYMM, YYYYMMDD, or full YYYYMMDDhhmmss). Passed to the CDX `from` parameter."
          },
          "toDate": {
            "title": "To date (Wayback)",
            "type": "string",
            "description": "Latest snapshot timestamp prefix (same formats as From date). Passed to the CDX `to` parameter."
          },
          "collapse": {
            "title": "Collapse",
            "enum": [
              "timestamp:8",
              "timestamp:6",
              "digest",
              "none"
            ],
            "type": "string",
            "description": "CDX `collapse` to reduce near-duplicates. `timestamp:8` ≈ one per day, `timestamp:6` ≈ one per month, `digest` = unique content digests, `none` = no collapse (can be huge).",
            "default": "timestamp:8"
          },
          "uniquesOnly": {
            "title": "Uniques only (dupecount=0)",
            "type": "boolean",
            "description": "When the CDX API returns dupecount, keep only rows with dupecount=0 (waybackpack `--uniques-only`).",
            "default": false
          },
          "statusCodes": {
            "title": "HTTP status codes",
            "type": "array",
            "description": "Keep only CDX rows whose statuscode is in this list. Empty = keep all codes.",
            "default": [
              "200"
            ],
            "items": {
              "type": "string"
            }
          },
          "mimeTypes": {
            "title": "MIME type substrings",
            "type": "array",
            "description": "Keep CDX rows whose mimetype contains any of these substrings (case-insensitive). Empty = no MIME filter. Default keeps HTML-ish captures.",
            "default": [
              "text/html",
              "html"
            ],
            "items": {
              "type": "string"
            }
          },
          "selection": {
            "title": "Snapshot selection",
            "enum": [
              "latest",
              "oldest",
              "sample"
            ],
            "type": "string",
            "description": "After filters, which snapshots to fetch: `latest` (newest first), `oldest` (oldest first), or `sample` (evenly spaced across the filtered list).",
            "default": "latest"
          },
          "maxSnapshotsPerTarget": {
            "title": "Max snapshots per URL/domain",
            "minimum": 1,
            "maximum": 500,
            "type": "integer",
            "description": "Cap how many snapshots are fetched for each input URL or domain after filters/selection.",
            "default": 3
          },
          "maxItems": {
            "title": "Max Markdown items",
            "minimum": 0,
            "maximum": 100000,
            "type": "integer",
            "description": "Stop after this many successfully extracted snapshot rows (0 = no limit). Also caps cost.",
            "default": 50
          },
          "fetchRaw": {
            "title": "Fetch raw archived bytes (id_)",
            "type": "boolean",
            "description": "Use Wayback `id_` flag so the Machine returns the original capture without toolbar rewriting (waybackpack `--raw`). Recommended for extraction.",
            "default": true
          },
          "outputFormat": {
            "title": "Output",
            "enum": [
              "items",
              "chunks",
              "items_and_chunks"
            ],
            "type": "string",
            "description": "`items`: one dataset row per extracted snapshot. `chunks`: RAG chunk rows + a short per-snapshot summary row. `items_and_chunks`: both full items and chunk rows.",
            "default": "items"
          },
          "includeComments": {
            "title": "Include comment sections",
            "type": "boolean",
            "description": "Pass include_comments to trafilatura (default off — usually noise on archived pages).",
            "default": false
          },
          "includeTables": {
            "title": "Include tables",
            "type": "boolean",
            "description": "Pass include_tables to trafilatura.",
            "default": true
          },
          "favorPrecision": {
            "title": "Favor precision (less boilerplate)",
            "type": "boolean",
            "description": "trafilatura favor_precision=True — less noise, possibly less text.",
            "default": false
          },
          "chunkSize": {
            "title": "Chunk size",
            "minimum": 100,
            "maximum": 20000,
            "type": "integer",
            "description": "Maximum chunk size (unit below). Used when Output includes chunks.",
            "default": 1000
          },
          "chunkOverlap": {
            "title": "Chunk overlap",
            "minimum": 0,
            "maximum": 10000,
            "type": "integer",
            "description": "Trailing context repeated at the start of the next chunk.",
            "default": 150
          },
          "chunkUnit": {
            "title": "Chunk size unit",
            "enum": [
              "tokens",
              "characters"
            ],
            "type": "string",
            "description": "`tokens` ≈ 4 characters per token; `characters` is exact.",
            "default": "tokens"
          },
          "requestTimeoutSecs": {
            "title": "Request timeout (seconds)",
            "minimum": 10,
            "maximum": 300,
            "type": "integer",
            "description": "Timeout for CDX and snapshot HTTP requests.",
            "default": 60
          },
          "maxRetries": {
            "title": "Max retries",
            "minimum": 0,
            "maximum": 8,
            "type": "integer",
            "description": "Retries for Wayback CDX / snapshot fetches (waybackpack Session max_retries).",
            "default": 3
          },
          "delaySecs": {
            "title": "Delay between snapshot fetches (seconds)",
            "minimum": 0,
            "maximum": 30,
            "type": "number",
            "description": "Polite pause between snapshot body downloads. CDX queries are not delayed.",
            "default": 0.5
          },
          "userAgent": {
            "title": "User-Agent",
            "type": "string",
            "description": "Optional. Wayback asks tools to identify themselves (include a contact). Default mentions waybackpack + this Actor."
          }
        }
      },
      "runsResponseSchema": {
        "type": "object",
        "properties": {
          "data": {
            "type": "object",
            "properties": {
              "id": {
                "type": "string"
              },
              "actId": {
                "type": "string"
              },
              "userId": {
                "type": "string"
              },
              "startedAt": {
                "type": "string",
                "format": "date-time",
                "example": "2025-01-08T00:00:00.000Z"
              },
              "finishedAt": {
                "type": "string",
                "format": "date-time",
                "example": "2025-01-08T00:00:00.000Z"
              },
              "status": {
                "type": "string",
                "example": "READY"
              },
              "meta": {
                "type": "object",
                "properties": {
                  "origin": {
                    "type": "string",
                    "example": "API"
                  },
                  "userAgent": {
                    "type": "string"
                  }
                }
              },
              "stats": {
                "type": "object",
                "properties": {
                  "inputBodyLen": {
                    "type": "integer",
                    "example": 2000
                  },
                  "rebootCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "restartCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "resurrectCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "computeUnits": {
                    "type": "integer",
                    "example": 0
                  }
                }
              },
              "options": {
                "type": "object",
                "properties": {
                  "build": {
                    "type": "string",
                    "example": "latest"
                  },
                  "timeoutSecs": {
                    "type": "integer",
                    "example": 300
                  },
                  "memoryMbytes": {
                    "type": "integer",
                    "example": 1024
                  },
                  "diskMbytes": {
                    "type": "integer",
                    "example": 2048
                  }
                }
              },
              "buildId": {
                "type": "string"
              },
              "defaultKeyValueStoreId": {
                "type": "string"
              },
              "defaultDatasetId": {
                "type": "string"
              },
              "defaultRequestQueueId": {
                "type": "string"
              },
              "buildNumber": {
                "type": "string",
                "example": "1.0.0"
              },
              "containerUrl": {
                "type": "string"
              },
              "usage": {
                "type": "object",
                "properties": {
                  "ACTOR_COMPUTE_UNITS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_WRITES": {
                    "type": "integer",
                    "example": 1
                  },
                  "KEY_VALUE_STORE_LISTS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_INTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_EXTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_SERPS": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_UNBLOCKER_UNITS": {
                    "type": "integer",
                    "example": 0
                  }
                }
              },
              "usageTotalUsd": {
                "type": "number",
                "example": 0.00005
              },
              "usageUsd": {
                "type": "object",
                "properties": {
                  "ACTOR_COMPUTE_UNITS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_WRITES": {
                    "type": "number",
                    "example": 0.00005
                  },
                  "KEY_VALUE_STORE_LISTS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_INTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_EXTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_SERPS": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_UNBLOCKER_UNITS": {
                    "type": "integer",
                    "example": 0
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Wayback CDX to Markdown for RAG OpenAPI definition

OpenAPI is a standard for designing and describing RESTful APIs, allowing developers to define API structure, endpoints, and data formats in a machine-readable way. It simplifies API development, integration, and documentation.

OpenAPI is effective when used with AI agents and GPTs by standardizing how these systems interact with various APIs, for reliable integrations and efficient communication.

By defining machine-readable API specifications, OpenAPI allows AI models like GPTs to understand and use varied data sources, improving accuracy. This accelerates development, reduces errors, and provides context-aware responses, making OpenAPI a core component for AI applications.

You can download the OpenAPI definitions for Wayback CDX → Markdown for RAG from the options below:

OpenAPI.json

If you’d like to learn more about how OpenAPI powers GPTs, read our blog post.

You can also check out our other API clients:

Wayback CDX → Markdown for RAG API in Python

Wayback CDX → Markdown for RAG API in JavaScript

Wayback CDX → Markdown for RAG API through CLI

Wayback CDX → Markdown for RAG API

Instagram Scraper

apify/instagram-scraper

Extract Instagram posts, reels, profiles, places, hashtags, carousels, and comments. Get data from Instagram using one or more Instagram URLs or search queries: content, context, metrics, metadata. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Apify

413K

4.7

(601)

Google Maps Scraper

compass/crawler-google-places

Extract data from thousands of Google Maps locations and businesses, including reviews, reviewer details, images, contact info, including full name, email, and job title, opening hours, prices & more. Export data, run via API, schedule and monitor runs, or integrate with other tools.

Compass

629K

4.7

(1,817)

Instagram Profile Scraper

apify/instagram-profile-scraper

Scrape all Instagram profile info. Just add Instagram usernames, IDs or URLs and extract name, join date, number of followers, location, bio, website, related profiles, video&post count, latest posts. Export scraped data, schedule scraper via API, and integrate with other tools or AI workflows.

Apify

233K

4.7

(175)

TikTok Scraper

clockworks/tiktok-scraper

Extract data from TikTok videos, hashtags, and users. Use URLs or search queries to scrape TikTok profiles, hashtags, posts, URLs, shares, followers, hearts, names, video, and music-related data. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Clockworks

298K

4.8

(370)

Facebook Posts Scraper

apify/facebook-posts-scraper

Extract posts, videos, and engagement metrics from Facebook pages. Get text captions, reactions, video transcripts, images, external links, collaborators, and more from Facebook pages and profiles. Export ad data, schedule runs via API, and integrate with other tools or AI workflows.

Apify

117K

4.6

(234)

Linkedin Company Details Scraper (No Cookies) ✅ Bulk

harvestapi/linkedin-company

Extract detailed information from LinkedIn Companies such as company name, address, phone numbers, website, employee count, and more. Find LinkedIn company URLs by name in Bulk. No cookies or account required.

HarvestAPI

21K

4.5

(35)

LinkedIn Profile Scraper + Email ✅ No Cookies

harvestapi/linkedin-profile-scraper

Extract detailed information from LinkedIn profiles in bulk, including complete work experience, education history, skills and more. No cookies or account required.

HarvestAPI

70K

4.6

(92)

Google Search Results Scraper

apify/google-search-scraper

Scrape Google Search Engine Results Pages (SERPs). Select the country or language and extract organic and paid results, AI Mode, AI overviews, ads, queries, People Also Ask, prices, reviews, like a Google SERP API. Export data, run the scraper via API, schedule runs, or integrate with other tools.

Apify

195K

4.6

(218)

Google Maps Reviews Scraper

compass/Google-Maps-Reviews-Scraper

Extract all reviews of Google Maps places using place URLs. Get review text, published date, response from owner, review URL, and reviewer's details. Download scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Compass

59K

4.9

(197)

Instagram Post Scraper

apify/instagram-post-scraper

Scrape Instagram posts. Just add one or more Instagram usernames and get your data in seconds including caption, metrics, images, mentions, coauthors, recent comments, sponsored status, video duration, views. Export scraped data, schedule scraper via API, integrate with other tools or AI workflows.

Apify

134K

4.3

(139)