# ETF Standards & Spec Updates Monitor (`datapilot/etf-standards-spec-updates-monitor`) Actor

Extracts standard ID, title, revision, document type, official document link, and scrape timestamp. Supports multiple result limits, pagination, Residential Proxy, automatic retries, duplicate filtering, and structured Apify Dataset output. Useful for monitoring new RFCs, Internet-Drafts,

- **URL**: https://apify.com/datapilot/etf-standards-spec-updates-monitor.md
- **Developed by:** [Data Pilot](https://apify.com/datapilot) (community)
- **Categories:** Other, Integrations
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 result scrapes

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

📡 **IETF Standards** & Spec Updates Monitor is a powerful Apify Actor designed to discover, track, and collect Internet Engineering Task Force document data directly from the official IETF Datatracker API. This tool provides comprehensive **IETF Standards** intelligence including standard/document ID, title, revision number, document type, and a direct link to the official document page for any keyword. Whether you're building a protocol research dashboard, tracking RFC and draft activity, or conducting standards compliance research, the IETF Standards & Spec Updates Monitor delivers actionable **IETF Standards** insights efficiently.

With direct Datatracker API access, keyword and document-type filtering, automatic pagination with global deduplication, and reliable Apify Dataset delivery, the IETF Standards & Spec Updates Monitor ensures comprehensive **IETF Standards** coverage across RFCs and Internet-Drafts alike. It focuses on key **IETF Standards** signals including document type, revision, and title, making it an essential tool for protocol research and standards tracking.

***

### 📋 Table of Contents

- [Features](#-features)
- [Data Source](#-data-source)
- [How It Works](#-how-it-works)
- [Input](#-input)
- [Output](#-output)
- [Technical Stack](#-technical-stack)
- [Data Fields](#-data-fields-explained)
- [Use Cases](#-use-cases)
- [Quick Start](#-quick-start)
- [Configuration](#-configuration)
- [Performance](#-performance)
- [Important Notes](#-important-notes)
- [License & Legal](#-license--legal)

***

### 🔥 Features

- **Direct API Access** – Queries the official IETF Datatracker API (`datatracker.ietf.org/api/v1`) directly for fast, structured **IETF Standards** data.
- **Keyword Filtering** – Matches document names against a given keyword using the API's `name__icontains` filter.
- **Document Type Filtering** – Narrow results to only RFCs, only Internet-Drafts, or both document types.
- **Latest-Documents Mode** – Supports an empty keyword to pull the most recent standards documents.
- **Automatic Pagination** – Pages through all available results using `limit`/`offset` until the configured maximum is reached.
- **Global Deduplication** – Tracks seen document names to prevent duplicate entries in the output.
- **Retry-Resilient Requests** – Automatically retries on rate limits (429) and auth errors (401/403) with backoff delays.
- **Residential Proxy Support** – Apify residential proxy for reliable **IETF Standards** API access.
- **Configurable Result Cap** – Control how many documents are pulled per run.
- **Incremental Dataset Push** – Pushes each batch of results to the Dataset as it's collected, not just at the end.

***

### 📊 Data Source

#### **IETF Datatracker Public API**

- **Authority**: The Internet Engineering Task Force's official document tracking system (`datatracker.ietf.org`)
- **Access Method**: Browser-impersonated HTTP requests via `curl_cffi`, run through an executor for async compatibility
- **Coverage**: Any publicly listed **IETF Standards** document — RFCs and Internet-Drafts — matching the given keyword and type filter
- **Data**: Structured JSON document records including name, title, revision, and type
- **Access**: Public API, no API key required
- **Update Frequency**: Reflects the IETF Datatracker's current published document set at time of run

***

### ⚙️ How It Works

The IETF Standards & Spec Updates Monitor accepts an optional keyword and document type filter, then queries the IETF Datatracker's document API using `limit`/`offset` pagination. When a keyword is provided, it's matched against document names via a case-insensitive contains filter; when a document type is specified, the query is restricted to RFCs (`name__istartswith=rfc`), drafts (`name__istartswith=draft`), or both types together. Each returned document is normalized into a clean record containing the standard ID, title, revision, document type, and a direct link to its official Datatracker page. A global set of seen document names prevents duplicates across pages. Results are pushed to the Apify Dataset incrementally, batch by batch, as they're collected.

**Key Processing Steps:**

1. **Input Parsing** – Accept a keyword, document type filter, and max result count
2. **Proxy Setup** – Configure Apify residential proxy for the impersonated HTTP session
3. **Filter Construction** – Build the API query with keyword and document-type parameters
4. **API Request** – GET the Datatracker document endpoint with `limit`, `offset`, and filter parameters
5. **Retry Handling** – Back off and retry on rate limits (429) or auth errors (401/403)
6. **Deduplication Check** – Skip any document name already seen in a previous page
7. **Document Normalization** – Extract standard ID, title, revision, and type, and build the document link
8. **Pagination** – Continue requesting subsequent pages using offset increments until results or the max cap are exhausted
9. **Incremental Push** – Push each batch of normalized results to the Dataset as it's ready
10. **Rate Limiting** – Apply a short delay between paginated requests

**Key Benefits:**

- Pull **IETF Standards** data for any keyword without manually searching the Datatracker
- Track newly published or revised RFCs and Internet-Drafts in a given protocol area
- Filter precisely by document type to focus on finalized RFCs or in-progress drafts
- Feed protocol research pipelines, standards compliance tools, or engineering reference dashboards
- Automate recurring **IETF Standards** checks as new documents and revisions are published

***

### 📥 Input

The Actor accepts the following input parameters:

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `keyword` / `keywords` | string | *(none — latest documents)* | Search term matched against document names. |
| `docType` | string | `"ALL"` | Document type filter: `"rfc"`, `"draft"`, or `"ALL"` for both. |
| `maxResults` | integer | `50` | Maximum number of **IETF Standards** documents to collect. |
| `proxyConfiguration` | object | `{"useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"]}` | Apify proxy configuration for the search session. |

**Example Input:**

```json
{
  "keyword": "quic",
  "docType": "rfc",
  "maxResults": 50,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  }
}
```

**Fetch the Latest Documents (No Keyword):**

```json
{
  "maxResults": 100
}
```

**Search Drafts Only:**

```json
{
  "keyword": "tls",
  "docType": "draft"
}
```

***

### 📤 Output

The Actor pushes **IETF Standards** records with the following structure:

| Field | Type | Description |
|-------|------|-------------|
| `search_keyword` | string | The keyword that matched this result, or `"LATEST_SPECS"` if none was provided |
| `standard_id` | string | The document's unique name/identifier (e.g., `"rfc9114"`, `"draft-ietf-quic-http"`) |
| `title` | string | The document's official title |
| `revision` | string | The document's current revision number |
| `doc_type` | string | The document type classification from the Datatracker |
| `link` | string | Direct URL to the document's official Datatracker page |
| `scraped_at` | string | ISO 8601 timestamp of when the record was scraped |

**Example IETF Standards Record:**

```json
{
  "search_keyword": "quic",
  "standard_id": "rfc9114",
  "title": "HTTP/3",
  "revision": "",
  "doc_type": "rfc",
  "link": "https://datatracker.ietf.org/doc/rfc9114/",
  "scraped_at": "2026-08-21T09:00:00+00:00"
}
```

***

### 🧰 Technical Stack

- **HTTP Client:** `curl_cffi` with Chrome impersonation for reliable access to the IETF Datatracker API
- **Concurrency:** `asyncio` with an executor bridge for running the impersonated client asynchronously
- **Query Syntax:** Datatracker API filter parameters (`name__icontains`, `name__istartswith`, `type__slug__in`) for keyword and type filtering
- **Deduplication:** In-memory `set` tracking of seen document names across the full run
- **Date Handling:** `datetime` with timezone-aware UTC timestamps
- **Proxy:** Apify Proxy with `RESIDENTIAL` configuration by default
- **Logging:** Apify Actor logging system
- **Platform:** Apify Actor serverless environment

***

### 📊 Data Fields Explained

#### **Document Identity**

- **standard\_id**: The unique document name assigned by the IETF Datatracker
- **title**: The official title of the standard or draft

#### **Document Details**

- **revision**: The current revision number of the document
- **doc\_type**: Whether the document is an RFC, Internet-Draft, or other tracked type
- **search\_keyword**: The keyword that surfaced this result, for traceability

#### **Access**

- **link**: Direct link to view the full **IETF Standards** document on the official Datatracker

***

### 🎯 Use Cases

- **Protocol Research** – Track new and revised RFCs or drafts in a specific technology area
- **Standards Compliance** – Monitor which specifications are relevant to a product or implementation
- **Engineering Reference** – Build a searchable internal index of relevant IETF documents
- **Competitive & Industry Tracking** – Watch standards activity in emerging areas like QUIC, TLS, or DNS
- **Academic Research** – Study standards evolution and revision patterns at scale
- **Technical Writing** – Source authoritative links to cite in documentation or specifications

***

### 🚀 Quick Start

#### **1. Prepare Input**

Go to Apify Console and enter:

```json
{
  "keyword": "quic"
}
```

#### **2. Run the Actor**

Click **Start**. The Actor will:

- Query the IETF Datatracker API with the given keyword and type filter
- Page through all matching results, retrying on rate limits automatically
- Deduplicate documents across pages
- Push batches to the Dataset incrementally as they're collected

#### **3. Monitor Progress**

Console shows:

```
Searching IETF Datatracker for: 'quic' (Type: ALL)
Saved 50 specs (Progress: 50/50)
🎉 Scraping complete! Successfully extracted 50 IETF standards documents.
```

#### **4. View & Download Results**

- **Results Tab**: All **IETF Standards** records
- **Export**: JSON, CSV, Excel, or HTML
- **Filter**: By document type or revision
- **API Access**: Available via the Apify API

***

### ⚙️ Configuration

Keyword search, all document types:

```json
{
  "keyword": "tls"
}
```

RFCs only:

```json
{
  "keyword": "http",
  "docType": "rfc",
  "maxResults": 100
}
```

Latest documents, no keyword:

```json
{
  "maxResults": 200
}
```

***

### 📈 Performance

#### **Processing Speed**

- One or more paginated requests, depending on total result volume
- 0.3 second delay between paginated requests to stay within fair-use expectations
- Automatic retry backoff (up to 3 attempts) on rate limits or transient errors

#### **Resource Usage**

- Memory: Low, since no browser instance is launched
- Network: Multiple lightweight JSON requests, scaling with result volume
- Proxy: One residential proxy tunnel shared across the impersonated session for the run

***

### ⚠️ Important Notes

#### **Legal & Compliance**

- **Fair Use**: Applies pacing delays between paginated requests to the public Datatracker API
- **Public Data Only**: Retrieves only publicly published **IETF Standards** documents already indexed by the Datatracker
- **Legal**: Not legal advice — consult qualified professionals before using this data for compliance-sensitive decisions

#### **Data Quality**

- **Freshness**: Reflects the IETF Datatracker's currently published document set at time of run
- **Completeness**: Field availability depends on what the Datatracker exposes per document; revision may be blank for some entries
- **Accuracy**: Sourced directly from the official IETF Datatracker API
- **Verification**: Cross-check high-stakes specification details against the live datatracker.ietf.org site

#### **Best Practices**

- Use specific keywords (protocol names, working group names) to keep result volume relevant
- Apply `docType` filtering to separate finalized RFCs from in-progress drafts
- Re-run periodically to catch newly published or revised documents
- Omit the keyword to pull the latest documents dataset-wide
- Increase `maxResults` for broader coverage of a protocol area

***

### 📄 License & Legal

**Terms of Use:**

- Use for legitimate research, engineering, and standards compliance purposes
- Respect the IETF Datatracker's Terms of Use
- Use **IETF Standards** data responsibly and in compliance with applicable laws

**Disclaimer:**
IETF Standards & Spec Updates Monitor is provided as-is for research and reference purposes. Users are responsible for compliance with the IETF Datatracker's terms and all applicable laws. This is not legal advice.

***

### ⚖️ IETF Standards Excellence

This Actor is optimized for **IETF Standards** research with:

- ✅ Direct, official Datatracker API access — no HTML scraping fragility
- ✅ Retry-resilient requests with automatic backoff
- ✅ Keyword and document-type filtering
- ✅ Global deduplication across paginated results
- ✅ Incremental Dataset push
- ✅ Production-ready code

# Actor input Schema

## `keyword` (type: `string`):

Topic, protocol, or document name keyword (e.g. 'OAuth', 'TLS', 'HTTP'). Leave blank for latest documents.

## `docType` (type: `string`):

Filter by document type (e.g. ALL, RFC only, or Drafts only).

## `maxResults` (type: `integer`):

Maximum number of standards documents to collect.

## `proxyConfiguration` (type: `object`):

Select Residential Proxy for reliable API requests.

## Actor input object example

```json
{
  "keyword": "OAuth",
  "docType": "ALL",
  "maxResults": 50,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ]
  }
}
```

# Actor output Schema

## `crawlResults` (type: `string`):

No description

## `standardsOverview` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("datapilot/etf-standards-spec-updates-monitor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("datapilot/etf-standards-spec-updates-monitor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call datapilot/etf-standards-spec-updates-monitor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,datapilot/etf-standards-spec-updates-monitor"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/e1eXpppwiXvcmd2jG/builds/Ele6qTVNuyTFxgwAR/openapi.json