Clinical Trials Protocol Scraper

🧬 Clinical Trials Protocol Scraper is a powerful Apify Actor designed to discover, track, and collect clinical study data directly from ClinicalTrials.gov's own public API. This tool provides comprehensive Clinical Trials Protocol intelligence including protocol ID, study title, recruitment status, lead sponsor, brief summary, and a direct link to the full study record for any keyword. Whether you're building a medical research dashboard, tracking pharmaceutical pipeline activity, or conducting healthcare research, the Clinical Trials Protocol Scraper delivers actionable Clinical Trials Protocol insights efficiently.

With retry-resilient API querying, keyword-based full-text search, token-based pagination, and reliable Apify Dataset delivery, the Clinical Trials Protocol Scraper ensures comprehensive Clinical Trials Protocol coverage across any list of search keywords. It focuses on key Clinical Trials Protocol signals including overall status, sponsor, and study summary, making it an essential tool for medical research and pharmaceutical intelligence.

---

📋 Table of Contents

• Features
• Data Source
• How It Works
• Input
• Output
• Technical Stack
• Data Fields
• Use Cases
• Quick Start
• Configuration
• Performance
• Important Notes
• License & Legal

---

🔥 Features

• Direct API Access – Queries ClinicalTrials.gov's official Studies API (v2) directly for fast, structured Clinical Trials Protocol data.
• Multi-Keyword Search – Accepts any number of search keywords and scans matching studies for each.
• Status Filtering – Optionally filters results by overall recruitment status (e.g., recruiting, completed, terminated).
• Token-Based Pagination – Automatically pages through all available results using ClinicalTrials.gov's nextPageToken mechanism.
• Retry-Resilient Requests – Automatically retries on rate limits (429) and auth errors (401/403) with backoff delays.
• Structured Protocol Parsing – Extracts NCT ID, title, status, lead sponsor, and summary from each study's protocol section.
• Direct Study Links – Builds a clickable link to the full study record on ClinicalTrials.gov for every result.
• Residential Proxy Support – Apify residential proxy for reliable Clinical Trials Protocol access.
• Configurable Result Cap – Control how many studies are pulled per keyword.
• Incremental Dataset Push – Pushes each batch of results to the Dataset as it's collected, not just at the end.
• Graceful Error Handling – Skips a failing page or keyword without stopping the whole run.

---

📊 Data Source

ClinicalTrials.gov Public API (v2)

• Authority: The U.S. National Library of Medicine's official ClinicalTrials.gov Studies API (clinicaltrials.gov/api/v2/studies)
• Access Method: Browser-impersonated HTTP requests via curl_cffi, run through an executor for async compatibility
• Coverage: Any publicly listed Clinical Trials Protocol matching the given keyword and optional status filter
• Data: Structured JSON study records, including protocol identification, status, sponsor, and description modules
• Access: Public API, no API key required
• Update Frequency: Reflects ClinicalTrials.gov's current live registry at time of run

---

⚙️ How It Works

The Clinical Trials Protocol Scraper accepts a list of search keywords and an optional recruitment status filter, then queries ClinicalTrials.gov's Studies API for each keyword. For each keyword, the Actor pages through all available results using the API's page-token pagination, retrying automatically on rate limits or transient auth errors with increasing backoff delays. Every study record is normalized into a clean protocol entry, extracting the NCT ID, brief or official title, overall status, lead sponsor name, and brief summary from the study's protocol section, along with a direct link to the full record. Results are pushed to the Apify Dataset incrementally, batch by batch, as they're collected across all keywords.

Key Processing Steps:

1. Input Parsing – Accept a list of keywords (or a single keyword), an optional status filter, and a max result count
2. Proxy Setup – Configure Apify residential proxy for the impersonated HTTP session
3. Keyword Loop – Iterate through each Clinical Trials Protocol search keyword
4. Search Request – Query the Studies API with the keyword, page size, and optional status filter
5. Retry Handling – Back off and retry on rate limits (429) or auth errors (401/403)
6. Study Normalization – Extract NCT ID, title, status, lead sponsor, and summary from each study's protocol section
7. Pagination – Continue requesting subsequent pages using the returned page token until results or the max cap are exhausted
8. Incremental Push – Push each batch of normalized results to the Dataset as it's ready
9. Rate Limiting – Apply a short delay between paginated requests

Key Benefits:

• Pull Clinical Trials Protocol data for any keyword without manually searching the registry
• Filter directly by recruitment status to focus on active or completed studies
• Monitor multiple keywords or therapeutic areas in a single run
• Feed pharmaceutical intelligence dashboards, academic research pipelines, or patient-matching tools
• Automate recurring Clinical Trials Protocol checks as new studies are registered

---

📥 Input

The Actor accepts the following input parameters:

Field	Type	Default	Description
keywords	array	["cancer"]	List of search terms to scan for matching Clinical Trials Protocol entries.
keyword	string	"cancer"	Optional single keyword, used if keywords is not provided.
status	string	(none)	Optional overall status filter (e.g., "RECRUITING"); use "ALL" or omit for no filter.
maxResults	integer	100	Maximum number of Clinical Trials Protocol records to collect per keyword.
proxyConfiguration	object	{"useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"]}	Apify proxy configuration for the search session.

Example Input:

{
  "keywords": ["breast cancer", "immunotherapy"],
  "status": "RECRUITING",
  "maxResults": 100,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  }
}

Search a Single Keyword:

{
  "keyword": "diabetes"
}

Search Multiple Keywords Without a Status Filter:

{
  "keywords": ["Alzheimer's", "Parkinson's"],
  "status": "ALL"
}

---

📤 Output

The Actor pushes Clinical Trials Protocol records with the following structure:

Field	Type	Description
search_keyword	string	The keyword that matched this Clinical Trials Protocol result
protocol_id	string	The study's NCT identification number
title	string	Brief or official study title
status	string	Overall recruitment status (e.g., "RECRUITING", "COMPLETED")
lead_sponsor	string	Name of the study's lead sponsor organization
summary	string	Brief summary of the study's purpose and design
link	string	Direct URL to the full study record on ClinicalTrials.gov
scraped_at	string	ISO 8601 timestamp of when the record was scraped

Example Clinical Trials Protocol Record:

{
  "search_keyword": "breast cancer",
  "protocol_id": "NCT05123456",
  "title": "A Phase 2 Study of Novel Combination Therapy in Advanced Breast Cancer",
  "status": "RECRUITING",
  "lead_sponsor": "Example Oncology Research Institute",
  "summary": "This study evaluates the safety and efficacy of a combination treatment regimen in patients with advanced breast cancer...",
  "link": "https://clinicaltrials.gov/study/NCT05123456",
  "scraped_at": "2026-08-13T09:00:00+00:00"
}

---

🧰 Technical Stack

• HTTP Client: curl_cffi with Chrome impersonation for reliable access to the ClinicalTrials.gov API
• Concurrency: asyncio with an executor bridge for running the impersonated client asynchronously
• Date Handling: datetime with timezone-aware UTC timestamps
• Proxy: Apify Proxy with RESIDENTIAL configuration by default
• Logging: Apify Actor logging system
• Platform: Apify Actor serverless environment

---

📊 Data Fields Explained

Protocol Identity

• protocol_id: The unique NCT number assigned to the study by ClinicalTrials.gov
• title: The study's brief title, falling back to the official title when needed

Study Details

• status: The study's current overall recruitment status
• lead_sponsor: The organization sponsoring and leading the study
• summary: A brief description of the study's purpose and design

Access

• link: Direct link to view the full Clinical Trials Protocol record on ClinicalTrials.gov
• search_keyword: The keyword that surfaced this result, for traceability across multi-keyword runs

---

🎯 Use Cases

• Pharmaceutical Intelligence – Track competitor or industry-wide trial activity for a given condition or drug class
• Academic Research – Study trial design, sponsor activity, and recruitment trends at scale
• Patient & Advocacy Support – Surface actively recruiting studies relevant to a specific condition
• Investor Research – Monitor a biotech or pharma company's trial pipeline via lead sponsor tracking
• Regulatory & Compliance Research – Reference registered protocol details for a given therapeutic area
• Media Monitoring – Source primary trial data for health and science journalism

---

🚀 Quick Start

1. Prepare Input

Go to Apify Console and enter:

{
  "keywords": ["cancer"]
}

2. Run the Actor

Click Start. The Actor will:

• Search ClinicalTrials.gov for each keyword, applying any status filter
• Page through all matching results using token-based pagination, retrying on rate limits automatically
• Normalize each study into a clean Clinical Trials Protocol record
• Push batches to the Dataset incrementally as they're collected

3. Monitor Progress

Console shows:

[1/1] Searching ClinicalTrials.gov for: 'cancer'
Saved 100 protocols for 'cancer' (Progress: 100/100)
🎉 Scraping completed! Total 100 clinical trial protocols saved to dataset.

4. View & Download Results

• Results Tab: All Clinical Trials Protocol records
• Export: JSON, CSV, Excel, or HTML
• Filter: By status or lead sponsor
• API Access: Available via the Apify API

---

⚙️ Configuration

Single keyword search:

{
  "keyword": "diabetes"
}

Multiple keywords with status filter:

{
  "keywords": ["immunotherapy", "gene therapy"],
  "status": "RECRUITING",
  "maxResults": 200
}

No status filter (all trial states):

{
  "keywords": ["cancer"],
  "status": "ALL"
}

---

📈 Performance

Processing Speed

• One or more paginated requests per keyword, depending on total result volume and maxResults
• 0.3 second delay between paginated requests to stay within fair-use expectations
• Automatic retry backoff (up to 3 attempts) on rate limits or transient errors

Resource Usage

• Memory: Low, since no browser instance is launched
• Network: Multiple lightweight JSON requests per keyword, scaling with result volume
• Proxy: One residential proxy tunnel shared across the impersonated session for the run

---

⚠️ Important Notes

Legal & Compliance

• Fair Use: Applies pacing delays between paginated requests to the public API
• Public Data Only: Retrieves only publicly registered Clinical Trials Protocol data already indexed by ClinicalTrials.gov
• Legal: Not legal advice — consult qualified professionals before using this data for compliance-sensitive decisions; not intended to support individual medical decisions

Data Quality

• Freshness: Reflects ClinicalTrials.gov's current live registry at time of run
• Completeness: Field availability depends on what each study's protocol section includes; some studies may have limited descriptions
• Accuracy: Sourced directly from the official ClinicalTrials.gov API
• Verification: Cross-check high-stakes protocol details against the live ClinicalTrials.gov website

Best Practices

• Use specific keywords to keep result volume manageable and relevant
• Apply a status filter to focus on actively recruiting or completed studies as needed
• Re-run periodically to catch newly registered or updated protocols
• Increase maxResults for broader keyword coverage, keeping pagination volume in mind
• Search multiple related keywords in one run to broaden therapeutic area coverage

---

📄 License & Legal

Terms of Use:

• Use for legitimate research, academic, and pharmaceutical intelligence purposes
• Respect ClinicalTrials.gov's Terms of Use
• Use Clinical Trials Protocol data responsibly and in compliance with applicable laws
• Not a substitute for professional medical advice

Disclaimer: Clinical Trials Protocol Scraper is provided as-is for research and analytics purposes. Users are responsible for compliance with ClinicalTrials.gov's terms and all applicable laws. This is not legal or medical advice.

---

⚖️ Clinical Trials Protocol Excellence

This Actor is optimized for Clinical Trials Protocol research with:

• ✅ Direct, official ClinicalTrials.gov API access — no HTML scraping fragility
• ✅ Retry-resilient requests with automatic backoff
• ✅ Full token-based pagination across all matching results
• ✅ Multi-keyword search with optional status filtering
• ✅ Incremental Dataset push
• ✅ Production-ready code