Clinical Trials Scraper for Pipeline Intel

Fast answer: what this Actor is for

Pull every trial matching a drug, disease, sponsor, or city from ClinicalTrials.gov, and get a read on the competitive landscape in the same run. Built for pharma and biotech teams tracking rival pipelines, investors sizing a therapeutic area, CROs and site networks hunting enrollment, and AI agents that need structured trial data instead of scraped HTML.

• Run it from the Apify UI for a one-off pull.
• Schedule it or call it by API to watch a competitor's pipeline week over week.
• Feed the dataset straight into spreadsheets, dashboards, and LLM workflows.

TL;DR

Search ClinicalTrials.gov by term, condition, sponsor, intervention, or location. Every trial comes back as a clean row: nctId, briefTitle, status and phase as normalized enums, leadSponsor with leadSponsorClass (INDUSTRY, NIH, ACADEMIC, and so on), conditions[], interventions[] with type, enrollmentCount as a real number, siteCount and countryCount, ISO 8601 startDate and primaryCompletionDate, durationDays, eligibility (minimumAge, sex, healthyVolunteers), primaryOutcomes[], and sites[] with facility, city, country, and lat/lon coordinates.

Then the part nobody else ships: one pipeline_summary record per run that turns those rows into a market view. Sponsor competition ranked by trial count, late-phase count, and total enrollment. Phase mix and status mix with percentages. Industry-sponsored share. Median enrollment. Site geography. Every record also carries an agentMarkdown field, and each run writes an AGENT_BRIEFING you can hand straight to an LLM.

Try it on a small query, then let us know what you think in a review.

What does this Actor do?

It queries the official ClinicalTrials.gov v2 API, normalizes every study into a typed record, and computes a landscape summary across the result set. No browser, no proxy, no scraping of rendered HTML, which is why it runs in seconds and costs a fraction of a cent.

Two record types land in one dataset:

1. trial, one per study, with the fields listed above. Dates are ISO 8601 even though the source mixes 2019, 2019-10, and 2019-10-05 formats. Phases are collapsed into a single enum, including combined phases like PHASE1_PHASE2. Enrollment and site counts are numbers, not strings, so you can sum and sort them without cleaning.

2. pipeline_summary, one per run, with topSponsors[] (each with trialCount, latePhaseCount, totalEnrollment), byPhase[] and byStatus[] with counts and percentages, topConditions[], topInterventionTypes[], topCountries[] by site count, plus industrySponsoredPct, latePhasePct, recruitingCount, medianEnrollment, and totalEnrollment.

Why scrape clinical trial data?

ClinicalTrials.gov is the single registry where drug developers are legally required to disclose what they are testing, on whom, where, and when. That makes it the closest thing to a public map of the pharma pipeline. The problem is the site is built for reading one study at a time. The export gives you flat CSV with no normalization, dates in three formats, phases as ragged arrays, and no way to ask "who is winning in this indication".

Teams end up doing this by hand: search, click, copy sponsor names into a sheet, eyeball which competitors are in phase 3, guess at enrollment totals. That is a day of work that goes stale in a week.

This Actor collapses that into one call. You get the rows and the read on the room together.

• Competitive intelligence. Point it at a drug class and see which sponsors own the late-stage trials, how many patients they have enrolled, and who just started recruiting.
• Investor diligence. Size a therapeutic area before a term sheet: how crowded is it, how much of it is industry money versus academic, what is the median trial size.
• Site and patient recruitment. Filter to RECRUITING and pull geo-coded sites, then map which facilities are actively enrolling near your catchment.
• AI agents and RAG. Every record ships with agentMarkdown and every run writes an AGENT_BRIEFING, so an LLM can reason over the landscape without you writing a formatter.
• Regulatory and medical affairs. Track status changes (a trial flipping to TERMINATED or COMPLETED) by scheduling this weekly and diffing the output.

Worked example: the GLP-1 obesity landscape

Input:

{
  "query": "GLP-1 obesity",
  "maxItems": 50
}

That run finished in 6 seconds. Real rows it returned:

NCT ID	Title	Status	Phase	Sponsor	Type	Enrollment	Sites	Countries
NCT07284901	Efficacy and Safety of KAI-9531 Once Weekly in Obesity or Overweight and Diabetes	RECRUITING	PHASE3	Kailera	INDUSTRY	1700	58	4
NCT06603571	Weight Management With LY3841136 and Tirzepatide, Alone or in Combination	ACTIVE_NOT_RECRUITING	PHASE2	Eli Lilly and Company	INDUSTRY	350	52	2
NCT06857942	Tirzepatide Following Ixekizumab in Plaque Psoriasis and Obesity (TOGETHER AMPLIFY-PsO)	RECRUITING	PHASE4	Eli Lilly and Company	INDUSTRY	200	40	1
NCT05671653	Effect of PF-07081532 on Blood Levels of Common Birth Control Pills	TERMINATED	PHASE1	Pfizer	INDUSTRY	32	1	1
NCT06517212	Tirzepatide Weight Loss for MRD+ Early Breast Cancer	RECRUITING	PHASE2	Baylor Research Institute	OTHER	48	1	1

And the pipeline_summary record from the same run, rendered from its agentMarkdown:

## Trial landscape: GLP-1 obesity

50 trials. 10 recruiting or about to. 18% industry sponsored. 22% in late phase (P3, P2/P3, P4).
Median enrollment 49, 7820 participants across the set.

### Phase mix
- NA: 23 (46%)
- PHASE4: 9 (18%)
- PHASE2: 5 (10%)
- PHASE1: 3 (6%)
- PHASE3: 2 (4%)

### Top sponsors
- University of Aarhus (OTHER): 2 trials, 0 late phase, 38 enrolled
- Eli Lilly and Company (INDUSTRY): 2 trials, 1 late phase, 550 enrolled
- Hvidovre University Hospital (OTHER): 2 trials, 0 late phase, 20 enrolled

In one call you learn that the space is mostly investigator-initiated mechanism studies, that Lilly is the industry player with late-stage weight, and that Kailera just put 1,700 patients into a phase 3 across 58 sites. That is the read a competitive intelligence analyst would spend a morning assembling.

Which inputs should I use?

Every field is optional, but give it at least one. They combine.

• query searches titles, conditions, and interventions together. Best starting point. Example: CAR-T lymphoma.
• condition narrows to a disease area. Example: non-small cell lung cancer.
• sponsor watches one company. Example: Eli Lilly. Pair it with nothing else to get that sponsor's whole registered pipeline.
• intervention tracks a molecule or device. Example: semaglutide.
• location filters by where trial sites sit. Example: Texas.
• statuses limits recruitment status. Pick RECRUITING to find trials enrolling right now.
• phases limits phase. PHASE3 plus PHASE4 gives you the late-stage picture only.
• maxItems caps how many trial records come back, up to 1,000. The pipeline summary is computed from exactly the trials you pull, so a bigger pull gives a more representative landscape.
• emitPipelineSummary defaults to true. Turn it off if you only want raw rows.

Three recipes that pay for themselves

Watch a rival's whole pipeline.

{ "sponsor": "Novo Nordisk", "maxItems": 200 }

The summary tells you how much of their registered work is late phase, how many patients they have committed, and which conditions they are pushing into. Schedule it weekly and the diff is your early warning system.

Find sites that are actively enrolling near you.

{ "condition": "pancreatic cancer", "statuses": ["RECRUITING"], "location": "Texas", "maxItems": 100 }

Every returned trial carries sites[] with facility names and lat/lon, so this drops straight into a map.

Size a therapeutic area before you invest.

{ "condition": "ALS", "phases": ["PHASE2", "PHASE3"], "maxItems": 300 }

industrySponsoredPct tells you whether real money is in the space or whether it is still academic. medianEnrollment tells you what a serious trial costs to run. topSponsors tells you who you would be competing with, and latePhaseCount per sponsor tells you how close they are.

Field reference

Trial record

Field	Type	Notes
nctId	string	Registry identifier, stable and idempotent
url	string	Direct link to the study page
briefTitle, officialTitle	string	Official title is null when the sponsor did not post one
status	enum	RECRUITING, COMPLETED, TERMINATED, WITHDRAWN, and others; UNKNOWN if absent
phase	enum	PHASE1 through PHASE4, plus PHASE1_PHASE2, PHASE2_PHASE3, NA, UNKNOWN
leadSponsor, leadSponsorClass	string, enum	INDUSTRY, NIH, FED, ACADEMIC, OTHER_GOV, NETWORK, OTHER
isIndustrySponsored	boolean	Convenience flag for filtering commercial work
collaborators	string[]	Co-sponsors, often where partnerships surface
conditions	string[]	Diseases under study
interventions	object[]	type (DRUG, BIOLOGICAL, DEVICE, PROCEDURE, RADIATION) and name
enrollmentCount, enrollmentType	number, string	ACTUAL or ESTIMATED; null when not posted
siteCount, countryCount	number	Numeric mirrors, sortable without cleaning
startDate, primaryCompletionDate, completionDate, lastUpdateDate	ISO 8601	Normalized from the registry's mixed formats
durationDays	number	Computed start to completion
minimumAge, maximumAge, sex, healthyVolunteers	mixed	Eligibility signals
sites	object[]	facility, city, state, country, zip, lat, lon
primaryOutcomes	string[]	The endpoints the trial is actually measuring
fieldCompletenessScore	number	0 to 100, weighted by what buyers use
agentMarkdown	string	One-line LLM-ready summary of the record

Pipeline summary record

trialCount, byPhase[], byStatus[], topSponsors[], topConditions[], topInterventionTypes[], topCountries[], industrySponsoredPct, recruitingCount, medianEnrollment, totalEnrollment, latePhasePct, agentMarkdown.

How much does it cost?

Pay per event. You pay for the trial records and the summary you receive, nothing else. There is no proxy surcharge because this Actor does not need a proxy: it talks to the official ClinicalTrials.gov API directly. A 50-trial run finishes in about 6 seconds.

Because the underlying data is a public API rather than a defended website, this is one of the cheapest data sources in the Skootle catalog, and that is reflected in the price.

FAQ

Is this legal? Yes. ClinicalTrials.gov is a public registry operated by the U.S. National Library of Medicine, and it publishes an official public API specifically so tools can query it. This Actor uses that API as intended. It reads only public registry data. It does not touch patient records, and no protected health information exists in the output.

How fresh is the data? As fresh as the registry. Sponsors update their entries and this Actor reads the live API on every run, so you see what ClinicalTrials.gov shows today. The lastUpdateDate field on each record tells you when the sponsor last touched that study.

Why do some trials show phase NA or UNKNOWN? Because the registry does. Observational studies and many investigator-initiated studies have no assigned phase, which the registry records as NA. Where a sponsor left the field blank entirely, this Actor normalizes it to UNKNOWN rather than guessing. Both are honest reflections of the source.

Why is enrollment sometimes missing? Sponsors are not always required to post it, particularly for very early studies. The field comes back null rather than zero, so your averages are not silently wrong. Every record carries a fieldCompletenessScore so you can filter to only the well-populated rows if you want.

Can I track a competitor over time? Yes, and this is the highest-value way to use it. Schedule the Actor weekly with a sponsor filter, then diff the datasets. New NCT IDs are new programs. Status flips to TERMINATED are failures. Status flips to RECRUITING on a phase 3 mean a program just got serious.

What happens if my query matches nothing? The run fails with a clear message instead of quietly reporting success with an empty dataset. You are never billed for a run that gave you nothing. This is a deliberate design choice across every Skootle Actor.

Can I get results as CSV or Excel? Yes. Every Apify dataset exports to CSV, Excel, JSON, XML, and RSS from the UI or the API. The dataset also ships with two prepared views, one for trials and one for the pipeline summary.

Your feedback

Found a bug, a missing field, or a query that behaves oddly? Please open it on the Issues tab rather than the reviews page. Issues get a response within 48 hours and I use every Actor I publish, so breakages get fixed the same week. Reviews are best saved for whether the tool actually earned its keep.

More from Skootle

Production scrapers for niches that move money:

• SAM.gov Federal Contracts Scraper for federal opportunity intelligence, same HTTP-only architecture, same agent-grade output.
• SEC EDGAR Filings Scraper to pair a sponsor's trial pipeline with its filings.
• arXiv Papers Monitor for the research feeding the pipeline.
• Crexi Commercial Real Estate Scraper if your interest is in the sites rather than the science.