Clinical Trials Scraper & Pharma Pipeline Data

Fast answer: what this Actor is for

Pull every trial matching a drug, disease, sponsor, or city from ClinicalTrials.gov, and get a read on the competitive landscape in the same run. Built for pharma and biotech teams tracking rival pipelines, investors sizing a therapeutic area, CROs and site networks hunting enrollment, and AI agents that need structured trial data instead of scraped HTML.

• Run it from the Apify UI for a one-off pull.
• Schedule it or call it by API to watch a competitor's pipeline week over week.
• Feed the dataset straight into spreadsheets, dashboards, and LLM workflows.

TL;DR

Search ClinicalTrials.gov by term, condition, sponsor, intervention, or location. Every trial comes back as a clean row: nctId, briefTitle, status and phase as normalized enums, leadSponsor with leadSponsorClass (INDUSTRY, NIH, ACADEMIC, and so on), conditions[], interventions[] with type, enrollmentCount as a real number, siteCount and countryCount, ISO 8601 startDate and primaryCompletionDate, durationDays, eligibility (minimumAge, sex, healthyVolunteers), primaryOutcomes[], and sites[] with facility, city, country, and lat/lon coordinates.

Then the part nobody else ships: one pipeline_summary record per run that turns those rows into a market view. Sponsor competition ranked by trial count, late-phase count, and total enrollment. Phase mix and status mix with percentages. Industry-sponsored share. Median enrollment. Site geography. Every record also carries an agentMarkdown field, and each run writes an AGENT_BRIEFING you can hand straight to an LLM.

Try it on a small query, then let us know what you think in a review.

What does this Actor do?

It queries the official ClinicalTrials.gov v2 API, normalizes every study into a typed record, and computes a landscape summary across the result set. No browser, no proxy, no scraping of rendered HTML, which is why it runs in seconds and costs a fraction of a cent.

Two record types land in one dataset:

1. trial, one per study, with the fields listed above. Dates are ISO 8601 even though the source mixes 2019, 2019-10, and 2019-10-05 formats. Phases are collapsed into a single enum, including combined phases like PHASE1_PHASE2. Enrollment and site counts are numbers, not strings, so you can sum and sort them without cleaning.

2. pipeline_summary, one per run, with topSponsors[] (each with trialCount, latePhaseCount, totalEnrollment), byPhase[] and byStatus[] with counts and percentages, topConditions[], topInterventionTypes[], topCountries[] by site count, plus industrySponsoredPct, latePhasePct, recruitingCount, medianEnrollment, and totalEnrollment.

Why use ClinicalTrials.gov data?

ClinicalTrials.gov is the single registry where drug developers are legally required to disclose what they are testing, on whom, where, and when. That makes it the closest thing to a public map of the pharma pipeline. The problem is the site is built for reading one study at a time. The export gives you flat CSV with no normalization, dates in three formats, phases as ragged arrays, and no way to ask "who is winning in this indication".

Teams end up doing this by hand: search, click, copy sponsor names into a sheet, eyeball which competitors are in phase 3, guess at enrollment totals. That is a day of work that goes stale in a week.

This Actor collapses that into one call. You get the rows and the read on the room together.

Who needs this?

• Competitive intelligence. Point it at a drug class and see which sponsors own the late-stage trials, how many patients they have enrolled, and who just started recruiting.
• Investor diligence. Size a therapeutic area before a term sheet: how crowded is it, how much of it is industry money versus academic, what is the median trial size.
• Site and patient recruitment. Filter to RECRUITING and pull geo-coded sites, then map which facilities are actively enrolling near your catchment.
• AI agents and RAG. Every record ships with agentMarkdown and every run writes an AGENT_BRIEFING, so an LLM can reason over the landscape without you writing a formatter.
• Regulatory and medical affairs. Track status changes (a trial flipping to TERMINATED or COMPLETED) by scheduling this weekly and diffing the output.

How to use Clinical Trials Intelligence

1. Open the Actor Input tab and provide at least one search criterion.
2. Start with one of query, condition, sponsor, intervention, or location.
3. Set maxItems to a small value for the first run.
4. Optionally set recruitment statuses, phases, and emitPipelineSummary.
5. Start the run, then export or deduplicate records by nctId before comparing runs over time.

Examples

Example 1: map a GLP-1 obesity landscape

Input:

{
  "query": "GLP-1 obesity",
  "maxItems": 50
}

That run finished in 6 seconds. Real rows it returned:

NCT ID	Title	Status	Phase	Sponsor	Type	Enrollment	Sites	Countries
NCT07284901	Efficacy and Safety of KAI-9531 Once Weekly in Obesity or Overweight and Diabetes	RECRUITING	PHASE3	Kailera	INDUSTRY	1700	58	4
NCT06603571	Weight Management With LY3841136 and Tirzepatide, Alone or in Combination	ACTIVE_NOT_RECRUITING	PHASE2	Eli Lilly and Company	INDUSTRY	350	52	2
NCT06857942	Tirzepatide Following Ixekizumab in Plaque Psoriasis and Obesity (TOGETHER AMPLIFY-PsO)	RECRUITING	PHASE4	Eli Lilly and Company	INDUSTRY	200	40	1
NCT05671653	Effect of PF-07081532 on Blood Levels of Common Birth Control Pills	TERMINATED	PHASE1	Pfizer	INDUSTRY	32	1	1
NCT06517212	Tirzepatide Weight Loss for MRD+ Early Breast Cancer	RECRUITING	PHASE2	Baylor Research Institute	OTHER	48	1	1

And the pipeline_summary record from the same run, rendered from its agentMarkdown:

## Trial landscape: GLP-1 obesity

50 trials. 10 recruiting or about to. 18% industry sponsored. 22% in late phase (P3, P2/P3, P4).
Median enrollment 49, 7820 participants across the set.

### Phase mix
- NA: 23 (46%)
- PHASE4: 9 (18%)
- PHASE2: 5 (10%)
- PHASE1: 3 (6%)
- PHASE3: 2 (4%)

### Top sponsors
- University of Aarhus (OTHER): 2 trials, 0 late phase, 38 enrolled
- Eli Lilly and Company (INDUSTRY): 2 trials, 1 late phase, 550 enrolled
- Hvidovre University Hospital (OTHER): 2 trials, 0 late phase, 20 enrolled

In one call you learn that the space is mostly investigator-initiated mechanism studies, that Lilly is the industry player with late-stage weight, and that Kailera just put 1,700 patients into a phase 3 across 58 sites. That is the read a competitive intelligence analyst would spend a morning assembling.

Input parameters

Every field is optional, but give it at least one. They combine.

• query searches titles, conditions, and interventions together. Best starting point. Example: CAR-T lymphoma.
• condition narrows to a disease area. Example: non-small cell lung cancer.
• sponsor watches one company. Example: Eli Lilly. Pair it with nothing else to get that sponsor's whole registered pipeline.
• intervention tracks a molecule or device. Example: semaglutide.
• location filters by where trial sites sit. Example: Texas.
• statuses limits recruitment status. Pick RECRUITING to find trials enrolling right now.
• phases limits phase. PHASE3 plus PHASE4 gives you the late-stage picture only.
• maxItems caps how many trial records come back, up to 1,000. The pipeline summary is computed from exactly the trials you pull, so a bigger pull gives a more representative landscape.
• emitPipelineSummary defaults to true. Turn it off if you only want raw rows.

Example 2: watch a sponsor's pipeline

{ "sponsor": "Novo Nordisk", "maxItems": 200 }

The summary tells you how much of their registered work is late phase, how many patients they have committed, and which conditions they are pushing into. Schedule it weekly and the diff is your early warning system.

Example 3: find actively enrolling sites

{ "condition": "pancreatic cancer", "statuses": ["RECRUITING"], "location": "Texas", "maxItems": 100 }

Every returned trial carries sites[] with facility names and lat/lon, so this drops straight into a map.

Example 4: compare late-stage trials in a condition

{ "condition": "ALS", "phases": ["PHASE2", "PHASE3"], "maxItems": 300 }

industrySponsoredPct tells you whether real money is in the space or whether it is still academic. medianEnrollment tells you what a serious trial costs to run. topSponsors tells you who you would be competing with, and latePhaseCount per sponsor tells you how close they are.

Clinical Trials Intelligence output format

Trial record

Field	Type	Notes
nctId	string	Registry identifier, stable and idempotent
url	string	Direct link to the study page
briefTitle, officialTitle	string	Official title is null when the sponsor did not post one
status	enum	RECRUITING, COMPLETED, TERMINATED, WITHDRAWN, and others; UNKNOWN if absent
phase	enum	PHASE1 through PHASE4, plus PHASE1_PHASE2, PHASE2_PHASE3, NA, UNKNOWN
leadSponsor, leadSponsorClass	string, enum	INDUSTRY, NIH, FED, ACADEMIC, OTHER_GOV, NETWORK, OTHER
isIndustrySponsored	boolean	Convenience flag for filtering commercial work
collaborators	string[]	Co-sponsors, often where partnerships surface
conditions	string[]	Diseases under study
interventions	object[]	type (DRUG, BIOLOGICAL, DEVICE, PROCEDURE, RADIATION) and name
enrollmentCount, enrollmentType	number, string	ACTUAL or ESTIMATED; null when not posted
siteCount, countryCount	number	Numeric mirrors, sortable without cleaning
startDate, primaryCompletionDate, completionDate, lastUpdateDate	ISO 8601	Normalized from the registry's mixed formats
durationDays	number	Computed start to completion
minimumAge, maximumAge, sex, healthyVolunteers	mixed	Eligibility signals
sites	object[]	facility, city, state, country, zip, lat, lon
primaryOutcomes	string[]	The endpoints the trial is actually measuring
fieldCompletenessScore	number	0 to 100, weighted by what buyers use
agentMarkdown	string	One-line LLM-ready summary of the record

Pipeline summary record

trialCount, byPhase[], byStatus[], topSponsors[], topConditions[], topInterventionTypes[], topCountries[], industrySponsoredPct, recruitingCount, medianEnrollment, totalEnrollment, latePhasePct, agentMarkdown.

Sample record

The following redacted, schema-valid trial row came from the bounded Cloud canary on 2026-09-25 (query: "GLP-1 obesity", maxItems: 1). The source record is public ClinicalTrials.gov registry data.

{
  "outputSchemaVersion": "2026-07-13",
  "recordType": "trial",
  "source": "clinicaltrials.gov",
  "nctId": "NCT02585427",
  "status": "COMPLETED",
  "phase": "NA",
  "leadSponsorClass": "OTHER_GOV",
  "isIndustrySponsored": false,
  "enrollmentCount": 25,
  "siteCount": 1,
  "countryCount": 1,
  "fieldCompletenessScore": 100,
  "url": "https://clinicaltrials.gov/study/NCT02585427"
}

How much does it cost?

Pay per event. You pay for the trial records and the summary you receive, nothing else. There is no proxy surcharge because this Actor does not need a proxy: it talks to the official ClinicalTrials.gov API directly. A 50-trial run finishes in about 6 seconds.

Because the underlying data is a public API rather than a defended website, this is one of the cheapest data sources in the Skootle catalog, and that is reflected in the price.

Source, permissions, and legal use

This Actor requests the public ClinicalTrials.gov v2 API. It does not use a browser, proxy, login, patient record, or access-control bypass. Registry fields are public study metadata. Buyers remain responsible for checking source records and for their own compliance, redistribution, and downstream-use obligations.

During the Actor run

The Actor validates that at least one search criterion is provided, requests the official API in pages of up to 100 studies, and retries transient fetch failures up to three times per page. It normalizes and schema-validates each trial before writing it. When enabled, it adds one pipeline_summary record and stores an AGENT_BRIEFING. A zero-row result or a fetch or parsing failure ends the run with a clear failure instead of a silent empty success.

FAQ

Is this legal?

ClinicalTrials.gov is a public registry operated by the U.S. National Library of Medicine and provides a public API. This Actor reads public registry metadata through that API. It does not handle patient records. Check the source record and your own downstream-use requirements before relying on or redistributing results.

How fresh is the data?

As fresh as the registry. Sponsors update their entries and this Actor reads the live API on every run, so you see what ClinicalTrials.gov shows today. The lastUpdateDate field on each record tells you when the sponsor last touched that study.

Why do some trials show phase NA or UNKNOWN?

Because the registry does. Observational studies and many investigator-initiated studies have no assigned phase, which the registry records as NA. Where a sponsor left the field blank entirely, this Actor normalizes it to UNKNOWN rather than guessing. Both are honest reflections of the source.

Why is enrollment sometimes missing?

Sponsors are not always required to post it, particularly for very early studies. The field comes back null rather than zero, so your averages are not silently wrong. Every record carries a fieldCompletenessScore so you can filter to only the well-populated rows if you want.

Can I track a competitor over time?

Yes, and this is the highest-value way to use it. Schedule the Actor weekly with a sponsor filter, then diff the datasets. New NCT IDs are new programs. Status flips to TERMINATED are failures. Status flips to RECRUITING on a phase 3 mean a program just got serious.

What happens if my query matches nothing?

The run fails with a clear message instead of quietly reporting success with an empty dataset. You are never billed for a run that gave you nothing. This is a deliberate design choice across every Skootle Actor.

Can I get results as CSV or Excel?

Yes. Every Apify dataset exports to CSV, Excel, JSON, XML, and RSS from the UI or the API. The dataset also ships with two prepared views, one for trials and one for the pipeline summary.

Your feedback

Found a bug, a missing field, or a query that behaves oddly? Please open it on the Issues tab rather than the reviews page. Issues get a response within 48 hours and I use every Actor I publish, so breakages get fixed the same week. Reviews are best saved for whether the tool actually earned its keep.

Maintenance status

• Last verified: 2026-09-25
• Verified behavior: bounded Cloud canary returned one schema-valid trial record and one pipeline_summary record from the current build.
• Maintenance promise: use the Actor Issues tab to report reproducible defects or feature requests.

Why choose Clinical Trials Intelligence?

• Normalized ClinicalTrials.gov trial rows plus an optional per-run pipeline summary.
• Typed fields for filtering, export, and downstream automation.
• A bounded input cap and a clear failure when no trial rows are produced.
• Choose raw registry records when you do not need a normalized dataset or landscape summary.

Other Skootle Actors you might want to check

Production scrapers for niches that move money:

• SAM.gov Federal Contracts Scraper for federal opportunity intelligence, same HTTP-only architecture, same agent-grade output.
• SEC EDGAR Filings Scraper to pair a sponsor's trial pipeline with its filings.
• arXiv Papers Monitor for the research feeding the pipeline.
• Crexi Commercial Real Estate Scraper if your interest is in the sites rather than the science.