ClinicalTrials.gov Scraper — Why Trials Stop

ClinicalTrials.gov is free, official and has a good search UI. Scraping the study record and handing it back is not a product — the registry already does that better.

The field nobody uses is whyStopped. When a sponsor halts a trial they must say why, in free text. Two real examples:

"Initial PI departed institution."

"Study was prematurely terminated due to difficulties experienced in recruiting patients in a reasonable timeframe."

Completely different events. The first is administrative and says nothing about the drug. The second is an accrual failure — the most common cause of death in clinical research and a direct read on how hard that indication is to run.

As free text it is unusable at scale, which is why it sits unread across 34,000+ terminated studies. This Actor classifies it.

Measured on 300 live melanoma trials

Reason	Count
Slow accrual	85
Business decision	38
Safety	37
Sponsor decision, no rationale given	21
Futility / lack of efficacy	18
Funding	18
Administrative	12
Investigator or site	9
No reason given at all	9
COVID / regulatory / drug supply	8
Unclassified	45

197 of the 300 never posted results. A trial that stops and publishes nothing is how a negative finding disappears, and resultsWithheld counts them.

sponsor_decision_undisclosed is its own category on purpose. "Sponsor decision" was the single most common thing sponsors wrote — a reason in form only. Lumping it in with genuine strategic rationale would hide that.

The other signals, all computed from the record

enrolledNobody — a stopped trial whose actual enrolment is one or zero. It never got going, whatever reason was given.

isOverdue — the primary completion date is still marked ESTIMATED and has already passed. The record was never updated to say what happened. Only detectable by comparing the date's type against today, which is why nobody reports it.

isLatePhase — a halted Phase 2 or 3 is a material event; a halted NA study usually is not.

durationMonths — how long a trial survived before it was killed.

Input

{
  "condition": "melanoma",
  "stoppedOnly": true,
  "phases": ["PHASE2", "PHASE3"],
  "maxStudies": 500
}

Field	Default	Notes
condition	melanoma	Disease or condition.
intervention	—	Drug, device or procedure.
sponsor	—	Lead sponsor only, not collaborators.
searchTerm	—	Extra registry search terms.
stoppedOnly	true	Applied registry-side, not after downloading.
statuses	—	Specific statuses; overrides stoppedOnly.
phases	—	PHASE1…PHASE4, NA.
maxStudies	500	Real ceiling — the registry is large.
pageSize	100	Up to 1000.

An unfiltered query is refused rather than quietly downloading the whole registry.

Output

study — one row per trial:

{
  "type": "study",
  "nctId": "NCT06185088",
  "status": "TERMINATED",
  "whyStopped": "Initial PI departed institution.",
  "stopReason": "investigator_or_site",
  "stopReasonGiven": true,
  "phase": "NA",
  "isLatePhase": false,
  "leadSponsor": "University of Michigan",
  "isIndustrySponsored": false,
  "enrollment": 1,
  "enrolledNobody": true,
  "durationMonths": 3,
  "resultsWithheld": false,
  "countries": ["United States", "Canada"],
  "url": "https://clinicaltrials.gov/study/NCT06185088"
}

analysis — one cohort summary with the full reason breakdown, results- withheld rate, median months before stopping and the sponsor stopping the most trials.

Who uses this

• Biotech and pharma competitive intelligence — what kills trials in an indication before you commit to running one there
• Site selection and CRO planning — accrual failure rate by condition is the number that decides feasibility
• Investors — a Phase 3 halted for futility is a very different signal to one halted because the PI left
• Research integrity — the results-withheld rate, by sponsor

Reliability

• Official ClinicalTrials.gov API v2. Public, unauthenticated, no key.
• Paging is by opaque nextPageToken. There is no offset parameter — passing one is silently ignored and every page comes back identical.
• Registry dates arrive as YYYY, YYYY-MM or YYYY-MM-DD and are normalised without inventing precision.
• One malformed record never fails the run; it lands in RUN_SUMMARY.

Limits

• The classifier reads English free text written by thousands of different sponsors. On the live melanoma run 15% fell through to other — those keep their raw whyStopped so you can read them yourself. The classifier is a filter over the sponsor's words, never a replacement for them.
• Where reasons overlap the strongest signal wins: a trial stopped "for safety after slow enrolment" is classified as a safety stop.
• stopReason is null for trials still running. It is not a prediction.
• Registry records are self-reported and frequently stale — which is precisely what isOverdue measures.