ClinicalTrials.gov Scraper

Search the ClinicalTrials.gov registry by condition, intervention, sponsor or location, or pull specific studies by NCT id. Status, phase, enrolment, eligibility, interventions, sponsors, sites and dates.

Reads the registry's official API v2. No key, no login.

Enrolment is often a target, not a fact

This is the field that most quietly produces wrong numbers, so it is worth knowing before you sum anything.

enrollment comes with an enrollment_type of either ACTUAL or ESTIMATED. An estimate is a recruitment target the sponsor hopes to hit; a trial that never finished recruiting still carries it. Add the two together and you overstate participation across every study still enrolling.

On a test run of 40 recruiting diabetes studies, all 40 reported an estimated enrolment — which is correct, because a trial that is still recruiting has not achieved a number yet. The run summary counts them so the shape of your data is visible before you aggregate it.

Why this Actor connects directly

:

Route	Result
Direct	HTTP 200, working JSON API
Through a residential proxy	CONNECT tunnel failed, response 491

A 491 on the CONNECT is our proxy refusing to tunnel to the host, not the registry refusing us. Both .gov targets measured this session behaved the same way, so a proxy error on a government host says nothing about the site.

ClinicalTrials.gov publishes this API for exactly this use, so the Actor goes direct and paces itself politely.

Everything lives in a module

A study is a protocolSection containing a dozen modules, each holding a slice of the record, and several fields are wrapped again inside that:

identificationModule nctId, briefTitle, officialTitle
statusModule overallStatus, startDateStruct { date, type }
designModule studyType, phases, designInfo, enrollmentInfo { count, type }
eligibilityModule minimumAge, maximumAge, sex, stdAges
contactsLocationsModule locations [ { country, city, facility } ]

Dates are inside *DateStruct objects. Enrolment is inside enrollmentInfo. Allocation and masking are inside designInfo, and masking is inside maskingInfo inside that. Reading a module and hoping for a flat field returns nothing and raises nothing, so every extraction here names its full path and the tests pin them.

Fields

• Identity: nct_id, title, official_title, url
• Status: status, why_stopped, has_results
• Design: study_type, phases, allocation, masking, primary_purpose
• Scale: enrollment, enrollment_type, location_count, countries
• Subject: conditions, keywords, interventions, intervention_types, primary_outcomes
• Who runs it: lead_sponsor, lead_sponsor_class, collaborators
• Who can join: minimum_age, maximum_age, sex, healthy_volunteers, std_ages
• When: start_date, primary_completion_date, completion_date, first_posted, last_update_posted
• Plain language: brief_summary

countries is deduplicated while location_count counts sites, so a study with 28 sites across Italy shows 28 and ["Italy"]. Those are different facts and both are useful.

lead_sponsor_class separates INDUSTRY from NIH, academic and other funders, which is usually the first cut in any competitive analysis.

Input reference

Field	Type	Default
condition	disease or condition	diabetes
intervention	drug or device	—
sponsor	sponsoring organisation	—
location	country, state or city	—
search	any other free-text term	—
status	one recruitment status	any
nct_ids	specific studies by id	—
limit	1-5000	100
retries	1-6	3

Filters combine. Condition plus status plus sponsor narrows to exactly the studies you want before pagination, which costs far fewer requests than filtering afterwards.

Typical uses

• Competitive intelligence. Every trial a sponsor is running, with phase, status and enrolment. Filter lead_sponsor_class: INDUSTRY to drop academic studies.
• Pipeline tracking. Filter by intervention to follow a drug across sponsors and indications.
• Site selection. countries and location_count show where trials for a condition actually run.
• Recruitment monitoring. status: RECRUITING with a condition gives the live trials, and minimum_age / sex / healthy_volunteers are the eligibility filters patients are matched on.
• Trial outcome research. has_results flags studies that have posted results, and why_stopped explains the ones that halted.

Notes on behaviour

An NCT id that does not exist returns a not_found error naming it, never a row of nulls. A malformed id is rejected as bad_input before a request is spent, and the two are reported separately because they mean different things.

A 400 means the registry rejected the query and is not retried, since retrying a bad parameter cannot help. A 429 or 500 is retried with backoff.

totalCount is reported in the summary, so when the registry matches more studies than your limit you can see it rather than assuming you have everything. Searching diabetes returns 24,325 studies in total; a limit of 100 gets you the first hundred.