Stanford University Scraper

Stanford Profiles Full Scraper

Combine directory crawling and profile-detail extraction in one go—no more juggling two actors.

🚀 What it does • List phase: starts at your School of H&S directory page, follows real “next” links (or ?p=), and collects only valid profile URLs. • Detail phase: fetches each profile and pulls out name, email, departments, affiliations, education histories, honors, bio, and more. • Smart stops: if a page yields no new links, the crawl halts—no phantom pages, no wasted cycles.

🛠️ Key features • All-in-one workflow: one actor, one dataset, zero hassle. • Pagination guardrails: leverages and “no-new-links” breaks to avoid endless loops. • Configurable concurrency: scrape dozens of profiles in parallel without overwhelming the server. • Flexible inputs: start from a URL, inject extra URLs by textarea or dataset ID, and cap list pages if you want.

📥 Inputs

Field Type Default Notes startUrl String https://profiles.stanford.edu/browse/school-of-humanities-and-sciences?affiliations=capFaculty Directory page to kick off the crawl maxPages Integer 0 Max list pages (0 = unlimited) urlsText String "" Extra profile URLs (newline/comma separated) sourceDatasetId String "" Dataset ID whose items contain a url field maxConcurrency Integer 5 Worker threads for detail scraping

📤 Output

A single dataset of JSON records, e.g.:

{ "url": "https://profiles.stanford.edu/jane-doe", "name": "Jane Doe", "email": "jane.doe@stanford.edu", "departments": ["History"], "faculty": ["School of Humanities and Sciences"], "personalWebsite": "https://janedoe.com", "bio": "Professor of History…", "academicAppointments": [ "Associate Professor (2020–)" ], "professionalEducation": [ "PhD, Stanford University (2015)" ], "honorsAwards": [ "Guggenheim Fellowship (2023)" ] }

💡 Use cases • Academic research: map faculty expertise, collaborations, or department distributions. • Lead generation: gather contact info for outreach to professors or postdocs. • Data integration: power your internal CRM, directory, or AI pipeline with fresh Stanford data.

⸻

Got feedback or hitting a wall? Drop a note on the actor’s issue tracker—and happy scraping!