# NCES College Navigator Scraper (`automation-lab/nces-college-navigator-scraper`) Actor Extract NCES College Navigator institution profiles with IPEDS IDs, websites, enrollment, tuition, admissions, retention, and graduation data. - **URL**: https://apify.com/automation-lab/nces-college-navigator-scraper.md - **Developed by:** [Stas Persiianenko](https://apify.com/automation-lab) (community) - **Categories:** Other - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per event This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## NCES College Navigator Scraper Extract public institution profiles from NCES College Navigator for higher-ed market research, vendor lead lists, admissions benchmarking, recruiting analysis, and school comparison workflows. The actor searches `nces.ed.gov/collegenavigator`, follows institution detail pages, and saves structured rows with IPEDS IDs, names, addresses, websites, enrollment, tuition, admissions, retention, graduation, and related public profile fields. ### What does NCES College Navigator Scraper do? NCES College Navigator Scraper turns public College Navigator pages into a clean Apify dataset. It can: - 🔎 Search College Navigator by keyword - 🏫 Extract exact institution URLs - 🆔 Capture IPEDS unit IDs - 🌐 Save school websites - 📍 Normalize address, city, state, and ZIP fields - ☎️ Capture public phone/contact fields - 🎓 Extract awards, sector, control, and campus settings - 📊 Save enrollment, tuition, retention, and graduation summaries ### Who is it for? This actor is useful for teams that need repeatable public higher-education data. - Higher-ed vendors building target account lists - Admissions researchers comparing peer institutions - Recruiting teams sourcing colleges by program or region - Consultants benchmarking schools and market segments - Data teams enriching CRM records with IPEDS identifiers - Analysts monitoring public NCES profile changes ### Why use this actor? College Navigator is public but not optimized for bulk analysis. This actor helps you avoid manual copying by turning pages into rows that can be exported to CSV, JSON, Excel, Google Sheets, BI tools, CRMs, and enrichment pipelines. ### Data you can extract | Field | Description | | --- | --- | | `unitId` | IPEDS institution identifier | | `institutionName` | Public institution name | | `sourceUrl` | College Navigator detail URL | | `address`, `city`, `state`, `zip` | Location fields | | `phone` | General information phone | | `website` | Institution website shown by NCES | | `institutionType`, `sector`, `control` | Institution classification | | `awardsOffered` | Awards and credential levels | | `studentPopulation` | Student population summary | | `totalEnrollment` | Total enrollment where visible | | `tuitionAndFees` | Latest visible tuition and fees | | `retentionRates` | NCES retention summary text | | `graduationRates` | NCES graduation summary text | ### How much does it cost to scrape NCES College Navigator? The actor uses pay-per-event pricing. You pay a small start fee per run and a per-institution fee for every saved dataset item. The exact current prices are shown on the Apify actor page before launch. Use a small `maxItems` value for trial runs, then increase it for production lists. ### How to use NCES College Navigator Scraper 1. Open the actor on Apify. 2. Enter a keyword such as `university`, `nursing`, `engineering`, or a school name. 3. Optionally enter a two-letter state code. 4. Optionally paste exact College Navigator institution URLs. 5. Set `maxItems`. 6. Run the actor. 7. Export the dataset. ### Input options #### `keyword` Search term used on College Navigator. Examples: - `university` - `community college` - `nursing` - `computer science` - `technical institute` #### `state` Optional two-letter state code. The actor applies this after loading institution detail pages, because the public College Navigator state selector is an ASP.NET multi-select that is not always reliable as a simple URL parameter. #### `institutionUrls` Use exact College Navigator URLs when you already know the schools. Example: ```json [ { "url": "https://nces.ed.gov/collegenavigator/?id=110635" } ] ```` #### `maxItems` Maximum number of institution rows to save. #### `includeDetails` Keep this enabled for full institution profile data. ### Example input ```json { "keyword": "university", "state": "CA", "maxItems": 25, "includeDetails": true } ``` ### Example output ```json { "unitId": "488031", "institutionName": "Abraham Lincoln University", "city": "Glendale", "state": "California", "website": "www.alu.edu/", "institutionType": "4-year, Private for-profit", "studentPopulation": 112, "tuitionAndFees": "$6,480", "sourceUrl": "https://nces.ed.gov/collegenavigator/?id=488031" } ``` ### Tips for better results - Start broad, then filter in your spreadsheet. - Use state filters for regional lists. - Use direct URLs for deterministic benchmark sets. - Keep trial runs small. - Increase `maxItems` only after confirming the output shape. ### Integrations Use this actor with: - Google Sheets for school-list workflows - Airtable for vendor account research - HubSpot or Salesforce enrichment pipelines - Snowflake, BigQuery, or Postgres warehouses - Apify webhooks for scheduled monitoring - Zapier or Make for no-code routing ### API usage #### Node.js ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('automation-lab/nces-college-navigator-scraper').call({ keyword: 'university', state: 'CA', maxItems: 25, includeDetails: true, }); console.log(run.defaultDatasetId); ``` #### Python ```python from apify_client import ApifyClient client = ApifyClient('MY-APIFY-TOKEN') run = client.actor('automation-lab/nces-college-navigator-scraper').call(run_input={ 'keyword': 'nursing', 'maxItems': 100, 'includeDetails': True, }) print(run['defaultDatasetId']) ``` #### cURL ```bash curl -X POST 'https://api.apify.com/v2/acts/automation-lab~nces-college-navigator-scraper/runs?token=MY-APIFY-TOKEN' \ -H 'Content-Type: application/json' \ -d '{"keyword":"university","state":"CA","maxItems":25,"includeDetails":true}' ``` ### MCP usage Use the Apify MCP server with Claude Code, Claude Desktop, Cursor, or another MCP-capable assistant when you want an agent to launch school-data runs and then analyze the returned dataset. MCP URL: ```text https://mcp.apify.com/?tools=automation-lab/nces-college-navigator-scraper ``` Claude Code setup: ```bash claude mcp add apify-nces-college-navigator "https://mcp.apify.com/?tools=automation-lab/nces-college-navigator-scraper" ``` Claude Desktop / JSON config example: ```json { "mcpServers": { "apify-nces-college-navigator": { "url": "https://mcp.apify.com/?tools=automation-lab/nces-college-navigator-scraper" } } } ``` Example prompts: - "Scrape NCES College Navigator for California universities and summarize tuition ranges." - "Build a CSV of public nursing school profiles from College Navigator." - "Extract these IPEDS College Navigator URLs and compare enrollment fields." ### Scheduling You can schedule runs on Apify to refresh school lists monthly or quarterly. This is useful when your team tracks institution profiles, enrollment snapshots, or public website/contact changes. ### Data quality notes NCES pages can omit fields for some schools. The actor returns `null` when a public field is unavailable or not shown in the page HTML. Graph-based summaries such as retention and graduation are captured from NCES-provided alternative text when available. ### Legality The actor extracts public information from a US government website. Always use the data responsibly, respect Apify terms, respect NCES website policies, and follow applicable privacy and compliance requirements in your jurisdiction. ### FAQ #### Can I scrape all US colleges? Yes, set a broad keyword and a higher `maxItems`, or run several targeted searches. For very large extracts, use sensible limits and schedules. #### Why did my state-filtered run inspect more schools than it saved? College Navigator's state selector is not exposed as a simple stable GET parameter. The actor filters by state after loading detail pages. #### Why are some tuition or admissions fields empty? NCES does not publish every field for every institution. Missing public data is returned as `null`. #### Does this actor require a proxy? No proxy is required for the current HTTP implementation. ### Related scrapers Explore other Automation Lab actors at `https://apify.com/automation-lab/` for education, lead generation, and public-data workflows. ### Support If a run fails or a field appears stale, open an Apify issue with the run ID and example institution URL. ### Changelog Initial version extracts public NCES College Navigator institution profile fields from search and direct URLs. # Actor input Schema ## `keyword` (type: `string`): Search text used on College Navigator, such as a school name, program term, or market keyword. ## `state` (type: `string`): Optional two-letter US state code, for example CA, TX, NY, or FL. ## `institutionUrls` (type: `array`): Optional College Navigator detail URLs. These are processed before search results. ## `unitIds` (type: `array`): Optional six-digit IPEDS institution IDs, such as 110635 or 243744. ## `maxItems` (type: `integer`): Maximum number of institution records to save. Keep the prefill small for cheap test runs. ## `includeDetails` (type: `boolean`): When enabled, visits each institution profile to extract enrollment, tuition, admissions, retention, and graduation data. ## Actor input object example ```json { "keyword": "university", "state": "CA", "institutionUrls": [ { "url": "https://nces.ed.gov/collegenavigator/?id=110635" } ], "unitIds": [ "110635", "243744" ], "maxItems": 20, "includeDetails": true } ``` # Actor output Schema ## `overview` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "keyword": "university", "state": "CA", "institutionUrls": [ { "url": "https://nces.ed.gov/collegenavigator/?id=110635" } ], "unitIds": [ "110635", "243744" ], "maxItems": 20, "includeDetails": true }; // Run the Actor and wait for it to finish const run = await client.actor("automation-lab/nces-college-navigator-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "keyword": "university", "state": "CA", "institutionUrls": [{ "url": "https://nces.ed.gov/collegenavigator/?id=110635" }], "unitIds": [ "110635", "243744", ], "maxItems": 20, "includeDetails": True, } # Run the Actor and wait for it to finish run = client.actor("automation-lab/nces-college-navigator-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "keyword": "university", "state": "CA", "institutionUrls": [ { "url": "https://nces.ed.gov/collegenavigator/?id=110635" } ], "unitIds": [ "110635", "243744" ], "maxItems": 20, "includeDetails": true }' | apify call automation-lab/nces-college-navigator-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=automation-lab/nces-college-navigator-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "NCES College Navigator Scraper", "description": "Extract NCES College Navigator institution profiles with IPEDS IDs, websites, enrollment, tuition, admissions, retention, and graduation data.", "version": "0.1", "x-build-id": "67IaR0nYPJckCRHmO" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/automation-lab~nces-college-navigator-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-automation-lab-nces-college-navigator-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/automation-lab~nces-college-navigator-scraper/runs": { "post": { "operationId": "runs-sync-automation-lab-nces-college-navigator-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/automation-lab~nces-college-navigator-scraper/run-sync": { "post": { "operationId": "run-sync-automation-lab-nces-college-navigator-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "keyword": { "title": "Keyword", "type": "string", "description": "Search text used on College Navigator, such as a school name, program term, or market keyword." }, "state": { "title": "State", "minLength": 2, "maxLength": 2, "type": "string", "description": "Optional two-letter US state code, for example CA, TX, NY, or FL." }, "institutionUrls": { "title": "Institution URLs", "type": "array", "description": "Optional College Navigator detail URLs. These are processed before search results.", "items": { "type": "object", "required": [ "url" ], "properties": { "url": { "type": "string", "title": "URL of a web page", "format": "uri" } } } }, "unitIds": { "title": "IPEDS unit IDs", "type": "array", "description": "Optional six-digit IPEDS institution IDs, such as 110635 or 243744.", "items": { "type": "string" } }, "maxItems": { "title": "Maximum institutions", "minimum": 1, "maximum": 10000, "type": "integer", "description": "Maximum number of institution records to save. Keep the prefill small for cheap test runs.", "default": 20 }, "includeDetails": { "title": "Fetch detail pages", "type": "boolean", "description": "When enabled, visits each institution profile to extract enrollment, tuition, admissions, retention, and graduation data.", "default": true } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```