# Hal Open Archive Scraper (`bigibyte/hal-open-archive-scraper`) Actor Search and extract French research paper records from the HAL Open Archive by title, author, year, and URI. Save clean metadata rows without manual browsing, and export your results directly to JSON, CSV, or Excel format. - **URL**: https://apify.com/bigibyte/hal-open-archive-scraper.md - **Developed by:** [BigiByte](https://apify.com/bigibyte) (community) - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ### HAL Open Archive Scraper: French Research Papers

HAL Open Archive Scraper

Collect structured bibliographic data and metadata from HAL Open Archive records. **BigiByte**'s HAL Open Archive Scraper extracts document titles, author details, document URIs, and publication years into clean, structured datasets—saving you from manual search and pagination. *** ### What Does This Scraper Do? HAL Open Archive holds a vast collection of French and international scientific publications. This tool allows researchers, academic institutions, and librarians to pull structured search results directly into tabular formats without needing to click through manual search result pages. #### Key Use Cases - **Literature Sweeps:** Quickly collect records on specific research topics (e.g., climate change, renewable energy, AI ethics) with author full names and titles in a single dataset. - **Archive Monitoring:** Rerun saved queries periodically to track newly added papers and monitor document additions over time. - **Library Exports:** Download organized CSV or Excel lists containing URIs and publication years for curated reading lists. - **Metadata Auditing:** Audit and verify records to ensure they include valid HAL URIs and publication dates. - **Automated Data Refreshes:** Keep your internal databases or research feeds updated automatically. *** ### Input Parameters Configure your search query and output preferences using simple input parameters: | Parameter | Type | Default | Description | | --- | --- | --- | --- | | `mode` | String | `search` | Defines the operation mode. | | `query` | String | `climate` | The search keyword or phrase to match against records. | | `fields` | String | `docid,halId_s,title_s,authFullName_s,abstract_s,docType_s,doiId_s,journalTitle_s,producedDateY_i,publicationDate_s,language_s,domain_s,openAccess_bool,uri_s,fileMain_s` | Comma-separated list of HAL fields to extract. | | `maxItems` | Integer | `100` | Maximum number of successful rows to collect (capped at 1,000). | | `timeoutSecs` | Integer | `300` | Overall execution timeout in seconds. | | `requestTimeoutSecs` | Integer | `60` | Timeout per individual network request in seconds. | | `proxyConfiguration` | Object | Apify Proxy **on** | Requests are routed through Apify Proxy because HAL does not answer the Apify container's direct IP. Switch to the `RESIDENTIAL` group if datacenter IPs are blocked. | #### Example Input ```json { "mode": "search", "query": "climate", "fields": "docid,halId_s,title_s,authFullName_s,abstract_s,docType_s,doiId_s,journalTitle_s,producedDateY_i,publicationDate_s,language_s,domain_s,openAccess_bool,uri_s,fileMain_s", "maxItems": 100, "timeoutSecs": 300, "requestTimeoutSecs": 60 } ``` *** ### Extracted Output Data Each record saved in your dataset contains clean, structured metadata: ```json { "docid": "4672891", "halId_s": "hal-04672891", "title_s": [ "Climate learning scenarios for adaptation decision analyses: Review and classification" ], "authFullName_s": [ "Vanessa Völz", "Jochen Hinkel" ], "abstract_s": "Economic decision analysis is an important tool for developing cost-efficient adaptation pathways...", "docType_s": "ART", "language_s": ["en"], "domain_s": ["0.sde"], "uri_s": "https://hal.science/hal-04672891v1", "scrapedAt": "2026-10-20T09:10:22Z", "doiId_s": "10.1016/j.crm.2023.100512", "journalTitle_s": "Climate Risk Management", "publicationDate_s": "2023", "fileMain_s": "https://hal.science/hal-04672891/document", "openAccess_bool": true, "producedDateY_i": 2023 } ``` #### Output Field Reference | Field | Type | Description | | --- | --- | --- | | `docid` | String | Unique HAL document ID (duplicates are automatically filtered out). | | `halId_s` | String | Human-readable HAL identifier (e.g. `hal-04672891`). | | `title_s` | Array | Document title(s). | | `authFullName_s` | Array | Full names of listed authors. | | `abstract_s` | String | Document abstract, when available. | | `docType_s` | String | HAL document type code (e.g. `ART`, `COMM`, `THESE`). | | `doiId_s` | String | DOI of the document, when recorded. | | `journalTitle_s` | String | Journal title, for journal articles. | | `producedDateY_i` | Integer | Year of publication/production. | | `publicationDate_s` | String | Full publication date, when available. | | `language_s` | Array | Document language code(s) (e.g. `en`, `fr`). | | `domain_s` | Array | HAL subject domain code(s). | | `openAccess_bool` | Boolean | `true` when a full text is openly available on HAL. | | `uri_s` | String | Direct web URI for the document. | | `fileMain_s` | String | Direct URL to the main uploaded file, when available. | | `scrapedAt` | String | UTC timestamp indicating when the row was saved. | | `error` | String | Error details (only present if a specific record failed to process). | | `httpStatus` | Integer | HTTP response status code for error rows. | > Fields that HAL does not return for a given document (such as `doiId_s`, `journalTitle_s`, or `fileMain_s`) are omitted from that record rather than left blank. *** ### How Data Collection Works - **Pagination & Limits:** Requests process up to 30 documents per step, scaling automatically down to match your exact `maxItems` cap. - **Automatic Retries & Stability:** Built-in error handling manages transient network issues or unexpected page responses gracefully without stopping the overall run. - **Deduplication:** Duplicate document IDs (`docid`) are automatically dropped so your dataset remains clean and unique. - **Flexible Export:** Download your data in JSON, CSV, or Excel formats with a single click. *** ### Related Tools & Custom Development If you plan to map, cross-reference, or join academic records with external datasets, additional data enrichment passes or CSV cleanups are standard next steps. **BigiByte** builds tailored data collection solutions and custom pipelines for research repositories and open-data portals worldwide. Need custom dataset integrations, automated scheduled monitors, or specialized extraction tools? **BigiByte** builds custom scrapers and APIs tailored to your needs: *** ### Frequently Asked Questions **How many documents can I collect in a single run?** The default target is 100 items per run, with a maximum cap of 1,000 successful records per task. **Can I export data directly to CSV or Excel?** Yes! Once a run completes, you can export the full dataset directly to JSON, CSV, or Excel. **What happens if a query encounters an error?** If a request encounters a server error or rate limitation, the scraper retries automatically. If an issue persists, an error row is created for that item while the overall task continues running smoothly. *** ### Trademark Notice & Disclaimer **Trademark Notice:** HAL Open Archive and related names, logos, and trademarks belong to their respective owners. Their use in this documentation is strictly to identify the source of data and does not imply affiliation, sponsorship, endorsement, or partnership. **Disclaimer:** This tool is an independent solution developed by BigiByte and is not officially affiliated with, endorsed by, or sponsored by HAL Open Archive. All data is collected from publicly accessible records. Users are responsible for adhering to the source's terms of service, rate guidelines, and licensing or attribution requirements. *** ### Is Web Scraping Legal? This tool collects publicly available research records from HAL Open Archive without bypassing access barriers. While gathering public research data is standard practice, consider the following best practices: - **Personal Data:** Author names and public academic profiles may contain personal data. Ensure compliance with applicable privacy regulations (such as GDPR or CCPA) when storing or processing this information. - **Copyright & Reuse:** Abstracts, articles, and full-text documents remain subject to copyright and Creative Commons licenses. Respect attribution requirements and reuse limitations specified by publishers and authors. - **Terms & Attribution:** Credit HAL Open Archive and the respective authors when republishing or distributing dataset insights. - **Legal Context:** This information provides general context and does not constitute legal advice. To learn more, read Apify's guide: [Is web scraping legal?](https://blog.apify.com/is-web-scraping-legal/) # Actor input Schema ## `mode` (type: `string`): Search returns one row per HAL document from the public search API. ## `query` (type: `string`): Solr query sent as q. A plain word such as climate works. More specific Solr syntax is passed through unchanged. ## `fields` (type: `string`): Comma-separated Solr field list (fl). Defaults to the HAL id, title, authors, abstract, document type, DOI, journal, dates, language, domain, open-access flag, URI, and file link. ## `maxItems` (type: `integer`): Maximum successful document rows to save in one run. Values above 1000 are clamped to 1000. ## `timeoutSecs` (type: `integer`): Stop scheduling new requests after this many seconds. Rows already saved are kept. ## `requestTimeoutSecs` (type: `integer`): Per-request timeout in seconds for each call to the HAL search API. ## `proxyConfiguration` (type: `object`): Apify Proxy is enabled by default. HAL does not respond to the raw Apify datacenter IP, so requests are routed through Apify Proxy. If results are still blocked, switch to the RESIDENTIAL group here. ## Actor input object example ```json { "mode": "search", "query": "climate", "fields": "docid,halId_s,title_s,authFullName_s,abstract_s,docType_s,doiId_s,journalTitle_s,producedDateY_i,publicationDate_s,language_s,domain_s,openAccess_bool,uri_s,fileMain_s", "maxItems": 100, "timeoutSecs": 300, "requestTimeoutSecs": 60, "proxyConfiguration": { "useApifyProxy": true } } ``` # Actor output Schema ## `datasetUrl` (type: `string`): Default dataset for this run. ## `itemsUrl` (type: `string`): Dataset items for this run. ## `overview` (type: `string`): Table of HAL documents from this run. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "mode": "search", "query": "climate", "fields": "docid,halId_s,title_s,authFullName_s,abstract_s,docType_s,doiId_s,journalTitle_s,producedDateY_i,publicationDate_s,language_s,domain_s,openAccess_bool,uri_s,fileMain_s", "maxItems": 100, "timeoutSecs": 300, "requestTimeoutSecs": 60, "proxyConfiguration": { "useApifyProxy": true } }; // Run the Actor and wait for it to finish const run = await client.actor("bigibyte/hal-open-archive-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "mode": "search", "query": "climate", "fields": "docid,halId_s,title_s,authFullName_s,abstract_s,docType_s,doiId_s,journalTitle_s,producedDateY_i,publicationDate_s,language_s,domain_s,openAccess_bool,uri_s,fileMain_s", "maxItems": 100, "timeoutSecs": 300, "requestTimeoutSecs": 60, "proxyConfiguration": { "useApifyProxy": True }, } # Run the Actor and wait for it to finish run = client.actor("bigibyte/hal-open-archive-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "mode": "search", "query": "climate", "fields": "docid,halId_s,title_s,authFullName_s,abstract_s,docType_s,doiId_s,journalTitle_s,producedDateY_i,publicationDate_s,language_s,domain_s,openAccess_bool,uri_s,fileMain_s", "maxItems": 100, "timeoutSecs": 300, "requestTimeoutSecs": 60, "proxyConfiguration": { "useApifyProxy": true } }' | apify call bigibyte/hal-open-archive-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,bigibyte/hal-open-archive-scraper" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/eRhLymPtUcoy1gyog/builds/UpPyMmKbj6q1cvwmh/openapi.json