Hal Open Archive Scraper

HAL Open Archive Scraper: French Research Papers

Collect structured bibliographic data and metadata from HAL Open Archive records. BigiByte's HAL Open Archive Scraper extracts document titles, author details, document URIs, and publication years into clean, structured datasets—saving you from manual search and pagination.

---

What Does This Scraper Do?

HAL Open Archive holds a vast collection of French and international scientific publications. This tool allows researchers, academic institutions, and librarians to pull structured search results directly into tabular formats without needing to click through manual search result pages.

Key Use Cases

• Literature Sweeps: Quickly collect records on specific research topics (e.g., climate change, renewable energy, AI ethics) with author full names and titles in a single dataset.
• Archive Monitoring: Rerun saved queries periodically to track newly added papers and monitor document additions over time.
• Library Exports: Download organized CSV or Excel lists containing URIs and publication years for curated reading lists.
• Metadata Auditing: Audit and verify records to ensure they include valid HAL URIs and publication dates.
• Automated Data Refreshes: Keep your internal databases or research feeds updated automatically.

---

Input Parameters

Configure your search query and output preferences using simple input parameters:

Parameter	Type	Default	Description
mode	String	search	Defines the operation mode.
query	String	climate	The search keyword or phrase to match against records.
fields	String	docid,halId_s,title_s,authFullName_s,abstract_s,docType_s,doiId_s,journalTitle_s,producedDateY_i,publicationDate_s,language_s,domain_s,openAccess_bool,uri_s,fileMain_s	Comma-separated list of HAL fields to extract.
maxItems	Integer	100	Maximum number of successful rows to collect (capped at 1,000).
timeoutSecs	Integer	300	Overall execution timeout in seconds.
requestTimeoutSecs	Integer	60	Timeout per individual network request in seconds.
proxyConfiguration	Object	Apify Proxy on	Requests are routed through Apify Proxy because HAL does not answer the Apify container's direct IP. Switch to the RESIDENTIAL group if datacenter IPs are blocked.

Example Input

{
    "mode": "search",
    "query": "climate",
    "fields": "docid,halId_s,title_s,authFullName_s,abstract_s,docType_s,doiId_s,journalTitle_s,producedDateY_i,publicationDate_s,language_s,domain_s,openAccess_bool,uri_s,fileMain_s",
    "maxItems": 100,
    "timeoutSecs": 300,
    "requestTimeoutSecs": 60
}

---

Extracted Output Data

Each record saved in your dataset contains clean, structured metadata:

{
    "docid": "4672891",
    "halId_s": "hal-04672891",
    "title_s": [
        "Climate learning scenarios for adaptation decision analyses: Review and classification"
    ],
    "authFullName_s": [
        "Vanessa Völz",
        "Jochen Hinkel"
    ],
    "abstract_s": "Economic decision analysis is an important tool for developing cost-efficient adaptation pathways...",
    "docType_s": "ART",
    "language_s": ["en"],
    "domain_s": ["0.sde"],
    "uri_s": "https://hal.science/hal-04672891v1",
    "scrapedAt": "2026-10-20T09:10:22Z",
    "doiId_s": "10.1016/j.crm.2023.100512",
    "journalTitle_s": "Climate Risk Management",
    "publicationDate_s": "2023",
    "fileMain_s": "https://hal.science/hal-04672891/document",
    "openAccess_bool": true,
    "producedDateY_i": 2023
}

Output Field Reference

Field	Type	Description
docid	String	Unique HAL document ID (duplicates are automatically filtered out).
halId_s	String	Human-readable HAL identifier (e.g. hal-04672891).
title_s	Array	Document title(s).
authFullName_s	Array	Full names of listed authors.
abstract_s	String	Document abstract, when available.
docType_s	String	HAL document type code (e.g. ART, COMM, THESE).
doiId_s	String	DOI of the document, when recorded.
journalTitle_s	String	Journal title, for journal articles.
producedDateY_i	Integer	Year of publication/production.
publicationDate_s	String	Full publication date, when available.
language_s	Array	Document language code(s) (e.g. en, fr).
domain_s	Array	HAL subject domain code(s).
openAccess_bool	Boolean	true when a full text is openly available on HAL.
uri_s	String	Direct web URI for the document.
fileMain_s	String	Direct URL to the main uploaded file, when available.
scrapedAt	String	UTC timestamp indicating when the row was saved.
error	String	Error details (only present if a specific record failed to process).
httpStatus	Integer	HTTP response status code for error rows.

Fields that HAL does not return for a given document (such as doiId_s, journalTitle_s, or fileMain_s) are omitted from that record rather than left blank.

---

How Data Collection Works

• Pagination & Limits: Requests process up to 30 documents per step, scaling automatically down to match your exact maxItems cap.
• Automatic Retries & Stability: Built-in error handling manages transient network issues or unexpected page responses gracefully without stopping the overall run.
• Deduplication: Duplicate document IDs (docid) are automatically dropped so your dataset remains clean and unique.
• Flexible Export: Download your data in JSON, CSV, or Excel formats with a single click.

---

Related Tools & Custom Development

If you plan to map, cross-reference, or join academic records with external datasets, additional data enrichment passes or CSV cleanups are standard next steps. BigiByte builds tailored data collection solutions and custom pipelines for research repositories and open-data portals worldwide.

Need custom dataset integrations, automated scheduled monitors, or specialized extraction tools? BigiByte builds custom scrapers and APIs tailored to your needs: https://bigibyte.com

---

Frequently Asked Questions

How many documents can I collect in a single run?

The default target is 100 items per run, with a maximum cap of 1,000 successful records per task.

Can I export data directly to CSV or Excel?

Yes! Once a run completes, you can export the full dataset directly to JSON, CSV, or Excel.

What happens if a query encounters an error?

If a request encounters a server error or rate limitation, the scraper retries automatically. If an issue persists, an error row is created for that item while the overall task continues running smoothly.

---

Trademark Notice & Disclaimer

Trademark Notice: HAL Open Archive and related names, logos, and trademarks belong to their respective owners. Their use in this documentation is strictly to identify the source of data and does not imply affiliation, sponsorship, endorsement, or partnership.

Disclaimer: This tool is an independent solution developed by BigiByte and is not officially affiliated with, endorsed by, or sponsored by HAL Open Archive. All data is collected from publicly accessible records. Users are responsible for adhering to the source's terms of service, rate guidelines, and licensing or attribution requirements.

---

Is Web Scraping Legal?

This tool collects publicly available research records from HAL Open Archive without bypassing access barriers. While gathering public research data is standard practice, consider the following best practices:

• Personal Data: Author names and public academic profiles may contain personal data. Ensure compliance with applicable privacy regulations (such as GDPR or CCPA) when storing or processing this information.
• Copyright & Reuse: Abstracts, articles, and full-text documents remain subject to copyright and Creative Commons licenses. Respect attribution requirements and reuse limitations specified by publishers and authors.
• Terms & Attribution: Credit HAL Open Archive and the respective authors when republishing or distributing dataset insights.
• Legal Context: This information provides general context and does not constitute legal advice.

To learn more, read Apify's guide: Is web scraping legal?