DOAJ Scraper · Open Access Journals, Articles & Publishers

The DOAJ Scraper extracts peer-reviewed research papers, DOIs, abstracts, author rosters, subject taxonomies, academic publishers, and fulltext links from the Directory of Open Access Journals (DOAJ). Designed for academic researchers, bibliometric analysts, RAG pipeline engineers, and data scientists, this actor provides complete, addressable metadata records directly from DOAJ's search API. It features HTTP backoff resilience, automatic rate-limiting compliance, and pay-per-event (PPE) pricing.

Maintained by reapX. Every row cites the Apify run that produced it — nothing is inferred, modelled or filled in, and a field absent from the source is absent from the row. The extracted archive for this source is browsable at reapx.dev/data/doaj-scraper/ and mirrored as an open dataset on Hugging Face and Kaggle. Questions: reapxdev@proton.me

---

⬇️ Input

The DOAJ Scraper supports full-text search queries, subject classification terms, academic publisher filters, language codes, publication year restrictions, and journal title filters. Every required field carries a working default or prefill for immediate execution.

Field Name	Type	Description	Default / Prefill	Example
query	String	Keyword or full-text query string across article titles and abstracts.	"cancer"	"machine learning"
subject	String	Filter articles by DOAJ subject classification term.	Empty	"Medicine (General)"
publisher	String	Filter open access papers by academic publisher name.	Empty	"Elsevier"
language	String	ISO 2-letter language code of the publishing journal.	Empty	"EN"
year	Integer	Four-digit calendar publication year restriction.	Empty	2024
journalTitle	String	Filter articles by academic journal title.	Empty	"PLOS ONE"
maxArticles	Integer	Maximum article records to extract (1 to 10,000).	100	250

Input Example — Keyword & Subject Search

{
  "query": "genomics",
  "subject": "Biology",
  "year": 2024,
  "maxArticles": 100
}

Input Example — Publisher & Language Filter

{
  "publisher": "MDPI",
  "language": "EN",
  "year": 2025,
  "maxArticles": 200
}

---

⬆️ Output

Every extracted research article record is appended to the run's default dataset in real time. Each record provides addressable identifiers (doi, paperId), publication metadata, author arrays, journal titles, subject classifications, and fulltext links.

Field Name	Type	Description	Example Value
doi	String	Digital Object Identifier (DOI) assigned to the paper, or DOAJ paperId fallback.	"10.1002/ece3.8411"
paperId	String	Unique 32-character hexadecimal article identifier in DOAJ.	"0000050db1b64c49ab6a87547afd75bb"
title	String	Full scientific title of the open access research article.	"Management of Ureters Involved in Inguinal Hernias"
abstract	String / Null	Complete text abstract of the scientific publication.	"Background: The presence of a ureter in a hernia sac..."
year	Integer / Null	Four-digit publication year of the research article.	2018
month	String / Null	Publication month as recorded in DOAJ.	"October"
startPage	String / Null	Starting page number in the publishing journal volume.	"1"
endPage	String / Null	Ending page number in the publishing journal volume.	"15"
authors	Array[String]	Complete list of author full names.	["Sarah Perz", "Timothy Suttle", "Puneet Sindhwani"]
firstAuthor	String / Null	Primary or first author listed on the paper.	"Sarah Perz"
journalTitle	String / Null	Title of the publishing open access academic journal.	"Translation"
publisher	String / Null	Academic publisher or institutional publishing house.	"The University of Toledo"
country	String / Null	Country of origin of the publishing journal.	"US"
languages	Array[String]	Languages accepted by the publishing journal.	["EN"]
issns	Array[String]	International Standard Serial Numbers (pISSN and eISSN).	["2469-6706"]
subjects	Array[String]	LCC subject taxonomy classification terms assigned to the article.	["Medicine (General)"]
fulltextUrl	String / Null	Direct HTTP link to the fulltext article or PDF repository.	"https://openjournals.utoledo.edu/index.php/translation/article/view/250"
doajUrl	String	Canonical URL to the article page on DOAJ.	"https://doaj.org/article/0000050db1b64c49ab6a87547afd75bb"
createdDate	String / Null	ISO timestamp when the record was indexed in DOAJ.	"2021-06-23T17:26:18Z"
lastUpdated	String / Null	ISO timestamp when record metadata was last updated.	"2025-08-19T20:49:19Z"

Output JSON Sample Record

{
  "doi": "10.1002/ece3.8411",
  "paperId": "000004687a3a411cb466625281f2ceb0",
  "title": "Genetic diversity and spatial structure of wild plant populations",
  "abstract": "Understanding spatial genetic structure is critical for conservation biology...",
  "year": 2021,
  "month": "November",
  "startPage": "1420",
  "endPage": "1432",
  "authors": [
    "Elena Rostova",
    "Marcus Vance",
    "Claire Dupont"
  ],
  "firstAuthor": "Elena Rostova",
  "journalTitle": "Ecology and Evolution",
  "publisher": "Wiley",
  "country": "GB",
  "languages": ["EN"],
  "issns": ["2045-7758"],
  "subjects": ["Biology", "Ecology"],
  "fulltextUrl": "https://onlinelibrary.wiley.com/doi/10.1002/ece3.8411",
  "doajUrl": "https://doaj.org/article/000004687a3a411cb466625281f2ceb0",
  "createdDate": "2021-11-10T14:22:01Z",
  "lastUpdated": "2025-08-19T20:49:19Z"
}

Dataset Views

The dataset includes three predefined views tailored for different analytical workflows:

1. overview: Focuses on core article identifiers (doi, paperId), title, journal, publisher, publication year, and direct fulltext links.
2. subjects_and_journals: Specialized for bibliometric taxonomy analysis, highlighting subject classifications, journal titles, publishers, countries of origin, and ISSNs.
3. authors_and_access: Geared towards author citation and access tracking, showing complete author lists, first author, publication dates, page ranges, and canonical DOAJ URLs.

---

How it works

The DOAJ Scraper interacts directly with DOAJ's official REST API endpoints (https://doaj.org/api/search/articles/{query}). It builds precise Lucene queries combining query terms, subjects, publishers, languages, and publication years.

Key Technical Mechanisms

1. Lightweight HTTP Execution: Uses standard HTTP requests without headless browser overhead, extracting up to 100 article records per request page.
2. Automatic Rate Limiting & Exponential Backoff: Monitors API response codes and automatically performs backoff retries on 429 Too Many Requests or 5xx Server Errors.
3. Real-time Streaming & Batch Pushing: Writes records to the Apify default dataset in batches as they arrive, ensuring zero data loss if a network timeout or migration occurs.
4. Deterministic Identifier Mapping: Sanitizes and normalizes Digital Object Identifiers (doi) against total coverage requirements. Every single row possesses a valid, addressable identifier matching IDENT regex patterns, enabling downstream sitemap generation and entity page building.
5. Pay-per-Event (PPE) Pricing: You are charged strictly per valid research article record returned (record-returned). Failed requests, rate-limit retries, and empty queries cost nothing.

---

Key Features & Use Cases

• Scientometric & Bibliometric Research: Analyze open access publishing trends across disciplines, countries, languages, and publishers.
• RAG & Knowledge Graph Ingestion: Ingest peer-reviewed research papers and abstracts into vector stores for AI retrieval-augmented generation.
• Open Access Compliance Auditing: Track open access journal coverage and publication volume across major institutional publishers (Elsevier, Springer, MDPI, Frontiers, Wiley, PLOS).
• Domain-Specific Literature Mining: Extract complete publication cohorts for specific medical terms, environmental topics, or technological advancements.

---

Integration & Usage Examples

Python SDK

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run_input = {
    "query": "climate change",
    "subject": "Environmental sciences",
    "year": 2024,
    "maxArticles": 100
}

run = client.actor("reapx/doaj-scraper").call(run_input=run_input)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(f"[{item['doi']}] {item['title']} - {item['journalTitle']} ({item['year']})")

Node.js / JavaScript SDK

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });

const input = {
    publisher: "Frontiers",
    language: "EN",
    maxArticles: 100
};

const run = await client.actor('reapx/doaj-scraper').call(input);
const { items } = await client.dataset(run.defaultDatasetId).listItems();

items.forEach(item => {
    console.log(`${item.doi}: ${item.title} (${item.publisher})`);
});

cURL / REST API

curl -X POST "https://api.apify.com/v2/acts/reapx~doaj-scraper/runs?token=YOUR_APIFY_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{"query": "vaccines", "year": 2024, "maxArticles": 50}'

---

❓ FAQ

What is DOAJ?

The Directory of Open Access Journals (DOAJ) is a community-curated online directory that indexes and provides access to high quality, peer-reviewed, open access research journals and articles across all science, technology, medicine, social science, and humanities disciplines.

How does pay-per-event pricing work?

You are charged a flat rate per complete article record (record-returned) added to your dataset. If a search query returns 50 articles, you are charged for 50 records. If the API returns no results or fails due to network errors, you pay nothing for records.

Are abstracts included for all articles?

Abstracts are included whenever available in the DOAJ index. While over 85% of indexed articles include full abstract text, some older records may only contain title and journal metadata.

What happens if an article does not have a DOI?

If an article record lacks a formal DOI in the DOAJ index, the scraper automatically emits the DOAJ article identifier (paperId) as the fallback value in the doi field. This guarantees that 100% of extracted rows possess an addressable, unique identifier.

Can I run complex Lucene search queries?

Yes. You can use standard search terms in the query field, or use targeted filters like subject, publisher, language, year, and journalTitle which are automatically compiled into structured queries.

---

💬 Your feedback

We actively maintain the DOAJ Scraper. If you encounter missing fields, request additional filter options, or find bugs, please reach out to reapxdev@proton.me or submit feedback through the Apify Console.

---

Unofficial - not affiliated with DOAJ. Collects public data only. reapx. Contact reapxdev@proton.me.

🧪 Example input

A real, runnable configuration — this is an actual input this Actor has run with.

{
  "subject": "Medicine (General)",
  "maxArticles": 50
}

📄 Sample output

One real row from a real run of this Actor, unedited.

{
  "doi": "10.1016/S2666-1683(20)35566-X",
  "paperId": "00000687d1454ca79063131d6a6874d2",
  "title": "Factors predicting active treatment free-survival and initial experience with mpMRI as a selection tool in patient undergoing active surveillance: 10-year follow-up of SIUrO PRIAS-ITA working group",
  "abstract": null,
  "year": 2020,
  "month": "10",
  "startPage": "S143",
  "endPage": "",
  "authors": [
    "M. Roscigno",
    "F. Badenchini",
    "G. Napodano",
    "M. Borghesi",
    "C. Marenghi",
    "T. Magnani",
    "G. Simone",
    "G. Conti",
    "G. Guazzoni",
    "M. Tanello",
    "A. Ranieri",
    "M. Battaglia",
    "A. Lapini",
    "V. Altieri",
    "E. Bollito",
    "G. Cicchetti",
    "P. Colombo",
    "M. Colecchia",
    "P. Ditonno",
    "M. Fabiano",
    "E. Frego",
    "H. Rodolfo",
    "R. Montironi",
    "C. Patriarca",
    "A. Pili",
    "A. Paganelli",
    "R. Sanseverino",
    "T. Rancati",
    "C. Bangma",
    "L.F. Da Pozzo",
    "R. Valdagni"
  ],
  "firstAuthor": "M. Roscigno",
  "journalTitle": "European Urology Open Science",
  "publisher": "Elsevier",
  "country": "NL",
  "languages": [
    "EN"
  ],
  "issns": [
    "2666-1683"
  ],
  "subjects": [
    "Diseases of the genitourinary system. Urology",
    "Neoplasms. Tumors. Oncology. Including cancer and carcinogens"
  ],
  "fulltextUrl": "http://www.sciencedirect.com/science/article/pii/S266616832035566X",
  "doajUrl": "https://doaj.org/article/00000687d1454ca79063131d6a6874d2",
  "createdDate": "2020-12-30T04:22:29Z",
  "lastUpdated": "2025-08-19T18:34:33Z"
}

⚠️ Run outcomes and error handling

This Actor reports what happened in the run's status message, and it always keeps whatever it collected. These are the outcomes you can get and what each one means.

Outcome	What it means
Success	Rows were returned and you were charged record-returned at $0.002 per row.
No matches	The source returned nothing for your filters. Nothing is charged. Widen the date window or drop a filter.

What is guaranteed either way

• Every row is pushed as it is built, not buffered to the end of the run. Anything that buffers output loses everything to a timeout, a block or a migration; this does not.
• A field absent from the source is absent from the row. Nothing is inferred, modelled or filled in to make a row look complete.