# DOAJ Scraper · Open Access Journals, Articles & Publishers (`reapx/doaj-scraper`) Actor

Scrape open access research articles, DOIs, authors, subjects, publishers, and abstracts from the Directory of Open Access Journals (DOAJ) API. Features pay-per-event pricing and automatic backoff.

- **URL**: https://apify.com/reapx/doaj-scraper.md
- **Developed by:** [Tarek Etman](https://apify.com/reapx) (community)
- **Categories:** Developer tools, Business, Other
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.30 / 1,000 record returneds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

![reapX — public sources in, addressable records out](https://reapx.dev/reapx.gif)

## DOAJ Scraper · Open Access Journals, Articles & Publishers

The **DOAJ Scraper** extracts peer-reviewed research papers, DOIs, abstracts, author rosters, subject taxonomies, academic publishers, and fulltext links from the Directory of Open Access Journals (DOAJ). Designed for academic researchers, bibliometric analysts, RAG pipeline engineers, and data scientists, this actor provides complete, addressable metadata records directly from DOAJ's search API. It features HTTP backoff resilience, automatic rate-limiting compliance, and pay-per-event (PPE) pricing.

> Maintained by **reapX**. Every row cites the Apify run that produced it — nothing is inferred, modelled or filled in, and a field absent from the source is absent from the row. The extracted archive for this source is browsable at [reapx.dev/data/doaj-scraper/](https://reapx.dev/data/doaj-scraper/) and mirrored as an open dataset on [Hugging Face](https://huggingface.co/datasets/reapxdev/doaj-scraper) and [Kaggle](https://www.kaggle.com/datasets/reapxdev/doaj-scraper). Questions: reapxdev@proton.me

***

### ⬇️ Input

The **DOAJ Scraper** supports full-text search queries, subject classification terms, academic publisher filters, language codes, publication year restrictions, and journal title filters. Every required field carries a working default or prefill for immediate execution.

| Field Name | Type | Description | Default / Prefill | Example |
| :--- | :--- | :--- | :--- | :--- |
| `query` | String | Keyword or full-text query string across article titles and abstracts. | `"cancer"` | `"machine learning"` |
| `subject` | String | Filter articles by DOAJ subject classification term. | *Empty* | `"Medicine (General)"` |
| `publisher` | String | Filter open access papers by academic publisher name. | *Empty* | `"Elsevier"` |
| `language` | String | ISO 2-letter language code of the publishing journal. | *Empty* | `"EN"` |
| `year` | Integer | Four-digit calendar publication year restriction. | *Empty* | `2024` |
| `journalTitle` | String | Filter articles by academic journal title. | *Empty* | `"PLOS ONE"` |
| `maxArticles` | Integer | Maximum article records to extract (1 to 10,000). | `100` | `250` |

#### Input Example — Keyword & Subject Search

```json
{
  "query": "genomics",
  "subject": "Biology",
  "year": 2024,
  "maxArticles": 100
}
```

#### Input Example — Publisher & Language Filter

```json
{
  "publisher": "MDPI",
  "language": "EN",
  "year": 2025,
  "maxArticles": 200
}
```

***

### ⬆️ Output

Every extracted research article record is appended to the run's default dataset in real time. Each record provides addressable identifiers (`doi`, `paperId`), publication metadata, author arrays, journal titles, subject classifications, and fulltext links.

| Field Name | Type | Description | Example Value |
| :--- | :--- | :--- | :--- |
| `doi` | String | Digital Object Identifier (DOI) assigned to the paper, or DOAJ paperId fallback. | `"10.1002/ece3.8411"` |
| `paperId` | String | Unique 32-character hexadecimal article identifier in DOAJ. | `"0000050db1b64c49ab6a87547afd75bb"` |
| `title` | String | Full scientific title of the open access research article. | `"Management of Ureters Involved in Inguinal Hernias"` |
| `abstract` | String / Null | Complete text abstract of the scientific publication. | `"Background: The presence of a ureter in a hernia sac..."` |
| `year` | Integer / Null | Four-digit publication year of the research article. | `2018` |
| `month` | String / Null | Publication month as recorded in DOAJ. | `"October"` |
| `startPage` | String / Null | Starting page number in the publishing journal volume. | `"1"` |
| `endPage` | String / Null | Ending page number in the publishing journal volume. | `"15"` |
| `authors` | Array\[String] | Complete list of author full names. | `["Sarah Perz", "Timothy Suttle", "Puneet Sindhwani"]` |
| `firstAuthor` | String / Null | Primary or first author listed on the paper. | `"Sarah Perz"` |
| `journalTitle` | String / Null | Title of the publishing open access academic journal. | `"Translation"` |
| `publisher` | String / Null | Academic publisher or institutional publishing house. | `"The University of Toledo"` |
| `country` | String / Null | Country of origin of the publishing journal. | `"US"` |
| `languages` | Array\[String] | Languages accepted by the publishing journal. | `["EN"]` |
| `issns` | Array\[String] | International Standard Serial Numbers (pISSN and eISSN). | `["2469-6706"]` |
| `subjects` | Array\[String] | LCC subject taxonomy classification terms assigned to the article. | `["Medicine (General)"]` |
| `fulltextUrl` | String / Null | Direct HTTP link to the fulltext article or PDF repository. | `"https://openjournals.utoledo.edu/index.php/translation/article/view/250"` |
| `doajUrl` | String | Canonical URL to the article page on DOAJ. | `"https://doaj.org/article/0000050db1b64c49ab6a87547afd75bb"` |
| `createdDate` | String / Null | ISO timestamp when the record was indexed in DOAJ. | `"2021-06-23T17:26:18Z"` |
| `lastUpdated` | String / Null | ISO timestamp when record metadata was last updated. | `"2025-08-19T20:49:19Z"` |

#### Output JSON Sample Record

```json
{
  "doi": "10.1002/ece3.8411",
  "paperId": "000004687a3a411cb466625281f2ceb0",
  "title": "Genetic diversity and spatial structure of wild plant populations",
  "abstract": "Understanding spatial genetic structure is critical for conservation biology...",
  "year": 2021,
  "month": "November",
  "startPage": "1420",
  "endPage": "1432",
  "authors": [
    "Elena Rostova",
    "Marcus Vance",
    "Claire Dupont"
  ],
  "firstAuthor": "Elena Rostova",
  "journalTitle": "Ecology and Evolution",
  "publisher": "Wiley",
  "country": "GB",
  "languages": ["EN"],
  "issns": ["2045-7758"],
  "subjects": ["Biology", "Ecology"],
  "fulltextUrl": "https://onlinelibrary.wiley.com/doi/10.1002/ece3.8411",
  "doajUrl": "https://doaj.org/article/000004687a3a411cb466625281f2ceb0",
  "createdDate": "2021-11-10T14:22:01Z",
  "lastUpdated": "2025-08-19T20:49:19Z"
}
```

#### Dataset Views

The dataset includes three predefined views tailored for different analytical workflows:

1. **`overview`**: Focuses on core article identifiers (`doi`, `paperId`), title, journal, publisher, publication year, and direct fulltext links.
2. **`subjects_and_journals`**: Specialized for bibliometric taxonomy analysis, highlighting subject classifications, journal titles, publishers, countries of origin, and ISSNs.
3. **`authors_and_access`**: Geared towards author citation and access tracking, showing complete author lists, first author, publication dates, page ranges, and canonical DOAJ URLs.

***

### How it works

The **DOAJ Scraper** interacts directly with DOAJ's official REST API endpoints (`https://doaj.org/api/search/articles/{query}`). It builds precise Lucene queries combining query terms, subjects, publishers, languages, and publication years.

#### Key Technical Mechanisms

1. **Lightweight HTTP Execution**: Uses standard HTTP requests without headless browser overhead, extracting up to 100 article records per request page.
2. **Automatic Rate Limiting & Exponential Backoff**: Monitors API response codes and automatically performs backoff retries on `429 Too Many Requests` or `5xx Server Errors`.
3. **Real-time Streaming & Batch Pushing**: Writes records to the Apify default dataset in batches as they arrive, ensuring zero data loss if a network timeout or migration occurs.
4. **Deterministic Identifier Mapping**: Sanitizes and normalizes Digital Object Identifiers (`doi`) against total coverage requirements. Every single row possesses a valid, addressable identifier matching `IDENT` regex patterns, enabling downstream sitemap generation and entity page building.
5. **Pay-per-Event (PPE) Pricing**: You are charged strictly per valid research article record returned (`record-returned`). Failed requests, rate-limit retries, and empty queries cost nothing.

***

### Key Features & Use Cases

- **Scientometric & Bibliometric Research**: Analyze open access publishing trends across disciplines, countries, languages, and publishers.
- **RAG & Knowledge Graph Ingestion**: Ingest peer-reviewed research papers and abstracts into vector stores for AI retrieval-augmented generation.
- **Open Access Compliance Auditing**: Track open access journal coverage and publication volume across major institutional publishers (Elsevier, Springer, MDPI, Frontiers, Wiley, PLOS).
- **Domain-Specific Literature Mining**: Extract complete publication cohorts for specific medical terms, environmental topics, or technological advancements.

***

### Integration & Usage Examples

#### Python SDK

```python
from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run_input = {
    "query": "climate change",
    "subject": "Environmental sciences",
    "year": 2024,
    "maxArticles": 100
}

run = client.actor("reapx/doaj-scraper").call(run_input=run_input)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(f"[{item['doi']}] {item['title']} - {item['journalTitle']} ({item['year']})")
```

#### Node.js / JavaScript SDK

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });

const input = {
    publisher: "Frontiers",
    language: "EN",
    maxArticles: 100
};

const run = await client.actor('reapx/doaj-scraper').call(input);
const { items } = await client.dataset(run.defaultDatasetId).listItems();

items.forEach(item => {
    console.log(`${item.doi}: ${item.title} (${item.publisher})`);
});
```

#### cURL / REST API

```bash
curl -X POST "https://api.apify.com/v2/acts/reapx~doaj-scraper/runs?token=YOUR_APIFY_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{"query": "vaccines", "year": 2024, "maxArticles": 50}'
```

***

### ❓ FAQ

#### What is DOAJ?

The Directory of Open Access Journals (DOAJ) is a community-curated online directory that indexes and provides access to high quality, peer-reviewed, open access research journals and articles across all science, technology, medicine, social science, and humanities disciplines.

#### How does pay-per-event pricing work?

You are charged a flat rate per complete article record (`record-returned`) added to your dataset. If a search query returns 50 articles, you are charged for 50 records. If the API returns no results or fails due to network errors, you pay nothing for records.

#### Are abstracts included for all articles?

Abstracts are included whenever available in the DOAJ index. While over 85% of indexed articles include full abstract text, some older records may only contain title and journal metadata.

#### What happens if an article does not have a DOI?

If an article record lacks a formal DOI in the DOAJ index, the scraper automatically emits the DOAJ article identifier (`paperId`) as the fallback value in the `doi` field. This guarantees that 100% of extracted rows possess an addressable, unique identifier.

#### Can I run complex Lucene search queries?

Yes. You can use standard search terms in the `query` field, or use targeted filters like `subject`, `publisher`, `language`, `year`, and `journalTitle` which are automatically compiled into structured queries.

***

### 💬 Your feedback

We actively maintain the **DOAJ Scraper**. If you encounter missing fields, request additional filter options, or find bugs, please reach out to `reapxdev@proton.me` or submit feedback through the Apify Console.

***

*Unofficial - not affiliated with DOAJ. Collects public data only. reapx. Contact reapxdev@proton.me.*

### 🧪 Example input

A real, runnable configuration — this is an actual input this Actor has run with.

```json
{
  "subject": "Medicine (General)",
  "maxArticles": 50
}
```

### 📄 Sample output

One real row from a real run of this Actor, unedited.

```json
{
  "doi": "10.1016/S2666-1683(20)35566-X",
  "paperId": "00000687d1454ca79063131d6a6874d2",
  "title": "Factors predicting active treatment free-survival and initial experience with mpMRI as a selection tool in patient undergoing active surveillance: 10-year follow-up of SIUrO PRIAS-ITA working group",
  "abstract": null,
  "year": 2020,
  "month": "10",
  "startPage": "S143",
  "endPage": "",
  "authors": [
    "M. Roscigno",
    "F. Badenchini",
    "G. Napodano",
    "M. Borghesi",
    "C. Marenghi",
    "T. Magnani",
    "G. Simone",
    "G. Conti",
    "G. Guazzoni",
    "M. Tanello",
    "A. Ranieri",
    "M. Battaglia",
    "A. Lapini",
    "V. Altieri",
    "E. Bollito",
    "G. Cicchetti",
    "P. Colombo",
    "M. Colecchia",
    "P. Ditonno",
    "M. Fabiano",
    "E. Frego",
    "H. Rodolfo",
    "R. Montironi",
    "C. Patriarca",
    "A. Pili",
    "A. Paganelli",
    "R. Sanseverino",
    "T. Rancati",
    "C. Bangma",
    "L.F. Da Pozzo",
    "R. Valdagni"
  ],
  "firstAuthor": "M. Roscigno",
  "journalTitle": "European Urology Open Science",
  "publisher": "Elsevier",
  "country": "NL",
  "languages": [
    "EN"
  ],
  "issns": [
    "2666-1683"
  ],
  "subjects": [
    "Diseases of the genitourinary system. Urology",
    "Neoplasms. Tumors. Oncology. Including cancer and carcinogens"
  ],
  "fulltextUrl": "http://www.sciencedirect.com/science/article/pii/S266616832035566X",
  "doajUrl": "https://doaj.org/article/00000687d1454ca79063131d6a6874d2",
  "createdDate": "2020-12-30T04:22:29Z",
  "lastUpdated": "2025-08-19T18:34:33Z"
}
```

### ⚠️ Run outcomes and error handling

This Actor reports what happened in the run's **status message**, and it always keeps whatever it collected. These are the outcomes you can get and what each one means.

| Outcome | What it means |
|---|---|
| **Success** | Rows were returned and you were charged `record-returned` at $0.002 per row. |
| **No matches** | The source returned nothing for your filters. **Nothing is charged.** Widen the date window or drop a filter. |

#### What is guaranteed either way

- **Every row is pushed as it is built**, not buffered to the end of the run. Anything that buffers output loses everything to a timeout, a block or a migration; this does not.
- **A field absent from the source is absent from the row.** Nothing is inferred, modelled or filled in to make a row look complete.

# Actor input Schema

## `query` (type: `string`):

Full-text search term or query string to search across DOAJ articles (e.g. <code>cancer</code>, <code>"machine learning"</code>, or <code>genomics</code>). Leave empty if filtering purely by subject, publisher, language, or year.<br><br>The more specific the search query, the faster the extraction completes and the lower the overall cost.

## `subject` (type: `string`):

Filter DOAJ articles by subject taxonomy classification (e.g. <code>Medicine (General)</code>, <code>Biology</code>, <code>Education</code>, or <code>Technology</code>). Leave this field empty to extract articles across all academic subjects.<br><br>Restricting by subject narrows results to a specific scientific discipline.

## `publisher` (type: `string`):

Filter open access articles by publishing house or academic institution (e.g. <code>Elsevier</code>, <code>Springer</code>, <code>MDPI</code>, <code>Frontiers</code>, or <code>PLOS</code>). Leave empty to include all publishers.<br><br>Specifying a publisher limits extraction to journals published by that house.

## `language` (type: `string`):

Filter publications by ISO two-letter language code (e.g. <code>EN</code> for English, <code>ES</code> for Spanish, <code>FR</code> for French, <code>DE</code> for German, <code>PT</code> for Portuguese). Leave empty for all languages.<br><br>Filtering by language focuses extraction on specific language editions.

## `year` (type: `integer`):

Filter publications by four-digit calendar publication year (e.g. <code>2024</code> or <code>2025</code>). Leave empty to query across all available historical publication years.<br><br>Setting a publication year restricts results to articles published in that single year.

## `journalTitle` (type: `string`):

Filter articles by exact or partial academic journal title (e.g. <code>PLOS ONE</code> or <code>Scientific Reports</code>). Leave empty to extract across all open access journals.<br><br>Filtering by journal isolates papers published in specific open access periodicals.

## `maxArticles` (type: `integer`):

Maximum number of DOAJ research article records to extract and save into the default dataset (1 to 10000). Prefilled with 100.<br><br>The higher the number, the longer the run takes and the more it costs under pay-per-event pricing.<br><br>The higher the number, the longer the run takes and the more you are charged, because you pay per row returned.

## Actor input object example

```json
{
  "query": "cancer",
  "maxArticles": 100
}
```

# Actor output Schema

## `results` (type: `string`):

Every DOAJ article record extracted by this run, one row per article, in the default dataset.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "query": "cancer",
    "maxArticles": 100
};

// Run the Actor and wait for it to finish
const run = await client.actor("reapx/doaj-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "query": "cancer",
    "maxArticles": 100,
}

# Run the Actor and wait for it to finish
run = client.actor("reapx/doaj-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "query": "cancer",
  "maxArticles": 100
}' |
apify call reapx/doaj-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,reapx/doaj-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/FpXYgGgn2lxUuvaA0/builds/W1nFc0uuOAmY0xi7C/openapi.json