# novaResearch : Deep Scholar Gap Finder with AI insights (`abhish1ek/nova-research-deep-scholar`) Actor

A Python-based scholar research tool which helps in identifying potential gaps post deep research & analysis within existing released research papers.From scholarly sources, rank papers, analyze trends, & identify evidence-backed research gaps.
-Research trends
-Identified Gaps & future directions\_

- **URL**: https://apify.com/abhish1ek/nova-research-deep-scholar.md
- **Developed by:** [Abhishek Tomar](https://apify.com/abhish1ek) (community)
- **Categories:** Automation, Developer tools, Agents
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $10.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## novaResearch - Deep Scholar Gap Finder with AI insights

AI-assisted research gap finder and literature review from arXiv, Crossref, and OpenAlex.

novaResearch Scholar is an Apify Actor that automates the first 80% of a literature review for a given research topic by aggregating and analyzing papers from multiple scholarly metadata sources (arXiv, Crossref, and OpenAlex). It is designed for researchers, data scientists, and practitioners who want a structured, machine-readable overview of a topic instead of manual keyword searches.

### What this Actor does

- Searches trusted scholarly sources (arXiv, Crossref, OpenAlex) for a given query.
- Normalizes metadata into a unified schema across all sources (titles, authors, DOIs, publication venues, dates, etc.).
- Deduplicates papers using DOI, arXiv ID, and fuzzy title matching.
- Ranks papers by relevance using a composite score (semantic overlap, recency, metadata completeness, source confidence).
- Produces a curated dataset of relevant papers plus a high-level analysis report (categories, trends, gaps) suitable for downstream AI workflows or dashboards.

This Actor focuses on **metadata and structured insights**, not full-text PDF scraping.

- "Generates a structured REPORT.json highlighting research gaps, trends, and clusters to guide further deep reading and research."

### Typical use cases

- Quickly mapping out the key papers and venues around a new research topic.
- Building a seed corpus for downstream LLM/RAG pipelines.
- Tracking how a topic evolves over time (e.g., new methods or datasets).
- Preparing summaries, slide decks, or internal reports backed by a structured paper list.

### Input

The input is configured via the Apify input UI, generated from the input schema.

Key fields:

- **query** (string, required): Main research topic or keyword, e.g. `"graph neural networks for recommender systems"`.
- **sources** (array): Which sources to include. Options: `["arxiv", "crossref", "openalex"]`. Default: all three.
- **maxPapers** (integer, 1-300): Maximum number of papers to retrieve before normalization, deduplication, and ranking. Default: 25.
- **maxPapersForLLM** (integer, 5-100): How many top-ranked papers to send into LLM-based gap synthesis. Lower values reduce cost and latency. Default: 20.
- **analysisDepth** (string): Controls how deeply the actor synthesizes trends and gaps. Options: `light`, `standard`, `deep`. Default: `standard`.
- **includeTrendAnalysis** (boolean): Whether to extract trend signals such as methods, benchmarks, institutions, and evaluation patterns. Default: true.
- **detectResearchGaps** (boolean): Whether to run heuristic and LLM-backed research gap synthesis. Default: true.
- **includePaperIdeas** (boolean): Whether to include possible future paper ideas in the output report, derived from identified research gaps. Default: false.
- **sortPreference** (string): Ranking preference for final paper ordering. Options: `balanced`, `relevance`, `recency`. Default: `balanced`.
- **dateFrom** (string, optional): Lower publication date bound in `YYYY-MM-DD` format.
- **dateTo** (string, optional): Upper publication date bound in `YYYY-MM-DD` format.
- **llmProvider** (string): Provider for structured gap synthesis. Currently only `nvidia` is supported.
- **llmApiKey** (string, optional): API key override for this run. If empty, uses the `NVIDIA_API_KEY` environment variable.
- **llmModel** (string): Model ID used for gap synthesis. Default: `nvidia/nemotron-3-super-120b-a12b`.

See `.actor/input_schema.json` for the authoritative schema definition.

#### Input normalization

The actor accepts several legacy and alternative field names for backward compatibility:

- `topic` is automatically mapped to `query`
- Snake-case variants (e.g., `max_papers`, `analysis_depth`, `llm_provider`, `llm_model`, `llm_api_key`) are automatically converted to camelCase
- `enableLLM` is mapped to `detectResearchGaps`

### Output

The Actor writes:

- A **dataset** of normalized paper items (one item per paper).
- A **REPORT** object in the key-value store summarizing the run (basic statistics and high-level analysis).
- An **OUTPUT** object with pointers to the dataset and report.
- A **RUN\_STATUS** object tracking run success/failure, timing, warnings, and error details.

Each paper item typically includes fields like:

- `paper_id`
- `title`
- `authors`
- `institutions`
- `doi`
- `source` (e.g. `"arxiv"`, `"crossref"`, `"openalex"`)
- `year`
- `published_at`
- `venue`
- `url`
- `pdf_url`
- `categories`
- `keywords`
- `methods`
- `benchmarks`
- `application_areas`
- `abstract`
- `citation_count`
- `relevance_score`
- `metadata_completeness`

You can consume the dataset via the Apify UI, API, or export it to CSV/JSON for further processing.

### How to use the Actor

1. Open the Actor on Apify.
2. In the input form, fill in:
   - **query** with your research topic.
   - Optional filters like date range, max results, and sources.
3. Click **Start** to run.
4. After the run finishes, open:
   - The **Dataset** tab to inspect or export the list of papers.
   - The **Key-value store** to view the `REPORT`, `OUTPUT`, and `RUN_STATUS` entries.
5. Use the exported data in your own notebooks, LLM pipelines, or dashboards.

For most exploratory use cases, start with a modest `maxPapers` value (e.g., 15-50) and refine the query before scaling up.

#### Report structure (REPORT.json)

The actor writes a structured report to `storage/key_value_stores/default/REPORT` that helps users quickly spot research gaps and the evidence behind them.

The report includes:

- **central\_research\_finding**: A single, explicit headline conclusion synthesized from the top-ranked gaps, including:
  - statement, supporting gap IDs, confidence (high/moderate/low), and whether full-text verification is required.

- **research\_gaps**: A dedicated list of identified gaps, each with:
  - gap title, description, gap type
  - why it matters
  - evidence summary
  - supporting paper IDs and details
  - signal strength, confidence basis, confidence rationale
  - suggested next research directions
  - **evidence\_quality**: a per-gap breakdown of supporting papers by publication status (peer-reviewed / preprint), full-text availability, metadata-only records, and direct experimental evidence — so a gap supported only by surveys is never presented as equivalent to one backed by empirical tests.

- **corpus\_quality** (inside the validation assessment): relevance, source balance, metadata completeness, full-text coverage, and publication-status confidence for the analyzed corpus.

- **paper\_ideas**: Possible future paper ideas derived from research gaps (when `includePaperIdeas` is enabled). Each includes:
  - proposed paper title
  - research question and motivation
  - suggested methodology
  - key references

- **human\_facing\_summary**: A narrative summary with:
  - key trends
  - concrete gaps
  - promising research directions

- **subtopic\_clusters** and **cluster\_evidence\_quality**:
  - Groups the literature into subthemes.
  - Provides a sense of how strong the evidence is in each cluster.

- **paper\_level\_tags**:
  - Per-paper tags indicating which task/method cluster a paper belongs to.
  - Includes reproducibility signals (code/dataset/artifact availability with 3-state tracking: available/not\_found/unknown).
  - Includes 12-dimension security research analysis.

- **trend analysis signals**:
  - Common methods, benchmarks, tasks, domain application areas
  - Frequent authors and institutions
  - Year and source distribution
  - Evaluation maturity (retrospective, prospective, real-world, ablation, external validation, human evaluation, calibration)
  - Reproducibility signals and maturity assessment
  - Safety/governance, interoperability, deployment signals
  - Institution and benchmark concentration analysis

- **RUN\_STATUS** (separate key-value store entry):
  - `ok`: boolean indicating run success
  - `warnings`: list of non-fatal issues encountered
  - `meta`: timing info, topic, analysis depth
  - `error`: details if the run failed (name, message, phase, stack trace)

#### How analysis depth works

The `analysisDepth` setting controls how many papers are used for trend analysis and how detailed the output is:

- **light**: Uses the top 1/3 papers (min 5) for trend analysis. Paper ideas (if enabled) are limited to 2 with brief detail.
- **standard** (default): Uses the top 1/2 papers (min 10) for trend analysis. Paper ideas limited to 5 with standard detail.
- **deep**: Uses all ranked papers for comprehensive analysis. Paper ideas limited to 8 with detailed suggestions.

#### How reliable is this analysis?

The report is intended as a **first-pass research synthesis**, not a replacement for a full manual literature review. The gaps and trends are inferred from metadata and heuristic signals combined with LLM analysis, so they should be read as a structured, evidence-guided overview rather than a final academic verdict.

### Scoring and ranking

Papers are ranked using a composite score:

- **50% semantic relevance**: Token overlap between the query and paper title, abstract, categories, keywords, and venue.
- **20% recency**: Exponential decay with a 3-year half-life (newer papers score higher).
- **15% metadata completeness**: Based on presence of title, abstract, authors, and PDF link.
- **15% source confidence**: arXiv (0.95), OpenAlex (0.90), Crossref (0.80).

Source confidence is intentionally **primary-research-forward**: arXiv and peer-reviewed OpenAlex records outrank Crossref, which also surfaces SSRN preprints, magazine pieces, and survey-type publications.

Survey-type publications (titles/keywords containing "survey", "systematic review", "literature review", "scoping review", or "state of the art") are **demoted by a 0.9× factor** so they cannot crowd empirical research out of the top ranks — they remain in the corpus and still contribute to gap analysis, but they rank below direct research.

A domain gate filters papers by matching query topic anchor terms against the paper's evidence text.

### Deduplication

Papers are deduplicated in three phases:

1. **DOI matching**: Papers with identical DOIs are merged.
2. **arXiv ID matching**: Remaining papers with identical arXiv IDs are merged.
3. **Fuzzy title matching**: Remaining papers with title similarity >= 0.95 (using SequenceMatcher), same publication year, and same lead author are merged.

When papers are merged, the version with the highest metadata completeness is preferred, and lists (authors, keywords, methods, etc.) are combined.

### LLM-backed gap synthesis

Research gaps are identified using a combination of heuristic signal extraction and LLM synthesis:

- **Heuristic layer**: Identifies gap candidates from trend signals (underexplored domains, methodological blind spots, benchmark gaps, data gaps, reproducibility gaps, etc.). This layer works independently and does not require an LLM.

- **LLM layer**: Sends the top-ranked papers and corpus summary to NVIDIA Nemotron 3 Super 120B A12B with a structured JSON schema.

- **18 gap types** are supported, including: underexplored domain, methodological blind spot, benchmark gap, data gap, contradictory findings, reproducibility gap, deployment gap, population bias, safety/governance gap, interoperability gap, and more.

- **Fallback behavior**: When the LLM is unavailable (no API key) or returns empty/invalid results, the actor automatically falls back to heuristic-only gap detection. This ensures gap analysis always produces output.

- **Dynamic prompt sizing**: Abstract truncation scales with the number of papers sent to the LLM (1200 chars for ≤10 papers, 800 for ≤25, 500 for ≤50, 300 for >50) to stay within context limits.

- **LLM results are cached** by content hash to avoid redundant API calls on identical inputs.

- **Query-driven gap synthesis**: Gap analysis is driven by the user's query. Agent-security validation gaps (agent authorization, tool safety, policy enforcement, formal assurance, inter-agent trust, etc.) are only generated when the query actually targets that domain (e.g. "agentic AI security", "LLM agent", "multi-agent", "cybersecurity"). For any other topic — such as DNA data storage or healthcare machine learning — the report emits corpus-wide methodological gaps (external validity, translation, reproducibility, label quality, longitudinal evidence, human factors) instead of fabricating security gaps, and the agent-security assessment reports `not_applicable_query_is_not_agent_security`.

### Example

Input (`query` = `"Storing digital data in synthetic DNA strands and using retrobiosynthesis"`):

```json
{
  "query": "Storing digital data in synthetic DNA strands and using retrobiosynthesis",
  "sources": ["arxiv", "crossref", "openalex"],
  "maxPapers": 25,
  "analysisDepth": "standard",
  "detectResearchGaps": true,
  "maxPapersForLLM": 20
}
```

Key output:

```json
{
  "central_research_finding": {
    "statement": "Across 60 analyzed papers, the evidence points to clear weaknesses in how the field validates and communicates its results: ...",
    "confidence": "moderate",
    "requires_full_text_verification": true
  },
  "research_gaps": [
    {
      "title": "There is limited evidence of real-world translation or adoption",
      "gap_type": "translation_gap",
      "evidence_quality": {
        "supporting_paper_count": 4,
        "preprint_count": 4,
        "full_text_available_count": 4,
        "direct_experimental_evidence_count": 1
      }
    }
  ],
  "corpus_quality": {
    "relevance": "moderate",
    "source_balance": "high",
    "metadata_completeness": "high",
    "full_text_coverage": "partial"
  },
  "agent_security_validation_assessment": {
    "overall_signal": "not_applicable_query_is_not_agent_security"
  }
}
```

For an agent-security query (e.g. `"frameworks and methods to create security in AI agentic systems"`), the same run instead produces agent-security gaps backed by security-dimension coverage (identity, authorization, tool security, policy enforcement, formal assurance, inter-agent trust, ...) with raw and weighted coverage ratios, a populated validation assessment, and an agent-security central finding.

### Performance, limits, and pricing

- The Actor uses HTTP APIs (no browser automation) to keep runs relatively fast and cost-efficient.
- Runtime and platform costs scale roughly with the number of results and sources enabled.
- To minimize compute units:
  - Narrow your query and date range.
  - Limit `maxPapers` when experimenting.
  - Reduce `maxPapersForLLM` to send fewer papers to the LLM.
  - Disable sources you do not currently need.
  - Use `analysisDepth: "light"` for faster, cheaper runs.

### Troubleshooting and tips

- If you get fewer papers than expected, try:
  - Broadening the query.
  - Expanding the date range or enabling more sources.
- If a particular source fails (e.g., temporary API issues), the actor continues with the remaining sources.
- Check `RUN_STATUS` for a summary of what happened during the run (counts per source, warnings, errors).
- If LLM synthesis is disabled (no API key configured), the actor falls back to heuristic gap detection only.

If you encounter issues or have ideas for improvements (extra sources, custom scoring, deeper analysis), feel free to leave a comment on the Actor page.

# Actor input Schema

## `query` (type: `string`):

Main topic to search and analyze, for example: multimodal learning healthcare

## `sources` (type: `array`):

Academic sources to query for papers.

## `maxPapers` (type: `integer`):

Maximum number of papers to retrieve before normalization, deduplication, and ranking.

## `analysisDepth` (type: `string`):

Controls how deeply the actor synthesizes trends and gaps.

## `includeTrendAnalysis` (type: `boolean`):

Whether to extract trend signals such as methods, benchmarks, institutions, and evaluation patterns.

## `detectResearchGaps` (type: `boolean`):

Whether to run heuristic and LLM-backed research gap synthesis.

## `includePaperIdeas` (type: `boolean`):

Whether to include possible future paper ideas in the output report, derived from identified research gaps.

## `maxPapersForLLM` (type: `integer`):

How many top-ranked papers to send into LLM-based gap synthesis. Lower values reduce cost and latency.

## `sortPreference` (type: `string`):

Ranking preference for final paper ordering.

## `dateFrom` (type: `string`):

Optional lower publication date bound in YYYY-MM-DD format.

## `dateTo` (type: `string`):

Optional upper publication date bound in YYYY-MM-DD format.

## `llmProvider` (type: `string`):

Provider used for structured gap synthesis. Use nvidia for ChatNVIDIA via langchain-nvidia-ai-endpoints.

## `llmApiKey` (type: `string`):

Optional provider API key override for this run. If left empty, the actor will use the environment variable, such as NVIDIA\_API\_KEY.

## `llmModel` (type: `string`):

Model ID used for gap synthesis. Default is NVIDIA Nemotron 3 Super 120B A12B.

## `llmTimeoutSeconds` (type: `number`):

Maximum time to wait for an LLM response before falling back to heuristic gap detection. Lower values make the actor faster but may reduce gap quality.

## Actor input object example

```json
{
  "query": "Climate change adaptation strategies",
  "sources": [
    "arxiv",
    "crossref",
    "openalex"
  ],
  "maxPapers": 15,
  "analysisDepth": "standard",
  "includeTrendAnalysis": true,
  "detectResearchGaps": true,
  "includePaperIdeas": true,
  "maxPapersForLLM": 20,
  "sortPreference": "balanced",
  "llmProvider": "nvidia",
  "llmModel": "nvidia/nemotron-3-super-120b-a12b",
  "llmTimeoutSeconds": 60
}
```

# Actor output Schema

## `papersDataset` (type: `string`):

Dataset containing normalized, deduplicated, ranked paper records collected by the Actor.

## `reportRecord` (type: `string`):

Structured report containing trend signals, evaluation and reproducibility evidence, limitations, and research gaps.

## `outputRecord` (type: `string`):

Top-level Actor output object for API consumers, including summary report fields and research gaps.

## `statusRecord` (type: `string`):

Run status record with success/failure, warnings, timing (startedAt/finishedAt), and error details.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "query": "Climate change adaptation strategies"
};

// Run the Actor and wait for it to finish
const run = await client.actor("abhish1ek/nova-research-deep-scholar").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "query": "Climate change adaptation strategies" }

# Run the Actor and wait for it to finish
run = client.actor("abhish1ek/nova-research-deep-scholar").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "query": "Climate change adaptation strategies"
}' |
apify call abhish1ek/nova-research-deep-scholar --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,abhish1ek/nova-research-deep-scholar"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Mu6ThWT2zEfog44th/builds/2bhzDqD5reOLmEdCP/openapi.json