GEO Dataset Finder

What does GEO Dataset Finder do?

GEO Dataset Finder searches the public NCBI Gene Expression Omnibus (GEO) and saves one flat metadata row per record to an Apify dataset. It calls NCBI E-utilities (esearch and esummary on the gds database) and, when GEO serves it, the official GEO accession display in brief SOFT text. It does not log in, scrape private pages, or download raw CEL, FASTQ, MTX, or other supplementary files.

A keyword such as a disease, tissue, or assay returns GEO Series records (GSE). Each row includes the series accession, the curated GDS id when GEO assigned one, the public title and summary, organism, platform, sample count, PubMed ids, release date, and a link to the GEO accession page. Pass GSE, GSM, GPL, or GDS accessions when you already know the ids you want.

Why use GEO Dataset Finder?

GEO is the main public archive for gene expression and functional genomics studies. Finding a relevant series usually means repeating the same search on the website and copying accessions by hand. This Actor turns that search into a dataset you can export, schedule, and connect to the rest of Apify.

• Search by free text, with an optional organism qualifier such as Homo sapiens or Mus musculus.
• Look up specific GSE, GSM, GPL, and GDS accessions without expanding a series into every related sample.
• Keep NCBI's usage policy: 3 requests per second without an API key, and 10 requests per second when you supply ncbiApiKey. HTTP 429 and 5xx responses are retried with backoff.
• Store only metadata and accession links. Summaries and overall-design text are truncated when they are long.
• Run on a schedule, call the Actor from the Apify API, or download the dataset as JSON, CSV, Excel, or HTML. No proxy is required because the endpoints are public.

What data can GEO Dataset Finder extract?

Field	Description
accession	GEO accession, usually a Series such as GSE12345
gdsId	Curated DataSet id when GEO linked exactly one
title	Public title
summary	Public summary (long text is truncated)
organism	Organism reported by GEO
overallDesign	Series overall design when the accession display returns it
platform	Platform accession, with a short platform title when GEO provides one
sampleCount	Sample count from the GEO summary
pubmedIds	PubMed ids cited on the record
geoUrl	Public GEO accession page
entryType	GSE, GDS, GSM, or GPL
releaseDate / updateDate	Release date, and last update when SOFT metadata includes it

Keyword search returns series (GSE). Sample (GSM) and platform (GPL) rows are returned only when you pass those accessions. Contact names, emails, and phone numbers from SOFT files are not copied into the dataset.

How to search NCBI GEO

1. Open GEO Dataset Finder in Apify Console and start a run, or call it from the Apify API.
2. Enter a search query such as lung adenocarcinoma. Add an organism when you want NCBI's organism field applied.
3. Or paste accessions such as GSE10072 or GDS3034. A query is required only when the accession list is empty.
4. Set maximum results (default 50, maximum 500). Exact accessions are resolved first, then the keyword search fills any remaining slots.
5. Start the run and open the output dataset. You can download it as JSON, HTML, CSV, or Excel.

How do I run GEO Dataset Finder locally?

Install the Apify CLI and, from this directory, install the pinned Python dependencies:

$pip install -r requirements.txt

The sample input lives at storage/key_value_stores/default/INPUT.json. Run the Actor with the local Apify environment (this writes to storage/, not to the Apify Console):

$apify run

To try a smaller search without editing that file:

$apify run --purge --input '{"query":"lung adenocarcinoma","organism":"Homo sapiens","maxResults":2}'

Then inspect storage/datasets/default/. Use pytest to run the offline tests for query building, SOFT and esummary parsing, and HTTP retry behavior. apify validate-schema checks the input, dataset, and output schemas.

How much will it cost to search NCBI GEO?

This Actor does not set its own price and does not charge per result. You pay only for the Apify platform compute used by the run. A search of 50 series is a small number of E-utilities calls plus one brief metadata request per record, so a typical run finishes quickly and uses a small fraction of a compute unit. NCBI E-utilities are free. An optional NCBI API key raises the allowed request rate; this Actor does not sell keys. No residential proxy is required.

To limit usage, keep maxResults small while you tune the query, and avoid scheduling overlapping runs against the same search.

Input

See the input tab for every field. query is a free-text GEO search. organism is appended only to that keyword search. accessions is a list of GSE, GSM, GPL, or GDS ids. If the query itself is only accessions, separated by spaces or commas, they are looked up exactly. maxResults defaults to 50 and cannot exceed 500. ncbiApiKey is optional and marked secret so it is masked in the Console; it is not written to the dataset or the log.

Output

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel. Each item is a flat object:

{
  "accession": "GSE12345",
  "gdsId": "GDS3034",
  "title": "Example expression series in human lung",
  "summary": "Example summary of a public gene expression experiment.",
  "organism": "Homo sapiens",
  "overallDesign": "Example design comparing treated and control samples.",
  "platform": "GPL96",
  "sampleCount": 12,
  "pubmedIds": ["12345678"],
  "geoUrl": "https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE12345",
  "entryType": "GSE",
  "releaseDate": "2008-02-20",
  "updateDate": "2018-08-10"
}

A second row for a curated DataSet uses accession GDS3034 and entryType GDS. Fields that GEO does not provide are JSON null. overallDesign and updateDate stay empty when the accession display does not return SOFT text.

Tips

• Prefer a specific query plus an organism. Broad terms match tens of thousands of series; maxResults keeps the run bounded.
• Pass a known GSE when you only need one study. That path does not pull every GSM under the series.
• If overall design is missing, the rest of the row still comes from E-utilities. GEO sometimes answers the accession page with an interstitial instead of SOFT text; the Actor skips that body and continues.
• Create an NCBI API key in your NCBI account if you run large or frequent searches, and store it in the secret input field.

FAQ, disclaimers, and support

Does GEO Dataset Finder download raw expression files?

No. The dataset contains metadata and links only. Open geoUrl on the NCBI site if you need the files GEO distributes.

Do I need an NCBI account?

No. The key is optional and only changes the request rate from 3 per second to 10 per second.

Why are individual samples missing from a keyword search?

Keyword search is limited to series records so the dataset is not flooded with one row per sample. Pass GSM or GPL accessions to retrieve those records. A series row still reports sampleCount and, when GEO curated the study, gdsId.

Is the data personal?

GEO records are scientific submissions that NCBI has made public. This Actor does not extract account data, contact emails, or private files. Summaries can still describe study populations. Our Actors are ethical and do not extract private user data such as email addresses, gender, or location. They only extract what has been published in a public archive. We therefore believe that our Actors, when used for ethical purposes by Apify users, are safe. However, you should be aware that your results could contain personal data. Personal data is protected by the GDPR in the European Union and by other regulations around the world. You should not scrape personal data unless you have a legitimate reason to do so. If you are unsure whether your reason is legitimate, consult your lawyers.

Questions and bug reports belong on the Issues tab. For programmatic runs, use the API tab on the Actor page.