Algolia Search Index Scraper — Hits, Facets & Filters

A generalised Algolia search-index scraper for extracting search results and discovering facets from any website that uses Algolia as its search backend — e-commerce catalogs, documentation sites, marketplaces, job boards, directories. Instead of scraping rendered HTML, this Actor talks directly to the site's own Algolia index over the Algolia Search REST API, so extraction is fast, reliable, and returns the exact structured data the site itself displays. Run it on the Apify platform for scheduling, API access, and monitoring without managing any infrastructure.

What does this Algolia scraper do?

It runs one of two modes against an index you point it at:

• Scrape hits — pages through the index and pushes every record to the dataset, exactly as Algolia returns it, with whatever attributes that index happens to store. You can narrow it with the same query, filters, facetFilters and numericFilters the site's own search UI uses.
• Discover facets — a single request that returns every facetable attribute on the index and its values with counts, plus the total nbHits. Use it first on an unfamiliar index to find out what you can filter by, then build a targeted scrape.

It needs three things, all of them visible in the site's own browser network traffic: an Application ID, a search-only API key, and an index name. It ships with Hacker News Search's public Algolia credentials pre-filled, so you can press Start and see real output before you go looking for your own.

Why scrape a site's Algolia index instead of its pages?

Thousands of sites (documentation portals, online stores, job boards, real-estate listings) use Algolia under the hood for instant search. Querying that index directly means you can:

• Read the site's structured records — the same JSON its own front end renders — rather than parsing markup that changes whenever the design does.
• Apply the exact filters, facets and query syntax the site's search UI exposes.
• Find out which attributes are filterable/facetable, and what values exist for each, before committing to a larger scraping or monitoring pipeline.
• Feed clean JSON straight into spreadsheets, databases or downstream automations via Apify's integrations.

How do I find a site's Algolia credentials?

1. Open the site and open DevTools → Network, filtering for algolia.net or algolianet.com.
2. Type something into the site's search box. You'll see POST .../1/indexes/<indexName>/query requests.
3. Read off the three values:
   • Application ID — the request host prefix, e.g. ABC123DEF-dsn.algolia.net, or the X-Algolia-Application-Id header.
   • API key — the X-Algolia-API-Key header. This is a search-only key; sites ship it to the browser on purpose.
   • Index name — the path segment after /1/indexes/.
4. Paste them into the Actor's input, add a query or filters if you want, and click Start. Or enable Discover facets only for a fast look at the index first.

What can I try it on right away?

Two public indexes that are documented as such and work with no setup:

Index	Application ID	API key	Index name	Facets it exposes
Hacker News Search (the Actor's default input)	UJ5WYC0L7X	28f0e1ec37a5e792e6845e67da5f20dd	Item_dev	_tags (story, comment, ask_hn, show_hn, …), created_at_i
Algolia's own instant_search demo catalog	latency	6be0576ff61c053d5f9a3225e2a90f76	instant_search	brand, categories, type, price, price_range, rating, free_shipping, hierarchicalCategories.lvl0–lvl4

For your own target, the credentials come out of the site's network traffic as described above.

What do Algolia's terms mean?

• Application ID — identifies the Algolia account. It also names the hosts: <appId>-dsn.algolia.net plus the -1/-2/-3.algolianet.com cluster fallbacks this Actor fails over to.
• Search-only API key — a key that can query indexes but not write to them. It is the key a site's front end carries, and the only kind this Actor needs.
• objectID — Algolia's unique identifier for a record within an index. The one field every hit is guaranteed to have.
• Facet — an attribute the index is configured to aggregate on, so search results can be counted and filtered by its values (brand, price_range, _tags). Requesting facets: ["*"] asks for all of them.
• filters vs facetFilters vs numericFilters — filters is one SQL-ish expression string (category:Shoes AND price < 50). facetFilters is an array where a nested array means OR (["category:Shoes", ["color:Red", "color:Blue"]] = Shoes AND (Red OR Blue)). numericFilters is an array of numeric comparisons (["price>=10", "price<=100"]).
• nbHits vs nbPages — nbHits is how many records match. nbPages is how many pages you are actually allowed to fetch, and the two often disagree badly (see the limit below).
• paginationLimitedTo — an index-side setting, default 1,000 records, that caps how deep anyone can page. Both public indexes above show it: Hacker News reports 46.2 million matching records but only one page at hitsPerPage: 1000, and Algolia's demo catalog reports 21,468 matches but 1,000 pages of one hit. This is the single biggest thing to plan around.
• _highlightResult / _snippetResult — extra sub-objects Algolia adds to each hit with <em>-marked matches. This Actor strips HTML tags from every string value it pushes, so those come through as plain text.

Can I really pull a whole catalog?

Usually not in one run, and this is Algolia's limit rather than the Actor's. Most indexes leave paginationLimitedTo at its default of 1,000 records, so a single query can never hand you more than that no matter how large maxItems or hitsPerPage are. The way around it is to slice the index into chunks that each stay under the cap:

1. Run Discover facets only to see the facetable attributes and how many records sit under each value.
2. Pick a facet that partitions the index reasonably — a category, a brand, a type, a date bucket.
3. Run one scrape per value via facetFilters, or per range via numericFilters (e.g. ["price>=0","price<50"], then ["price>=50","price<100"], and so on).

What input does it take?

Field	Type	Default	Description
algoliaAppId	string	Hacker News' UJ5WYC0L7X	Required. Algolia Application ID.
algoliaApiKey	string	Hacker News' public key	Required. Search-only Algolia API key.
indexName	string	Item_dev	Required. Index to query.
searchQuery	string	""	Free-text query. Empty matches all records (browse mode).
filters	string	""	Algolia filter expression, e.g. category:Shoes AND price < 50.
facetFilters	array	[]	Array of facet filters; a nested array is an OR group.
numericFilters	array	[]	Array of numeric comparisons, e.g. ["price>=10", "price<=100"].
facets	array	["*"]	Facet attributes to request counts for; ["*"] means all facetable ones.
maxValuesPerFacet	integer	100	Cap on facet values returned per facet (1–1000).
discoverFacetsOnly	boolean	false	Make one hitsPerPage=0 request and output only the facets and their counts — no items scraped.
attributesToRetrieve	array	[]	Limit which attributes come back per hit. Empty retrieves all.
hitsPerPage	integer	1000	Hits per Algolia request (1–1000).
maxItems	integer	0	Total item cap. 0 = every page Algolia will serve.
extraQueryParams	object	{}	Any other raw Algolia SearchParams field — distinct, optionalFilters, restrictSearchableAttributes, aroundLatLng, and so on — merged into the request body last, so it overrides everything above.

Example input for facet discovery:

{
    "algoliaAppId": "latency",
    "algoliaApiKey": "6be0576ff61c053d5f9a3225e2a90f76",
    "indexName": "instant_search",
    "facets": ["*"],
    "discoverFacetsOnly": true
}

Example input for a filtered slice of the same index:

{
    "algoliaAppId": "latency",
    "algoliaApiKey": "6be0576ff61c053d5f9a3225e2a90f76",
    "indexName": "instant_search",
    "facetFilters": ["brand:Apple"],
    "numericFilters": ["price>=500"],
    "hitsPerPage": 1000,
    "maxItems": 0
}

What does a row look like?

Each scraped record is pushed to the dataset as Algolia returns it, with HTML stripped from string values. The attributes are whatever the source index stores, so they differ per site — only objectID is guaranteed. From the instant_search demo catalog:

{
    "objectID": "6443034",
    "name": "Apple - MacBook Air\u00ae (Latest Model) - 13.3\" Display - Intel Core i5 - 8GB Memory - 128GB Flash Storage - Silver",
    "brand": "Apple",
    "categories": ["Computers & Tablets", "Laptops", "All Laptops", "MacBooks"],
    "type": "Apple",
    "price": 999.99,
    "price_range": "500 - 2000",
    "rating": 4,
    "free_shipping": true,
    "image": "https://cdn-demo.algolia.com/bestbuy-0118/6443034_sb.jpg",
    "url": "https://api.bestbuy.com/click/-/6443034/pdp"
}

When Discover facets only is enabled, the dataset instead contains a single item:

{
    "facets": {
        "brand": { "Insignia\u2122": 746, "Samsung": 633 },
        "price_range": { "1 - 50": 9927, "50 - 100": 3470 }
    },
    "facets_stats": {
        "price": { "min": 1, "max": 4999.99, "avg": 242.813, "sum": 5212710 }
    },
    "nbHits": 21468
}

The same facet data is also saved to the key-value store under the FACETS key. In hit-scraping mode, FACETS gets the facet counts seen on the last page fetched. Download the dataset as JSON, CSV, Excel, XML or HTML.

Which fields are in the dataset?

Field	Description
objectID	Algolia's unique identifier for the record — present on every hit
(all other fields)	Whatever attributes the source index stores; varies per site and per index

In facet-discovery mode the single row is facets, facets_stats and nbHits instead.

How much does it cost?

Pay-per-event: $0.50 per 1,000 records pushed to the dataset ($0.0005 each). Facet discovery is not charged per item — it outputs one row from a single request. There is no browser rendering, so compute is negligible next to the per-record price; a large scrape's bill is essentially the record count. maxItems is the way to bound it. If you set a maximum total charge on the run, the Actor notices when it is reached and stops cleanly before starting another page.

How do I get more out of it?

• Leave hitsPerPage at its maximum of 1000 to minimise requests.
• Use maxItems to cap a run when you only need a sample.
• Run Discover facets only first on a new index, then build a targeted filters/facetFilters query for the real scrape — and to work around paginationLimitedTo, as described above.
• Set attributesToRetrieve to just the fields you need, to shrink rows. Note that Algolia still attaches _highlightResult and _snippetResult sub-objects regardless, so drop those in post-processing if you don't want them.
• Pass any additional Algolia parameter (distinct, optionalFilters, restrictSearchableAttributes, aroundLatLng) via extraQueryParams.

FAQ

Do I need an Algolia account or a paid API key?

No. You need the search-only key the target site already ships to its own visitors, and an Apify account. The Actor has no Algolia account of its own.

Do I need a proxy?

No. The Actor calls Algolia's API hosts directly and has no proxy input. It fails over across <appId>-dsn.algolia.net and the -1/-2/-3.algolianet.com cluster hosts, and retries 429/5xx responses up to three times per host with backoff.

Why did I get far fewer records than nbHits said?

Because of paginationLimitedTo — Algolia caps how deep you can page, at 1,000 records by default. nbHits counts matches; nbPages tells you what you can actually fetch. Split the index with facet or numeric filters to get past it.

Can I scrape a private index or one I don't have a key for?

No. The Actor only queries with credentials you supply and does not bypass authentication. A 401, 403, 404 or 400 from Algolia fails the run immediately rather than retrying, so a wrong key or index name is obvious straight away.

What happens if the site rotates its key?

The run fails with the Algolia error and status code in the log. Re-check the site's current Application ID, key and index name in its network requests and update the input — search-only keys do get rotated.

Does it work for documentation sites and job boards, not just shops?

Yes. The Actor makes no assumptions about the record shape — it pushes whatever attributes the index stores. That is why the dataset has no fixed schema beyond objectID.

Can I schedule it?

Yes — use Apify's scheduler for recurring runs and a webhook to push each finished dataset onward. Join runs on objectID, which is stable within an index.

Is this allowed?

You are responsible for how you use the data. Only query indexes and keys you are authorised to query, and respect the target site's Terms of Service. The Actor accesses no private data and bypasses no authentication.

Global marketplace toolkit

Part of the Global marketplace toolkit — product and pricing data across international B2B/B2C marketplaces and classifieds sites:

• Shopify API Scraper — Get product data from almost any Shopify store's public API.
• DHgate API Scraper — Scrapes DHgate product search results and detail pages via mobile API.
• Made-in-China.com API Scraper — Scrapes product search results and details from Made-in-China.com's mobile app API.
• Back Market API Scraper — Scrape refurbished-device listings and full product detail from Back Market's API.
• Senukai.lt API Scraper — Scrape Senukai.lt product listings and full detail — prices, discounts, EAN/GTIN, images.
• Kleinanzeigen API Scraper — Scrape Kleinanzeigen (Germany) classified-ad listings and full details via mobile API.

Did you find this useful?

Rate this actor on Apify! Your feedback helps other users find it and helps us keep improving it.