# Klikindomaret (Indomaret Xpress) Scraper (`fanndev/klikindomaret-scraper`) Actor

Scrape Indomaret Klik Xpress groceries: product listings (price, discount, promo, stock) by keyword or category, the category tree, promo banners, search suggestions, and store resolution. HTTP-only internal API; auto-solves AWS WAF + Cloudflare. No login.

- **URL**: https://apify.com/fanndev/klikindomaret-scraper.md
- **Developed by:** [Faisal Ahdan naufal](https://apify.com/fanndev) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $3.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Klikindomaret (Indomaret Xpress) Scraper

Scrape **klikindomaret.com** — Indomaret's *Klik Xpress* grocery e-commerce (Indonesia) — over its internal JSON API. No login, no API key. HTTP-only for the heavy lifting, with an automatic browser step only where Cloudflare forces one.

### What it extracts (by `mode`)

| Mode | Records | What you get |
| --- | --- | --- |
| `productSearch` | `PRODUCT` | Product listings by **keyword** and/or **category ID** — name, PLU, price, `finalPrice`, discount %, promo, brand, dimensions, stock/quota, image URLs. Paginated to exhaustion (or `maxItems`). |
| `categoryTree` | `CATEGORY` | The full category hierarchy, flattened — one record per node with a breadcrumb `_categoryPath` and `id` (feed those IDs back into `productSearch`). |
| `promoBanners` | `BANNER`, `PROMO_FILTERS` | Homepage promo banners plus the promo-type / meta-category filter catalogue. |
| `searchSuggestions` | `SUGGESTIONS` | Popular keywords, last-seen/last-search, category recommendations. |
| `stores` | `STORE` | Resolve store code + area + coordinates for one or more district IDs. |

### Input

```jsonc
{
  "mode": "productSearch",
  "searchTerms": ["indomie", "kopi"],   // productSearch
  "categoryIds": ["1059"],              // productSearch (from a categoryTree run)
  "storeCode": "",                       // optional; auto-resolved from districtId
  "districtId": "141100100",             // default: Gambir, Jakarta Pusat
  "districtIds": [],                     // stores mode
  "maxItems": 0,                          // 0 = no cap
  "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"] }
}
```

Everything on Klik Xpress is **store-scoped** (click-and-collect / delivery), so catalog data always resolves against a store. Leave `storeCode` empty and the actor resolves the nearest store from `districtId`.

### Output

Raw-passthrough dataset envelope: every upstream field is kept verbatim; actor-added fields are prefixed `_` (`_input`, `_source`, `_scrapedAt`, plus `recordType`, and in `productSearch` a `_query`/`_queryType` for traceability). Failed or empty runs still emit one diagnostic row with `_error` — downstream joins never see a silently empty dataset.

### How it works (WAF)

The site is a Next.js static-export SPA (Webpack Module Federation micro-frontends). The real runtime API is on **`ap-mc.klikindomaret.com/assets-klikidm{module}/api/get/...`**, protected by two independent gates:

1. **AWS WAF** on every `/api/get/` route — needs an `x-aws-waf-token`. The token's proof-of-work is a `NetworkBandwidth` challenge that a headless **Node + jsdom** solver mints in ~2–3 s (no browser). The token is cached and re-minted before it expires.
2. **Cloudflare managed challenge** on the `search/result` route only (the product listing) — needs a `cf_clearance` cookie that only a real browser earns. **Camoufox** (stealth Firefox) solves it once per run; the cookie is then replayed over `curl_cffi` (Firefox TLS profile) for fast bulk paging. Camoufox is launched **only** for `productSearch`; the other four modes are pure HTTP.

TLS profile: `firefox133` (must match Camoufox's Firefox identity so the clearance cookie replays). On a Cloudflare block the actor re-bootstraps clearance; on an AWS block it re-mints the token; both with exponential backoff.

> `api-klik.klikindomaret.com` (the axios fallback base URL in the bundles) is hard Cloudflare-bot-gated on real routes and is deliberately **not** used.

### Local run

```bash
pip install -r requirements.txt
npm install
python -m camoufox fetch            # once, downloads the stealth Firefox
python test_local.py productSearch  # or: categoryTree | promoBanners | searchSuggestions | stores | token
```

### Known limits

- Product data is store-scoped; different `storeCode`/district can show different assortments, prices, and stock.
- `productSearch` requires the Camoufox step; on Apify give the run enough memory (default 4 GB) for the browser.
- The AWS WAF SDK URL and Cloudflare rules can rotate; if `productSearch` starts failing, re-check the `*.sdk.awswaf.com/.../challenge.js` URL in the homepage HTML (`src/constants.py:AWS_SDK_URL`).
- No login-gated data (cart, checkout, account) — public catalog only.

# Actor input Schema

## `mode` (type: `string`):

What to extract. 'productSearch' = product listings by keyword or category (with price, discount, promo, stock). 'categoryTree' = the full category hierarchy. 'promoBanners' = homepage promo banners plus the promo-type filters. 'searchSuggestions' = popular keywords and recommendations. 'stores' = resolve store/area info for one or more district IDs.

## `searchTerms` (type: `array`):

productSearch mode. Keywords to sweep, e.g. \["indomie", "susu"]. Each is paged until exhausted or Max items. If left empty and no Category IDs are given, the run falls back to the keyword 'indomie'.

## `categoryIds` (type: `array`):

productSearch mode. Category IDs to list products for (get them from a 'categoryTree' run's `id` field). Combined with searchTerms if both are given.

## `storeCode` (type: `string`):

Optional. The Indomaret Xpress store to scope catalog data to (e.g. 'TJKT'). Leave empty to resolve automatically from District ID. Everything on this site is store-scoped (click-and-collect).

## `districtId` (type: `string`):

Optional. Indonesian district (kecamatan) ID used to resolve the nearest store when Store code is empty. Defaults to 141100100 (Gambir, Jakarta Pusat).

## `districtIds` (type: `array`):

stores mode only. One or more district IDs to resolve store/area info for.

## `latitude` (type: `string`):

Optional. Latitude for the store context. Auto-resolved from the store/district when empty.

## `longitude` (type: `string`):

Optional. Longitude for the store context. Auto-resolved from the store/district when empty.

## `maxItems` (type: `integer`):

Stop after this many records (per query in productSearch mode). 0 means no limit -- scrape until the source is exhausted.

## `proxyConfiguration` (type: `object`):

Klikindomaret is behind Cloudflare + AWS WAF. Apify Residential proxy with country 'ID' is recommended for reliability; short runs often succeed without a proxy.

## Actor input object example

```json
{
  "mode": "productSearch",
  "searchTerms": [
    "indomie"
  ],
  "categoryIds": [],
  "districtIds": [],
  "maxItems": 100,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ]
  }
}
```

# Actor output Schema

## `results` (type: `string`):

Every product, category, banner, suggestion and store record produced by this run.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "searchTerms": [
        "indomie"
    ],
    "maxItems": 100
};

// Run the Actor and wait for it to finish
const run = await client.actor("fanndev/klikindomaret-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "searchTerms": ["indomie"],
    "maxItems": 100,
}

# Run the Actor and wait for it to finish
run = client.actor("fanndev/klikindomaret-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "searchTerms": [
    "indomie"
  ],
  "maxItems": 100
}' |
apify call fanndev/klikindomaret-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,fanndev/klikindomaret-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/tu2K2rofi93mr9Ibi/builds/tM77xOXzpJ6PaB0rf/openapi.json