# URL Wrangler (`rl1987/url-wrangler`) Actor Join relative URLs, decompose URLs into a node/edge tree, replace/remove query params, and extract fields — batch URL wrangling toolkit. - **URL**: https://apify.com/rl1987/url-wrangler.md - **Developed by:** [R.L.](https://apify.com/rl1987) (community) - **Categories:** Developer tools - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.01 / actor run This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. - **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use). - **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`. - **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`. - **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`. - **Any other language** — the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README **URL Wrangler** is a batch URL-processing toolkit for the [Apify platform](https://apify.com). Feed it a list of URLs and pick one of four operations — resolve relative URLs, decompose a URL into a labeled node/edge tree, upsert or strip query parameters, or pull out a single field — and get structured JSON back, one row per input URL. Run it via the [Apify Console](https://console.apify.com), the [API](https://docs.apify.com/api/v2), or on a [schedule](https://docs.apify.com/platform/schedules), with full run history, monitoring, and integrations (Zapier, Make, Google Sheets, webhooks) included. ### Why use URL Wrangler? Working with URLs at scale — cleaning up scraped links, rewriting tracking parameters before storing them, or reverse-engineering an unfamiliar link structure during OSINT/DFIR work — usually means writing one-off scripts around `urllib.parse`. URL Wrangler packages the common operations into a single Actor so you can: - **Deduplicate and clean up crawl output** by resolving every relative link a scraper found against its page URL (`urljoin`). - **Investigate a suspicious or unfamiliar URL** by breaking it into its components and auto-decoding embedded Base64 blobs, hex strings, UUIDs, and Unix timestamps (`unfurl`) — the same idea as the DFIR tool [`unfurl`](https://github.com/obsidianforensics/unfurl), reimplemented here dependency-free. - **Strip tracking parameters or rewrite campaign tags in bulk** before storing or forwarding URLs (`replace_params`). - **Pull out just the domain, path, or a custom-formatted string** from a batch of URLs for reporting or filtering (`extract`). ### How to use URL Wrangler 1. Open the Actor's **Input** tab. 2. Choose an **Operation**: `urljoin`, `unfurl`, `replace_params`, or `extract`. 3. Paste your list of URLs into **URLs**. 4. Fill in the fields relevant to your chosen operation (see [Input](#input) below, and [Examples](#examples) for a full input/output pair per operation — irrelevant fields are ignored). 5. Click **Start**. Results land in the run's default **Dataset**, one row per input URL, downloadable as JSON, CSV, Excel, or HTML. ### Input All fields live in one flat input object — an `operation` selector, the shared `urls` list, and a handful of operation-specific parameters that apply uniformly to every URL in that run. See the **Input** tab for the full schema with descriptions. | Field | Used by | Description | |---|---|---| | `operation` | all | `urljoin` | `unfurl` | `replace_params` | `extract` | | `urls` | all | The URLs to process (relative URLs, for `urljoin`) | | `baseUrl` | `urljoin` | Reference URL every entry in `urls` is resolved against | | `params` | `replace_params` | Key/value pairs to add or overwrite, via a key-value editor (no JSON typing) | | `remove` | `replace_params` | Keys to strip; supports `fnmatch` wildcards like `utm_*` | | `field` | `extract` | `domain` | `apex` | `subdomain` | `tld` | `path` | `keys` | `values` | `keypairs` | `format` | | `format` | `extract` (`field=format`) | Directive string, e.g. `%s://%d%p?%q` | | `unfurlDetail` | `unfurl` | `summary` (default) or `full` (adds a node/edge count summary) | | `outputFormat` | `unfurl` | Include a human-readable ASCII tree alongside the JSON graph | ### Output Every run pushes one dataset item per input URL: `{ operation, input, result, error }`. `error` is `null` on success, so a single malformed URL never aborts the run. ### Examples Each pair below is a complete Actor input alongside the corresponding dataset row it produces. #### `urljoin` Input: ```json { "operation": "urljoin", "urls": ["../c", "/d", "https://other.com/x"], "baseUrl": "https://example.com/a/b/" } ``` Output (first row): ```json { "operation": "urljoin", "input": "../c", "result": "https://example.com/a/c", "error": null } ``` #### `unfurl` Input: ```json { "operation": "unfurl", "urls": ["https://sub.example.com/users/1?ts=1700000000&data=aGVsbG8="], "outputFormat": true } ``` Output: ```json { "operation": "unfurl", "input": "https://sub.example.com/users/1?ts=1700000000&data=aGVsbG8=", "result": { "nodes": [ { "id": 1, "type": "url", "key": null, "value": "https://sub.example.com/users/1?ts=1700000000&data=aGVsbG8=", "label": "https://sub.example.com/users/1?ts=1700000000&data=aGVsbG8=" }, { "id": 2, "type": "url.scheme", "key": null, "value": "https", "label": "https" }, { "id": 3, "type": "url.hostname", "key": null, "value": "sub.example.com", "label": "sub.example.com" }, { "id": 6, "type": "url.query.param", "key": "ts", "value": "1700000000", "label": "ts: 1700000000" }, { "id": 7, "type": "decoded.timestamp", "key": null, "value": "2023-11-14T22:13:20+00:00", "label": "2023-11-14T22:13:20+00:00" }, { "id": 8, "type": "url.query.param", "key": "data", "value": "aGVsbG8=", "label": "data: aGVsbG8=" }, { "id": 9, "type": "decoded.base64", "key": null, "value": "hello", "label": "hello" } ], "edges": [ { "from": 1, "to": 2, "label": "url-parse" }, { "from": 1, "to": 3, "label": "url-parse" }, { "from": 1, "to": 6, "label": "query-split" }, { "from": 6, "to": 7, "label": "epoch-decode" }, { "from": 1, "to": 8, "label": "query-split" }, { "from": 8, "to": 9, "label": "base64-decode" } ], "text": "[1] https://sub.example.com/users/1?ts=1700000000&data=aGVsbG8=\n├─(url-parse)─[2] https\n├─(url-parse)─[3] sub.example.com\n├─(query-split)─[6] ts: 1700000000\n│ └─(epoch-decode)─[7] 2023-11-14T22:13:20+00:00\n└─(query-split)─[8] data: aGVsbG8=\n └─(base64-decode)─[9] hello" }, "error": null } ``` *(path-segment nodes trimmed above for brevity — the full graph includes every path segment as its own node.)* #### `replace_params` Input: ```json { "operation": "replace_params", "urls": ["https://example.com/?a=1&utm_source=x&gclid=z"], "params": [{ "key": "a", "value": "9" }], "remove": ["utm_*", "gclid"] } ``` Output: ```json { "operation": "replace_params", "input": "https://example.com/?a=1&utm_source=x&gclid=z", "result": "https://example.com/?a=9", "error": null } ``` #### `extract` Input: ```json { "operation": "extract", "urls": ["https://sub.example.com/users/1?id=1"], "field": "format", "format": "%s://%d%p?%q" } ``` Output: ```json { "operation": "extract", "input": "https://sub.example.com/users/1?id=1", "result": "https://sub.example.com/users/1?id=1", "error": null } ``` ### Data table | Field | Type | Present when | Description | |---|---|---|---| | `operation` | string | always | Operation that produced this row | | `input` | string | always | The original input URL | | `result` | varies | on success | Absolute URL string (`urljoin`), node/edge object (`unfurl`), rewritten URL string (`replace_params`), or field value/array/string (`extract`) | | `error` | string | null | always | Failure reason for this URL, or `null` | ### Pricing / cost estimation URL Wrangler does pure in-memory string/URL processing — no network requests, no browser, no proxy usage. Cost is driven entirely by compute time, which is minimal (typically well under 100ms per URL). On the [Apify Free plan](https://apify.com/pricing), you can process tens of thousands of URLs per run within the platform's free monthly compute unit allowance. ### Tips / advanced options - For `unfurl`, leave `outputFormat` off unless you want the ASCII tree — it roughly doubles the payload size per row for large graphs. - Base64/hex/timestamp/UUID decoding in `unfurl` recurses up to 5 levels deep (e.g. a query param that's itself Base64-encoded JSON containing another encoded value), capped to bound output size on adversarial input. - `replace_params`'s `remove` field supports `fnmatch`-style wildcards (`utm_*`, `*_id`), so you don't need to enumerate every tracking parameter by name. - The naive `apex`/`subdomain`/`tld` split in `extract` and `unfurl` uses the last two DNS labels — it's correct for `.com`/`.net`/`.org`-style domains but not for multi-part public suffixes like `.co.uk` (that needs a full [Public Suffix List](https://publicsuffix.org/), not currently bundled). ### Background reading The `unfurl` operation's node/edge tree is modeled on the DFIR community tool of the same name, and `extract`'s field/format model is modeled on a separate, unrelated CLI tool. Related reading: - [obsidianforensics/unfurl](https://github.com/obsidianforensics/unfurl) — the DFIR tool this Actor's `unfurl` operation is modeled on - [SANS ISC: Unfurl v2025.02 released](https://isc.sans.edu/diary/31716) — write-up on `unfurl`'s timestamp/IP/UUID decoding capabilities - [tomnomnom/unfurl](https://github.com/tomnomnom/unfurl) — the flat field-extraction CLI this Actor's `extract` operation is modeled on - [Google Search Central: URL canonicalization](https://developers.google.com/search/docs/crawling-indexing/canonicalization) — background on why URL normalization matters for deduplication - [Pinterest Engineering: Smarter URL Normalization at Scale (MIQPS)](https://medium.com/pinterest-engineering/smarter-url-normalization-at-scale-how-miqps-powers-content-deduplication-at-pinterest-4aa42e807d7d) — a production system for deciding which query parameters are semantically significant ### FAQ, limitations, and support - This Actor only processes URLs you provide — it does not fetch, crawl, or follow redirects for any of them. - `unfurl`'s decoders are heuristic (they detect *plausible* Base64/hex/epoch/UUID values); false negatives on obfuscated data and, rarely, false positives on coincidentally-decodable strings are possible. - Found a bug or want another operation (URL normalization/dedup, punycode/IDNA conversion, tracking-parameter presets)? Open an issue on the Actor's Issues tab — these are tracked as candidate follow-ups. ### Data pipeline toolkit Part of the **Data pipeline toolkit** — small, chainable Actors for cleaning, transforming, and generating data inside a larger pipeline: - [jq Helper – transform JSON with jq](https://apify.com/rl1987/jq-helper) — Run jq programs over inline JSON or a linked Apify dataset. - [DuckDB Helper – SQL over CSV, JSON, Parquet, Excel, SQLite](https://apify.com/rl1987/duckdb-wrapper) — Run a DuckDB SQL query over remote/local files, push results to a dataset. - [Regex Helper](https://apify.com/rl1987/regex-helper) — Apply named regular expressions to strings, extract structured matches. - [ZIP Code Helper](https://apify.com/rl1987/zip-code-helper) — Resolves US ZIP codes into city, state, county, and more. - [Postal Address Normaliser](https://apify.com/rl1987/postal-address-normaliser) — Parses and normalises postal addresses using libpostal. - [Phone Number Wrangler](https://apify.com/rl1987/phone-number-wrangler) — Validate, format, and parse phone numbers using libphonenumber. - [UUID Generator](https://apify.com/rl1987/uuid-generator) — Generate bulk UUIDs (v1, v3, v4, v5, v7) on demand. - [Secure Password & Passphrase Generator](https://apify.com/rl1987/password-generator) — Generate secure passwords and diceware passphrases per NIST guidance. - [Thumbnail Maker](https://apify.com/rl1987/thumbnail-maker) — Generates thumbnails from image URLs using ImageMagick. - [Katana Web Crawler (ProjectDiscovery)](https://apify.com/rl1987/pd-katana) — Crawl websites with Katana, stream results as JSONL. - [ProjectDiscovery Notify](https://apify.com/rl1987/pd-notify) — Stream records to Slack, Discord, Telegram, Email, and more. ### Did you find this useful? ⭐ Rate this actor on Apify! Your feedback helps other users find it and helps us keep improving it. # Actor input Schema ## `operation` (type: `string`): Which URL-wrangling operation to run over the URL list. ## `urls` (type: `array`): The URLs to process. For 'urljoin', these are the relative URLs resolved against Base URL. For the other operations, these are the URLs operated on directly. ## `baseUrl` (type: `string`): Reference URL every entry in URLs is resolved against. ## `params` (type: `array`): Query-string key/value pairs to add or overwrite on every URL. ## `remove` (type: `array`): Query-string keys to strip from every URL. Supports fnmatch wildcards, e.g. "utm\_\*". ## `field` (type: `string`): Which field to extract from every URL. ## `format` (type: `string`): Printf-style directive string, e.g. "%s://%d%p?%q". Directives: %s scheme, %u userinfo, %d domain, %S subdomain, %r root, %t tld, %P port, %p path, %e extension, %q query, %f fragment, %@ %: %? %# conditional separators, %a authority, %% literal percent. ## `unfurlDetail` (type: `string`): 'summary' returns the node/edge tree; 'full' also includes a node/edge count summary. ## `outputFormat` (type: `boolean`): Also render a human-readable ASCII tree alongside the JSON node/edge graph. ## Actor input object example ```json { "operation": "urljoin", "urls": [ "../c", "/d" ], "baseUrl": "https://example.com/a/b/", "field": "domain", "unfurlDetail": "summary", "outputFormat": false } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "urls": [ "../c", "/d" ], "baseUrl": "https://example.com/a/b/" }; // Run the Actor and wait for it to finish const run = await client.actor("rl1987/url-wrangler").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "urls": [ "../c", "/d", ], "baseUrl": "https://example.com/a/b/", } # Run the Actor and wait for it to finish run = client.actor("rl1987/url-wrangler").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "urls": [ "../c", "/d" ], "baseUrl": "https://example.com/a/b/" }' | apify call rl1987/url-wrangler --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,rl1987/url-wrangler" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/jYXgiMDWp7QAbQkmj/builds/IbGR0hTZMXNqnMtgO/openapi.json