URL Wrangler

URL Wrangler is a batch URL-processing toolkit for the Apify platform. Feed it a list of URLs and pick one of four operations — resolve relative URLs, decompose a URL into a labeled node/edge tree, upsert or strip query parameters, or pull out a single field — and get structured JSON back, one row per input URL. Run it via the Apify Console, the API, or on a schedule, with full run history, monitoring, and integrations (Zapier, Make, Google Sheets, webhooks) included.

Why use URL Wrangler?

Working with URLs at scale — cleaning up scraped links, rewriting tracking parameters before storing them, or reverse-engineering an unfamiliar link structure during OSINT/DFIR work — usually means writing one-off scripts around urllib.parse. URL Wrangler packages the common operations into a single Actor so you can:

• Deduplicate and clean up crawl output by resolving every relative link a scraper found against its page URL (urljoin).
• Investigate a suspicious or unfamiliar URL by breaking it into its components and auto-decoding embedded Base64 blobs, hex strings, UUIDs, and Unix timestamps (unfurl) — the same idea as the DFIR tool unfurl, reimplemented here dependency-free.
• Strip tracking parameters or rewrite campaign tags in bulk before storing or forwarding URLs (replace_params).
• Pull out just the domain, path, or a custom-formatted string from a batch of URLs for reporting or filtering (extract).

How to use URL Wrangler

1. Open the Actor's Input tab.
2. Choose an Operation: urljoin, unfurl, replace_params, or extract.
3. Paste your list of URLs into URLs.
4. Fill in the fields relevant to your chosen operation (see Input below, and Examples for a full input/output pair per operation — irrelevant fields are ignored).
5. Click Start. Results land in the run's default Dataset, one row per input URL, downloadable as JSON, CSV, Excel, or HTML.

Input

All fields live in one flat input object — an operation selector, the shared urls list, and a handful of operation-specific parameters that apply uniformly to every URL in that run. See the Input tab for the full schema with descriptions.

Field	Used by	Description
operation	all	urljoin | unfurl | replace_params | extract
urls	all	The URLs to process (relative URLs, for urljoin)
baseUrl	urljoin	Reference URL every entry in urls is resolved against
params	replace_params	Key/value pairs to add or overwrite, via a key-value editor (no JSON typing)
remove	replace_params	Keys to strip; supports fnmatch wildcards like utm_*
field	extract	domain | apex | subdomain | tld | path | keys | values | keypairs | format
format	extract (field=format)	Directive string, e.g. %s://%d%p?%q
unfurlDetail	unfurl	summary (default) or full (adds a node/edge count summary)
outputFormat	unfurl	Include a human-readable ASCII tree alongside the JSON graph

Output

Every run pushes one dataset item per input URL: { operation, input, result, error }. error is null on success, so a single malformed URL never aborts the run.

Examples

Each pair below is a complete Actor input alongside the corresponding dataset row it produces.

urljoin

Input:

{
  "operation": "urljoin",
  "urls": ["../c", "/d", "https://other.com/x"],
  "baseUrl": "https://example.com/a/b/"
}

Output (first row):

{
  "operation": "urljoin",
  "input": "../c",
  "result": "https://example.com/a/c",
  "error": null
}

unfurl

Input:

{
  "operation": "unfurl",
  "urls": ["https://sub.example.com/users/1?ts=1700000000&data=aGVsbG8="],
  "outputFormat": true
}

Output:

{
  "operation": "unfurl",
  "input": "https://sub.example.com/users/1?ts=1700000000&data=aGVsbG8=",
  "result": {
    "nodes": [
      { "id": 1, "type": "url", "key": null, "value": "https://sub.example.com/users/1?ts=1700000000&data=aGVsbG8=", "label": "https://sub.example.com/users/1?ts=1700000000&data=aGVsbG8=" },
      { "id": 2, "type": "url.scheme", "key": null, "value": "https", "label": "https" },
      { "id": 3, "type": "url.hostname", "key": null, "value": "sub.example.com", "label": "sub.example.com" },
      { "id": 6, "type": "url.query.param", "key": "ts", "value": "1700000000", "label": "ts: 1700000000" },
      { "id": 7, "type": "decoded.timestamp", "key": null, "value": "2023-11-14T22:13:20+00:00", "label": "2023-11-14T22:13:20+00:00" },
      { "id": 8, "type": "url.query.param", "key": "data", "value": "aGVsbG8=", "label": "data: aGVsbG8=" },
      { "id": 9, "type": "decoded.base64", "key": null, "value": "hello", "label": "hello" }
    ],
    "edges": [
      { "from": 1, "to": 2, "label": "url-parse" },
      { "from": 1, "to": 3, "label": "url-parse" },
      { "from": 1, "to": 6, "label": "query-split" },
      { "from": 6, "to": 7, "label": "epoch-decode" },
      { "from": 1, "to": 8, "label": "query-split" },
      { "from": 8, "to": 9, "label": "base64-decode" }
    ],
    "text": "[1] https://sub.example.com/users/1?ts=1700000000&data=aGVsbG8=\n├─(url-parse)─[2] https\n├─(url-parse)─[3] sub.example.com\n├─(query-split)─[6] ts: 1700000000\n│  └─(epoch-decode)─[7] 2023-11-14T22:13:20+00:00\n└─(query-split)─[8] data: aGVsbG8=\n   └─(base64-decode)─[9] hello"
  },
  "error": null
}

(path-segment nodes trimmed above for brevity — the full graph includes every path segment as its own node.)

replace_params

Input:

{
  "operation": "replace_params",
  "urls": ["https://example.com/?a=1&utm_source=x&gclid=z"],
  "params": [{ "key": "a", "value": "9" }],
  "remove": ["utm_*", "gclid"]
}

Output:

{
  "operation": "replace_params",
  "input": "https://example.com/?a=1&utm_source=x&gclid=z",
  "result": "https://example.com/?a=9",
  "error": null
}

extract

Input:

{
  "operation": "extract",
  "urls": ["https://sub.example.com/users/1?id=1"],
  "field": "format",
  "format": "%s://%d%p?%q"
}

Output:

{
  "operation": "extract",
  "input": "https://sub.example.com/users/1?id=1",
  "result": "https://sub.example.com/users/1?id=1",
  "error": null
}

Data table

Field	Type	Present when	Description
operation	string	always	Operation that produced this row
input	string	always	The original input URL
result	varies	on success	Absolute URL string (urljoin), node/edge object (unfurl), rewritten URL string (replace_params), or field value/array/string (extract)
error	string | null	always	Failure reason for this URL, or null

Pricing / cost estimation

URL Wrangler does pure in-memory string/URL processing — no network requests, no browser, no proxy usage. Cost is driven entirely by compute time, which is minimal (typically well under 100ms per URL). On the Apify Free plan, you can process tens of thousands of URLs per run within the platform's free monthly compute unit allowance.

Tips / advanced options

• For unfurl, leave outputFormat off unless you want the ASCII tree — it roughly doubles the payload size per row for large graphs.
• Base64/hex/timestamp/UUID decoding in unfurl recurses up to 5 levels deep (e.g. a query param that's itself Base64-encoded JSON containing another encoded value), capped to bound output size on adversarial input.
• replace_params's remove field supports fnmatch-style wildcards (utm_*, *_id), so you don't need to enumerate every tracking parameter by name.
• The naive apex/subdomain/tld split in extract and unfurl uses the last two DNS labels — it's correct for .com/.net/.org-style domains but not for multi-part public suffixes like .co.uk (that needs a full Public Suffix List, not currently bundled).

Background reading

The unfurl operation's node/edge tree is modeled on the DFIR community tool of the same name, and extract's field/format model is modeled on a separate, unrelated CLI tool. Related reading:

• obsidianforensics/unfurl — the DFIR tool this Actor's unfurl operation is modeled on
• SANS ISC: Unfurl v2025.02 released — write-up on unfurl's timestamp/IP/UUID decoding capabilities
• tomnomnom/unfurl — the flat field-extraction CLI this Actor's extract operation is modeled on
• Google Search Central: URL canonicalization — background on why URL normalization matters for deduplication
• Pinterest Engineering: Smarter URL Normalization at Scale (MIQPS) — a production system for deciding which query parameters are semantically significant

FAQ, limitations, and support

• This Actor only processes URLs you provide — it does not fetch, crawl, or follow redirects for any of them.
• unfurl's decoders are heuristic (they detect plausible Base64/hex/epoch/UUID values); false negatives on obfuscated data and, rarely, false positives on coincidentally-decodable strings are possible.
• Found a bug or want another operation (URL normalization/dedup, punycode/IDNA conversion, tracking-parameter presets)? Open an issue on the Actor's Issues tab — these are tracked as candidate follow-ups.

Data pipeline toolkit

Part of the Data pipeline toolkit — small, chainable Actors for cleaning, transforming, and generating data inside a larger pipeline:

• jq Helper – transform JSON with jq — Run jq programs over inline JSON or a linked Apify dataset.
• DuckDB Helper – SQL over CSV, JSON, Parquet, Excel, SQLite — Run a DuckDB SQL query over remote/local files, push results to a dataset.
• Regex Helper — Apply named regular expressions to strings, extract structured matches.
• ZIP Code Helper — Resolves US ZIP codes into city, state, county, and more.
• Postal Address Normaliser — Parses and normalises postal addresses using libpostal.
• Phone Number Wrangler — Validate, format, and parse phone numbers using libphonenumber.
• UUID Generator — Generate bulk UUIDs (v1, v3, v4, v5, v7) on demand.
• Secure Password & Passphrase Generator — Generate secure passwords and diceware passphrases per NIST guidance.
• Thumbnail Maker — Generates thumbnails from image URLs using ImageMagick.
• Katana Web Crawler (ProjectDiscovery) — Crawl websites with Katana, stream results as JSONL.
• ProjectDiscovery Notify — Stream records to Slack, Discord, Telegram, Email, and more.

Did you find this useful?

⭐ Rate this actor on Apify! Your feedback helps other users find it and helps us keep improving it.