Cookie Getter / Generator - Apify Actor

An Apify actor that fetches or generates new cookies and website network traffic for analysis. Create a list of websites where you want to fetch your cookies and network traffic from to analyze. Run the actor locally on your browser with debug to get your own data, or run in the cloud to generate the format and get new incognito cookies and network traffic.

Description

This actor uses Puppeteer to navigate to a specified URL and extracts all cookies, scroll and generate network data.

By default, the actor uses a fresh/incognito browser session (no personal cookies).

To fetch cookies from YOUR personal browser, you have two options:

Option 1: Connect to Your Running Browser (Recommended)

Connect the actor to your currently running Chrome browser using Chrome DevTools Protocol. This allows the actor to access all your browser cookies automatically.

Option 2: Use Your Browser Profile

Launch a browser using your Chrome user data directory/profile, which contains all your saved cookies.

Using Your Browser Cookies Locally:

1. Make sure all chrome tabs are closed using the command: killall "Google Chrome"
2. Validate: ps ax | grep -v grep | grep "chrome"
3. Start Chrome: npm run chrome-debug
4. Get endpoint: npm run get-endpoint
5. Add browserWSEndpoint to your INPUT.json
6. Run: npm start

The actor is useful for:

• Fetching cookies from your personal browser automatically
• Getting cookies from a fresh browser session
• Extracting session cookies after visiting a site
• Understanding what cookies a website sets

Input

The actor accepts the following input:

{
    "url": "https://example.com",
    "urls": ["https://example.com", "https://another-site.com"],
    "cookies": [
        {
            "name": "session_id",
            "value": "abc123",
            "domain": ".example.com",
            "path": "/",
            "httpOnly": true,
            "secure": true,
            "sameSite": "Lax"
        }
    ],
    "browserWSEndpoint": "ws://localhost:9222/devtools/browser/...",
}

Input Parameters

• url (required, unless urls: The URL of the website to fetch cookies from. Can be a string (single URL) or an array of strings (multiple URLs).
• urls (optional): Array of URLs to fetch cookies from. If provided, url is ignored. The actor will navigate to each URL sequentially and collect cookies from all of them.
• cookies (optional): Array of cookies to set before navigation. Useful for authenticated requests using cookies from your personal browser.
• browserWSEndpoint (optional): Chrome DevTools Protocol WebSocket endpoint to connect to your existing Chrome session. See Connecting to Your Browser section.
• userDataDir (optional): Path to Chrome user data directory to use your browser profile.
• useExistingPage (optional, default: true): If true and browserWSEndpoint is provided, uses an existing page from your browser session instead of creating a new one.

Output

The actor outputs a JSON object. The structure depends on the mode:

Single URL Mode (default)

{
    "mode": "single",
    "url": "https://example.com",
    "cookies": [
        {
            "name": "session_id",
            "value": "abc123",
            "domain": ".example.com",
            "path": "/",
            "expires": 1234567890,
            "httpOnly": true,
            "secure": true,
            "sameSite": "Lax"
        }
    ],
    "cookieCount": 1,
    "usedInputCookies": 0,
    "timestamp": "2024-01-01T00:00:00.000Z"
}

Multiple URLs Mode (when urls array is provided)

{
    "mode": "multiple",
    "urls": ["https://example.com", "https://another-site.com"],
    "results": [
        {
            "url": "https://example.com",
            "domain": "example.com",
            "cookies": [...],
            "cookieCount": 5,
            "finalUrl": "https://example.com/page"
        },
        {
            "url": "https://another-site.com",
            "domain": "another-site.com",
            "cookies": [...],
            "cookieCount": 3,
            "finalUrl": "https://another-site.com/home"
        }
    ],
    "totalUrls": 2,
    "successfulUrls": 2,
    "failedUrls": 0,
    "totalCookies": 8,
    "uniqueCookieCount": 8,
    "usedInputCookies": 0,
    "timestamp": "2024-01-01T00:00:00.000Z"
}

Accessing the Output

The output JSON is saved to Apify's key-value store with the key OUTPUT. You can access it in several ways:

1. Apify Console:

• Go to your actor run details page
• Click on the "Key-value store" tab
• Open the "OUTPUT" key to view/download the JSON

2. Apify API:

# Get the store ID from the run details, then:
curl "https://api.apify.com/v2/key-value-stores/{STORE_ID}/records/OUTPUT?token=YOUR_API_TOKEN"

3. Programmatically:

const { Actor } = require('apify');
const output = await Actor.getValue('OUTPUT');
console.log(output);

Usage

Local Development

Quick Start:

# 1. Install dependencies
npm install

# 2. Create input file
mkdir -p .storage/key_value_stores/default
echo '{"url": "https://bbc.com"}' > .storage/key_value_stores/default/INPUT.json

# 3. Run the actor
npm start

# 4. View output
cat .storage/key_value_stores/default/OUTPUT.json

Detailed Instructions: See LOCAL_RUN.md for complete local setup guide.

Using Your Browser Cookies Locally:

1. Start Chrome: npm run chrome-debug
2. Get endpoint: npm run get-endpoint
3. Add browserWSEndpoint to your INPUT.json
4. Run: npm start

Fetching Cookies from Multiple URLs:

{
    "urls": [
        "https://example.com",
        "https://another-site.com",
        "https://third-site.com"
    ],
    "browserWSEndpoint": "ws://localhost:9222/devtools/browser/..."
}

The actor will navigate to each URL sequentially and collect cookies from all of them. Results are organized by URL in the output.

Apify Platform

1. Push the actor to Apify:

$apify push

2. Run the actor from the Apify console or via API with the input:

{
    "url": "https://example.com"
}

Fetching Cookies from Your Personal Browser

The actor can automatically fetch cookies from your personal browser (Chrome, Safari, or iOS Safari). Choose one of these methods:

Method 1: Take Over Your Chrome Session (Easiest) ⭐

This connects the actor to your currently running Chrome browser session. The actor will take control of your browser, navigate to the website, and fetch cookies from that session.

Step 1: Start Chrome with Remote Debugging

Easy way (using helper script):

$npm run chrome-debug

Manual way:

On macOS:

$/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --remote-debugging-port=9222 --user-data-dir=/tmp/chrome-debug

On Linux:

$google-chrome --remote-debugging-port=9222 --user-data-dir=/tmp/chrome-debug

On Windows:

"C:\Program Files\Google\Chrome\Application\chrome.exe" --remote-debugging-port=9222 --user-data-dir="C:\temp\chrome-debug"

Step 2: Get the WebSocket Endpoint

Easy way (using helper script):

$npm run get-endpoint

This will automatically find and display the WebSocket endpoint.

Manual way:

1. Open http://localhost:9222/json/version in your browser
2. Copy the webSocketDebuggerUrl value (looks like: ws://localhost:9222/devtools/browser/...)

Step 3: Use in Actor Input

{
    "url": "https://bbc.com",
    "browserWSEndpoint": "ws://localhost:9222/devtools/browser/...",
    "useExistingPage": true
}

What happens:

• The actor connects to your existing Chrome browser
• It uses your current browser session (with all your cookies already loaded)
• It navigates to the specified URL in your browser
• It extracts all cookies from that navigation
• Your browser window will be visible and controlled by the actor

Note: Set useExistingPage: false if you want the actor to create a new tab instead of using an existing one.

For Cloud Users (Simplified):

If you're running the actor on Apify cloud, use the Cookie Extractor method below instead - it's much simpler!

Method 2: Cookie Extractor (Simplest for Cloud Users) ⭐

This is the easiest method for cloud users - no tunneling required!

Step 1: Extract Cookies Using the Web Tool

1. Open cookie-extractor.html in your browser (or run the local server):

   npm run cookie-server
   # Then visit: http://localhost:3000
   

2. Or simply open cookie-extractor.html directly in your browser

3. Visit the website you want cookies from (e.g., https://bbc.com)

4. Open the cookie extractor page and click "Extract Cookies from Current Domain"

5. Copy the JSON output

Step 2: Use in Actor Input

{
    "url": "https://bbc.com",
    "cookies": [
        {
            "name": "cookie_name",
            "value": "cookie_value",
            "domain": ".bbc.com",
            "path": "/"
        }
    ]
}

That's it! No tunneling, no complex setup. Just extract and paste.

Method 3: Use Your Browser Profile

Launch the browser using your Chrome user data directory:

Find Your Chrome Profile Path:

• macOS: ~/Library/Application Support/Google/Chrome
• Windows: C:\Users\YourUsername\AppData\Local\Google\Chrome\User Data
• Linux: ~/.config/google-chrome

Use in Actor Input:

{
    "url": "https://bbc.com",
    "userDataDir": "/path/to/your/chrome/profile"
}

Note: This requires the profile to be accessible from where the actor runs (works locally, but not on Apify cloud unless you upload the profile).

Method 4: iOS Safari (iPhone/iPad) 📱

Option A: Without USB Connection (Easiest)

Extract cookies directly on your iOS device:

Step 1: Open the Cookie Extractor on iOS

1. Host ios-cookie-extractor.html on a web server (or use a file sharing service)
2. On your iOS device, open Safari
3. Navigate to the website you want cookies from (e.g., https://bbc.com)
4. Open the cookie extractor page
5. Click "Extract Cookies from Current Domain"
6. Copy the JSON output
7. Use it in your actor input

Quick Setup (Same WiFi Network):

# On your computer (same WiFi as iOS device)
python3 -m http.server 8000 --bind 0.0.0.0

# Find your computer's IP, then on iOS visit:
# http://YOUR_IP:8000/ios-cookie-extractor.html

See ios-simple-guide.md for detailed instructions.

Option B: With USB Connection (For httpOnly Cookies)

To fetch ALL cookies including httpOnly ones (requires Mac + USB):

Prerequisites:

• Mac computer (required for Safari Web Inspector)
• iOS device connected via USB
• ios-webkit-debug-proxy installed

Step 1: Enable Web Inspector on iOS

1. On your iOS device: Settings → Safari → Advanced → Enable Web Inspector

Step 2: Connect Device to Mac

1. Connect your iOS device to your Mac via USB
2. Trust the computer on your device

Step 3: Set Up Proxy

# Install ios-webkit-debug-proxy (if not installed)
brew install ios-webkit-debug-proxy

# Start the proxy
npm run ios-setup
# Or manually: ios-webkit-debug-proxy

Step 4: Open Safari on iOS

1. Open Safari on your iOS device
2. Navigate to the website you want cookies from

Step 5: Get WebSocket Endpoint

$npm run get-endpoint

This will show the WebSocket endpoint for your iOS Safari (via the proxy).

Step 6: Use in Actor Input

{
    "url": "https://bbc.com",
    "browserWSEndpoint": "ws://localhost:9222/devtools/page/..."
}

Note: For cloud runs, you'll need to tunnel the connection (see Method 1 cloud setup).

See ios-setup.md for detailed instructions.

Method 5: Manual Cookie Extraction (Fallback)

If you can't use the above methods, you can run extract and pass cookies:

1. Extract Cookies from Your Browser

Chrome/Edge:

1. Open Developer Tools (F12)
2. Go to Application tab → Cookies
3. Select the website domain
4. Copy cookies as JSON or manually create the array

Using Browser Extension:

• Use extensions like "Cookie-Editor" or "EditThisCookie" to export cookies as JSON

Using JavaScript Console:

// Run this in the browser console on the target website
JSON.stringify(document.cookie.split(';').map(c => {
    const [name, value] = c.trim().split('=');
    return { name, value, domain: window.location.hostname, path: '/' };
}), null, 2)

2. Pass Cookies to the Actor

Include the cookies array in your input:

{
    "url": "https://example.com",
    "cookies": [
        {
            "name": "auth_token",
            "value": "your_token_here",
            "domain": ".example.com",
            "path": "/",
            "httpOnly": true,
            "secure": true,
            "sameSite": "Lax"
        }
    ]
}

3. The Actor Will:

• Set your cookies before navigation
• Navigate to the URL with your authentication
• Extract all cookies (including any new ones set during the visit)
• Return both your input cookies and any new cookies

Requirements

• browserWSEndpoint or userDataDir must be provided (needs access to your browser profile)
• Chrome browser with cookies stored

Usage

Step 1: Connect to Your Browser

Start Chrome with remote debugging:

$npm run chrome-debug

Step 2: Get the WebSocket Endpoint

$npm run get-endpoint

Step 3: Use in Actor Input

{
    "fetchAllFromHistory": true,
    "browserWSEndpoint": "ws://localhost:9222/devtools/browser/..."
}

Note: The url field is optional when fetchAllFromHistory is true. The historyLimit parameter is not used (kept for backward compatibility).

What It Does

1. Connects to your browser session
2. Fetches all cookies from your browser
3. Extracts unique domains from the cookies
4. Organizes cookies by domain
5. Returns a structured output with cookies grouped by domain

Output Structure

The output includes:

• totalDomains: Number of unique domains found from cookies
• domainsWithCookies: Number of domains that have cookies
• totalCookies: Total number of cookies found
• cookiesByDomain: Object mapping each domain to its cookies
• allCookies: Flat array of all cookies (for convenience)

Example Output

{
    "mode": "history",
    "totalDomains": 45,
    "domainsWithCookies": 45,
    "totalCookies": 234,
    "cookiesByDomain": {
        "example.com": {
            "domain": "example.com",
            "cookies": [...],
            "cookieCount": 3
        }
    },
    "allCookies": [...],
    "note": "Domains extracted from cookies in browser. Chrome DevTools Protocol does not provide access to browser history database."
}

Use Cases

• Export all your browser cookies for backup
• Analyze which websites have cookies stored
• Transfer cookies to another browser or tool
• Audit your cookie usage across all domains
• Get cookies from all websites you've visited (that have cookies)

Notes

• The actor waits for the page to fully load before extracting cookies
• Some cookies may be set after page load, so the actor waits an additional 2 seconds
• Cookies are extracted from the browser's cookie store after navigation
• The actor runs in headless mode by default
• If you provide cookies, they will be set before navigation for authenticated requests
• Cookie format must match Puppeteer's cookie format (name, value, domain are required; path, expires, httpOnly, secure, sameSite are optional)