# Sherlock Scraper (`automation-lab/sherlock-scraper`) Actor Check public username/profile existence across hundreds of Sherlock catalog sites. Export found, not-found, and unknown profile checks. - **URL**: https://apify.com/automation-lab/sherlock-scraper.md - **Developed by:** [Stas Persiianenko](https://apify.com/automation-lab) (community) - **Categories:** Developer tools, Social media, Lead generation - **Stats:** 6 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per event This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Sherlock Scraper Find public username profiles across hundreds of websites using the open-source Sherlock site catalog. Sherlock Scraper is an Apify actor for OSINT, brand protection, fraud review, recruiting enrichment, and username availability monitoring. Give it one or more usernames and it checks profile URL patterns from the Sherlock catalog with conservative HTTP rules. The output tells you which checks were found, not found, or uncertain. ### What does Sherlock Scraper do? Sherlock Scraper checks whether a username appears to have a public profile on many social networks, developer communities, forums, creator platforms, and directories. It returns one dataset row per username and site. Each row includes the checked URL, HTTP status, classification rule, response time, and timestamp. The actor is best-effort by design. Some websites block automated checks or return ambiguous redirects. Those rows are marked `unknown` rather than guessed as found or not found. ### Who is it for? #### OSINT researchers Use it to quickly map where a handle might exist before deeper manual investigation. #### Fraud and risk teams Check whether a claimed username has a broad cross-site footprint or appears only on one suspicious source. #### Brand protection teams Monitor usernames that match your brand, executive names, campaign names, or product handles. #### Recruiting and sales enrichment teams Enrich a known handle with public profile URLs from developer and community sites. #### Creators and agencies Check availability patterns before launching a new brand or creator identity. ### Why use this actor? - ๐Ÿ”Ž Batch username checks in one Apify run. - ๐Ÿงญ Uses the public Sherlock catalog, not a tiny hand-built list. - ๐Ÿงพ Returns evidence fields, not just a yes/no answer. - ๐Ÿ›ก๏ธ Marks uncertain checks as `unknown` to reduce false positives. - โš™๏ธ Lets you limit sites, concurrency, and timeouts. - ๐Ÿ“ฆ Exports cleanly to JSON, CSV, Excel, Google Sheets, and API workflows. ### What data can you extract? | Field | Description | | --- | --- | | `username` | Normalized queried username | | `site` | Sherlock catalog site name | | `url` | Public profile URL checked | | `exists` | `true`, `false`, or `null` for unknown | | `status` | `found`, `not_found`, or `unknown` | | `statusCode` | HTTP status code when available | | `matchedBy` | Rule used for classification | | `responseTimeMs` | Request duration in milliseconds | | `checkedAt` | ISO timestamp for the check | | `error` | Request error details when a site could not be checked | ### How much does it cost to check usernames with Sherlock? This actor uses pay-per-event pricing. You pay a small start fee per run and a per-result fee for each username/site row saved to the dataset. Keep the first test cheap by using the default prefill or a small `maxSites` value. For broad OSINT scans, increase `maxSites` after confirming that the output format fits your workflow. ### How to use Sherlock Scraper 1. Open the actor on Apify. 2. Add one or more usernames in `usernames`. 3. Optionally add site filters such as `GitHub`, `Reddit`, `Medium`, or `YouTube`. 4. Choose `maxSites` for the breadth of the scan. 5. Keep `includeNotFound` enabled when you need a full audit trail. 6. Disable it or set `onlyFound` when you only want likely profile URLs. 7. Run the actor. 8. Export the dataset or consume it through the Apify API. ### Input options #### `usernames` Required list of usernames or handles. Leading `@` characters are removed automatically. Examples: - `torvalds` - `github` - `apify` #### `sites` Optional list of Sherlock site names or partial names. If empty, the actor checks the first matching sites up to `maxSites`. Examples: - `GitHub` - `Reddit` - `Telegram` - `YouTube` - `Medium` #### `maxSites` Maximum catalog sites checked per username. Use a low number for testing and a higher number for broader monitoring. #### `includeNotFound` When enabled, you get every attempted check. When disabled, only `found` rows are saved. #### `onlyFound` Compatibility shortcut for found-only exports. If this is true, it overrides `includeNotFound`. #### `includeNsfw` Include sites marked NSFW by the upstream Sherlock catalog. Default is false. #### `concurrency` Number of HTTP checks running at the same time. The default is conservative to avoid noisy target rate limits. #### `timeoutSecs` Maximum time to wait for each site before returning `unknown`. ### Example input ```json { "usernames": ["torvalds", "github"], "sites": ["GitHub", "Reddit", "Medium", "YouTube", "Telegram"], "maxSites": 25, "includeNotFound": true, "concurrency": 5, "timeoutSecs": 12 } ```` ### Example output ```json { "username": "torvalds", "site": "GitHub", "url": "https://www.github.com/torvalds", "exists": true, "status": "found", "statusCode": 200, "matchedBy": "http_status", "responseTimeMs": 421, "checkedAt": "2026-06-25T00:00:00.000Z", "error": null } ``` ### Understanding `unknown` results `unknown` does not mean the username is absent. It means the site did not provide a reliable public signal during this run. Common reasons include: - temporary target outage, - DNS or TLS failure, - anti-bot response, - rate limit, - ambiguous redirect, - unusual status code. For investigations, sort by `status` and manually review important `unknown` URLs. ### Tips for accurate scans - Start with a small list of high-value sites. - Keep concurrency moderate for broad scans. - Re-run important unknown rows later. - Treat found rows as leads, not legal identity proof. - Keep not-found rows when you need audit evidence. - Use `sites` filters for brand monitoring workflows. ### Integrations #### Google Sheets Export the dataset to CSV or use the Apify Google Sheets integration to append found profiles to a monitoring sheet. #### Webhooks Trigger a webhook when a run finishes and send found profile rows into Slack, Make, Zapier, or your case-management tool. #### CRM enrichment Use usernames from CRM records and append public profile URL candidates back to the contact record. #### Brand monitoring Schedule daily or weekly runs for important handles and compare the resulting dataset with a previous run. ### API usage #### Node.js ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('automation-lab/sherlock-scraper').call({ usernames: ['torvalds'], sites: ['GitHub', 'Reddit'], maxSites: 10 }); console.log(run.defaultDatasetId); ``` #### Python ```python from apify_client import ApifyClient client = ApifyClient('YOUR_APIFY_TOKEN') run = client.actor('automation-lab/sherlock-scraper').call(run_input={ 'usernames': ['torvalds'], 'sites': ['GitHub', 'Reddit'], 'maxSites': 10, }) print(run['defaultDatasetId']) ``` #### cURL ```bash curl -X POST 'https://api.apify.com/v2/acts/automation-lab~sherlock-scraper/runs?token=YOUR_APIFY_TOKEN' \ -H 'Content-Type: application/json' \ -d '{"usernames":["torvalds"],"sites":["GitHub","Reddit"],"maxSites":10}' ``` ### MCP usage Use the Apify MCP server with: ```text https://mcp.apify.com?tools=automation-lab/sherlock-scraper ``` Claude Code setup: ```bash claude mcp add apify-sherlock "https://mcp.apify.com?tools=automation-lab/sherlock-scraper" ``` Claude Desktop JSON config: ```json { "mcpServers": { "apify-sherlock": { "url": "https://mcp.apify.com?tools=automation-lab/sherlock-scraper" } } } ``` Example prompts: - "Check whether the username torvalds exists on GitHub, Reddit, and Medium." - "Run a weekly Sherlock username monitor for our brand handles." - "Return only found public profile URLs for these candidate usernames." ### Scheduling You can schedule this actor in Apify for recurring monitoring. Useful schedules include: - daily checks for brand impersonation, - weekly OSINT enrichment for a watchlist, - monthly username availability checks before product launches. ### Data quality notes The actor uses public HTTP signals. No automated username checker can guarantee identity ownership. A found profile URL only means that a public page likely exists at that username pattern. Review high-risk results manually before making decisions. ### Legality and ethical use Only use this actor for lawful workflows and public information. Do not use it for harassment, doxxing, credential attacks, or violating platform terms. Respect local privacy laws and your organization's compliance requirements. ### Attribution This actor consumes the public site catalog from the open-source Sherlock project. Sherlock is an MIT-licensed OSINT project. This actor is not affiliated with or endorsed by the Sherlock project. ### FAQ #### Is Sherlock Scraper affiliated with the Sherlock project? No. This actor uses the public Sherlock site catalog and is not affiliated with or endorsed by the upstream project. #### Can this prove who owns a username? No. It finds public profile URL candidates. Treat results as leads for review, not identity proof. ### Troubleshooting #### Why are many rows `unknown`? Some sites block automated checks, return ambiguous redirects, or time out. Lower concurrency, narrow the `sites` list, or re-run later. #### Why did a known profile return not found? Sites change profile URLs and anti-bot rules. Check the returned `url`, `statusCode`, and `matchedBy` fields, then retry the specific site. #### Why did I get no rows? If `onlyFound` is true and no profiles are classified as found, the dataset can be empty. Enable `includeNotFound` for a full audit trail. ### Related scrapers You may also find these automation-lab actors useful: - https://apify.com/automation-lab/domain-availability-checker - https://apify.com/automation-lab/website-uptime-checker - https://apify.com/automation-lab/bulk-url-status-checker - https://apify.com/automation-lab/robots-sitemap-analyzer - https://apify.com/automation-lab/structured-data-extractor ### Changelog Initial version: batch username checking across the Sherlock public catalog with found, not-found, and unknown states. # Actor input Schema ## `usernames` (type: `array`): One or more usernames or handles to check. Leading @ signs are removed automatically. ## `sites` (type: `array`): Optional Sherlock site names to check. Matching is case-insensitive and accepts partial names such as GitHub, Reddit, Medium, or YouTube. ## `maxSites` (type: `integer`): Maximum number of matched catalog sites to check per username. Leave low for quick tests, raise for broader OSINT scans. ## `includeNotFound` (type: `boolean`): When enabled, the dataset includes every attempted username/site check. Disable to output only profiles classified as found. ## `onlyFound` (type: `boolean`): Compatibility alias for disabling not-found rows. If true, it overrides includeNotFound. ## `includeNsfw` (type: `boolean`): Include sites marked NSFW by the upstream Sherlock catalog. ## `concurrency` (type: `integer`): Number of profile URLs checked at the same time. Conservative defaults reduce target rate-limit noise. ## `timeoutSecs` (type: `integer`): Maximum seconds to wait for each target profile page before marking the check as unknown. ## Actor input object example ```json { "usernames": [ "torvalds", "github" ], "sites": [ "GitHub", "Reddit", "Medium", "YouTube", "Telegram" ], "maxSites": 20, "includeNotFound": true, "onlyFound": false, "includeNsfw": false, "concurrency": 5, "timeoutSecs": 12 } ``` # Actor output Schema ## `overview` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "usernames": [ "torvalds", "github" ], "sites": [ "GitHub", "Reddit", "Medium", "YouTube", "Telegram" ], "maxSites": 20, "includeNotFound": true, "onlyFound": false, "includeNsfw": false, "concurrency": 5, "timeoutSecs": 12 }; // Run the Actor and wait for it to finish const run = await client.actor("automation-lab/sherlock-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`๐Ÿ’พ Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // ๐Ÿ“š Want to learn more ๐Ÿ“–? Go to โ†’ https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "usernames": [ "torvalds", "github", ], "sites": [ "GitHub", "Reddit", "Medium", "YouTube", "Telegram", ], "maxSites": 20, "includeNotFound": True, "onlyFound": False, "includeNsfw": False, "concurrency": 5, "timeoutSecs": 12, } # Run the Actor and wait for it to finish run = client.actor("automation-lab/sherlock-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("๐Ÿ’พ Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # ๐Ÿ“š Want to learn more ๐Ÿ“–? Go to โ†’ https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "usernames": [ "torvalds", "github" ], "sites": [ "GitHub", "Reddit", "Medium", "YouTube", "Telegram" ], "maxSites": 20, "includeNotFound": true, "onlyFound": false, "includeNsfw": false, "concurrency": 5, "timeoutSecs": 12 }' | apify call automation-lab/sherlock-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=automation-lab/sherlock-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Sherlock Scraper", "description": "Check public username/profile existence across hundreds of Sherlock catalog sites. Export found, not-found, and unknown profile checks.", "version": "0.1", "x-build-id": "K8CoGnLJzHB1tF5i0" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/automation-lab~sherlock-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-automation-lab-sherlock-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/automation-lab~sherlock-scraper/runs": { "post": { "operationId": "runs-sync-automation-lab-sherlock-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/automation-lab~sherlock-scraper/run-sync": { "post": { "operationId": "run-sync-automation-lab-sherlock-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "usernames" ], "properties": { "usernames": { "title": "Usernames", "minItems": 1, "maxItems": 100, "type": "array", "description": "One or more usernames or handles to check. Leading @ signs are removed automatically.", "items": { "type": "string" } }, "sites": { "title": "Site filters", "type": "array", "description": "Optional Sherlock site names to check. Matching is case-insensitive and accepts partial names such as GitHub, Reddit, Medium, or YouTube.", "items": { "type": "string" } }, "maxSites": { "title": "Maximum sites per username", "minimum": 1, "maximum": 481, "type": "integer", "description": "Maximum number of matched catalog sites to check per username. Leave low for quick tests, raise for broader OSINT scans.", "default": 20 }, "includeNotFound": { "title": "Include not-found and unknown rows", "type": "boolean", "description": "When enabled, the dataset includes every attempted username/site check. Disable to output only profiles classified as found.", "default": true }, "onlyFound": { "title": "Only output found profiles", "type": "boolean", "description": "Compatibility alias for disabling not-found rows. If true, it overrides includeNotFound.", "default": false }, "includeNsfw": { "title": "Include NSFW catalog sites", "type": "boolean", "description": "Include sites marked NSFW by the upstream Sherlock catalog.", "default": false }, "concurrency": { "title": "Concurrent checks", "minimum": 1, "maximum": 25, "type": "integer", "description": "Number of profile URLs checked at the same time. Conservative defaults reduce target rate-limit noise.", "default": 5 }, "timeoutSecs": { "title": "Timeout per site", "minimum": 3, "maximum": 60, "type": "integer", "description": "Maximum seconds to wait for each target profile page before marking the check as unknown.", "default": 12 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```