Pastebin Scraper

Collect public paste URLs and text from Pastebin, GitHub Gist, Ideone, and Paste.org. Search keywords, extract regex matches, return optional full text and metadata, and continue interrupted collections. Textbin is experimental and is reported as unavailable when its public pages cannot be reached.

Collection uses public HTML pages and linked raw documents. No paid search/data API, source API key, or AI service is required. Apify proxies help with access; source outages and incomplete searches are reported explicitly.

Use cases

• Monitor public mentions of your brand, domains, project names, or other keywords.
• Collect public code and text for research, indexing, and downstream analysis.
• Extract structured indicators with your own regex patterns.
• Build a reusable history of paste URLs and re-fetch current contents when live discovery is unavailable or finds no matches.

Quick start

Run the prefilled Input in Apify Console. It collects up to three live URLs from the four supported discovery sources without downloading paste contents. Export results from the default dataset as JSON, CSV, Excel, or another supported format.

The preview caps public web attempts at ten and disables cross-run URL deduplication so repeated previews can return current URLs. Enable urlIndexDeduplicate for cumulative indexing. API clients omitting sources retain Pastebin-only discovery and the larger request budgets listed below.

Input examples

Collect and search content across several sites:

{
  "sources": ["pastebin", "github_gist", "ideone", "pasteorg"],
  "runMode": "scrape",
  "keywords": ["example.com", "your-brand"],
  "maxPastesPerRun": 25,
  "maxResults": 5,
  "includeRawText": true,
  "regexPatterns": ["[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}"],
  "useUrlIndex": false
}

Keywords use OR matching against fetched text, ignoring case unless keywordCaseSensitive is enabled. Regex extracts values from eligible pastes; it does not filter which pastes are saved. Empty keywords accept all successfully fetched text documents.

To collect known pastes, provide startUrls as objects containing url. Direct URLs infer their source and skip discovery. Existing pasteUrls remains available for Pastebin URLs and can be combined with startUrls.

{
  "runMode": "url_index",
  "sources": ["pastebin", "github_gist", "ideone", "pasteorg"],
  "maxPastesPerRun": 20,
  "urlIndexMaxRecordsPerRun": 20,
  "urlIndexDeduplicate": true,
  "urlIndexRecentIdLimit": 1000,
  "urlIndexSaveToDefaultDataset": true
}

URL-only mode appends lightweight records to urlIndexDatasetName and, by default, the run dataset. URL discovery does not verify that content is still available. The history reaches back only to URLs you previously collected.

Source coverage

Source	Discovery and collection
Pastebin	Public archive, homepage fallback, selected syntax archives, broader language archives, direct URLs.
GitHub Gist	Public Discover and HTML search pages; direct Gists and linked raw files. One Gist produces one result row.
Ideone	Public Recent pages, direct code pages and plain-text downloads. Existing code is collected without executing it.
Paste.org	Public Trending pages, direct paste pages and linked raw documents.
Textbin	Experimental public-page adapter; availability and extraction must be confirmed in the source report.

Optional searchEngineDiscovery adds best-effort URLs from public search-result HTML. Those URLs are fetched before content records are emitted; search snippets are never presented as verified paste content. Search engines may block or omit results.

Limits and advanced inputs

Input	Meaning / default
runMode	scrape for text; url_index for URLs.
sources	Discovery sites; omitted API input defaults to ["pastebin"].
maxPastesPerRun, maxResults	Live URLs selected and content results saved; defaults 10 and 3.
includeRawText, fetchDetailMetadata	Full text and optional extra metadata requests; both off by default.
maxFilesPerGist	At most 20 files per Gist by default, sharing the per-document byte cap.
maxRunTimeSecs	240-second collection budget, with shutdown time reserved before the platform timeout.
maxRequestsPerRun	200 public web attempts, including redirects and retries.
maxDiscoveryTimeSecs, maxDiscoveryRequests	Discovery is capped at 45 seconds and 20 attempts. Available documents can be processed before discovery finishes.
maxPagesPerSource	Pagination depth, default 3.
discoveryMode, syntaxArchives, maxSyntaxArchivesToScan	Existing Pastebin recent/expanded/deep controls. Automatic expansion requires autoExpandSyntaxArchives.
useUrlIndex, maxUrlIndexEntriesToSearch	Bounded historical-URL fallback, default enabled and capped at 100 entries. Direct URL input skips this fallback.
noMatchStopAfterPastes, stopAfterPastesWithoutNewMatch	Sparse-search guards, default 10 checks each; applied separately to live and historical phases. Set to 0 to disable.
urlIndexDatasetName, urlIndexStateStoreName	Named URL history and duplicate-suppression storage.
urlIndexRecentIdLimit	Number of recent URL identities remembered; increase deliberately for cumulative indexes.
saveNoMatchSummary	Save search diagnostics outside the paid dataset.

Raw documents are bounded at 1 MB by default; maxRawTextBytes supports up to 5 MB. HTML is bounded at 2 MB, with truncated coverage marked incomplete. Oversized or binary files are skipped. Use 128 MB for the small URL preview and 256 MB for multi-source content collection, at the default concurrency of four.

Output and coverage reports

Each content row includes existing fields such as url, paste_id, raw_text_preview, raw_text_length, matched_keywords, and regex_matches_flat. New fields identify the source and completeness:

{
  "record_type": "paste",
  "source_site": "github_gist",
  "native_id": "abcdef1234567890",
  "record_id": "github_gist:abcdef1234567890",
  "url": "https://gist.github.com/example/abcdef1234567890",
  "paste_id": null,
  "content_fetched": true,
  "content_complete": true,
  "matched_keywords": ["example.com"],
  "files_discovered": 2,
  "files_fetched": 2
}

This is an illustrative excerpt. Multi-file Gists include a files list with filenames, raw links, previews, fetch status, lengths and content hashes. Full text is included only when requested. Keywords and regex operate within each file, preventing matches across artificial file boundaries. Extra or failed files make coverage incomplete.

content_sha256 hashes collected text. Check content_complete before using it as a complete-document hash. Identical content on different source URLs retains distinct source records. Unknown metadata remains null. URL-only records have content_fetched: false and content_complete: null.

The Output tab links to:

• RUN_SUMMARY: complete, partial, or failed; per-source counts and failures; request metrics; stop reasons; and remaining work.
• SEARCH_SUMMARY: a genuine zero-match result or an explicitly incomplete search. Failed requests cannot establish that no matches exist.
• CHECKPOINT: saved progress for continuation.

A failed source does not discard successful records from other sources. If no document can be checked because collection fails entirely, the run fails. Controlled time, request, result or charge limits produce partial results and preserve pending work. Reports describe bounded selected coverage, never the entire history of a website.

Continue an interrupted run

Run again with the same source, direct URL, keyword and extraction settings, adding resumeFromRunId with the earlier run ID. The earlier run must belong to this Actor and be terminal. Per-run time, request, scan and result limits may change.

A continuation delivers new records into its own dataset; prior_runs links the collection chain. Migration/restart recovery within a run is automatic. Already delivered records are reconciled before resuming. If an interrupted write cannot be confirmed, the Actor stops with output_write_uncertain instead of blindly repeating a potentially billed write.

Pricing

$0.75 per 1,000 default-dataset records, plus the platform’s $0.00005 Actor-start event per GB, minimum one event. A three-record preview is $0.00230 in Actor events. One multi-file Gist is one billed record. Content and URL-only rows use the same rate; a named-index copy adds no second result event. Diagnostic summaries and checkpoints are not billed as results.

Set a maximum run charge in Console or through the API. Network/time limits also bound unsuccessful searches, which may legitimately return no matches. The current Pricing tab is authoritative.

Python API example

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run_input = {
    "sources": ["pastebin", "github_gist", "ideone", "pasteorg"],
    "runMode": "scrape",
    "keywords": ["example.com"],
    "maxPastesPerRun": 25,
    "maxResults": 5,
    "includeRawText": True,
    "useUrlIndex": False,
}
run = client.actor("thescrapelab/pastebin-osint-scraper").call(
    run_input=run_input,
    max_total_charge_usd=0.02,
    memory_mbytes=256,
)
if run is None:
    raise RuntimeError("The Actor did not return a run")
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["source_site"], item["url"], item.get("raw_text_preview"))

# To continue pending work later, reuse run_input and add:
# run_input["resumeFromRunId"] = run["id"]

Limitations and troubleshooting

Public listings and indexed results do not provide a complete historical database. Private, removed, expired, password-protected or inaccessible content cannot be collected. Site changes, HTTP 403/503 responses and rate limits can reduce coverage; inspect the per-source report rather than treating partial results as a complete search.

Use direct URLs for known documents, target relevant syntax archives, and increase budgets deliberately. No login, paid data API or API-based backup is used when a source is unavailable. For support, open an issue in this Actor’s Issues tab with the run URL and expected behavior.