Meta Tags Scraper

HTML pages may contain <meta> tags with metadata related to search engine indexing, page snippets and third party software integration. This metadata hidden within web pages is relevant for content performance within search engines and enhances link presentability on social media.

Input

Multiple input URLs are supported.

{
  "urls": [
    {"url": "https://stackoverflow.com/questions/5015483/test-if-an-attribute-is-present-in-a-tag-in-beautifulsoup"},
    {"url": "https://developers.google.com/search/docs/crawling-indexing/special-tags"}
  ]
}

Output

[{
  "url": "https://stackoverflow.com/questions/5015483/test-if-an-attribute-is-present-in-a-tag-in-beautifulsoup",
  "meta_name": "viewport",
  "content": "width=device-width, height=device-height, initial-scale=1.0, minimum-scale=1.0"
},
{
  "url": "https://stackoverflow.com/questions/5015483/test-if-an-attribute-is-present-in-a-tag-in-beautifulsoup",
  "meta_name": "twitter:card",
  "content": "summary"
},
{
  "url": "https://stackoverflow.com/questions/5015483/test-if-an-attribute-is-present-in-a-tag-in-beautifulsoup",
  "meta_name": "twitter:domain",
  "content": "stackoverflow.com"
},
{
  "url": "https://stackoverflow.com/questions/5015483/test-if-an-attribute-is-present-in-a-tag-in-beautifulsoup",
  "meta_name": "twitter:title",
  "content": "Test if an attribute is present in a tag in BeautifulSoup"
},
{
  "url": "https://stackoverflow.com/questions/5015483/test-if-an-attribute-is-present-in-a-tag-in-beautifulsoup",
  "meta_name": "twitter:description",
  "content": "I would like to get all the <script> tags in a document and then process each one based on the presence (or absence) of certain attributes.\n\nE.g., for each <script> tag, if the attribut..."
},
{
  "url": "https://developers.google.com/search/docs/crawling-indexing/special-tags",
  "meta_name": "google-signin-client-id",
  "content": "721724668570-nbkv1cfusk7kk4eni4pjvepaus73b13t.apps.googleusercontent.com"
},
{
  "url": "https://developers.google.com/search/docs/crawling-indexing/special-tags",
  "meta_name": "google-signin-scope",
  "content": "profile email https://www.googleapis.com/auth/developerprofiles https://www.googleapis.com/auth/developerprofiles.award"
},
{
  "url": "https://developers.google.com/search/docs/crawling-indexing/special-tags",
  "meta_name": "theme-color",
  "content": "#fff"
},
{
  "url": "https://developers.google.com/search/docs/crawling-indexing/special-tags",
  "meta_name": "X-UA-Compatible",
  "content": "IE=Edge"
},
{
  "url": "https://developers.google.com/search/docs/crawling-indexing/special-tags",
  "meta_name": "viewport",
  "content": "width=device-width, initial-scale=1"
},
{
  "url": "https://developers.google.com/search/docs/crawling-indexing/special-tags",
  "meta_name": "description",
  "content": "Google supports both page-level meta tags and tag attributes. Explore a list of the special tags that Google Search can support."
},
{
  "url": "https://developers.google.com/search/docs/crawling-indexing/special-tags",
  "meta_name": "twitter:card",
  "content": "summary_large_image"
},
{
  "url": "https://developers.google.com/search/docs/crawling-indexing/special-tags",
  "meta_name": "apple-itunes-app",
  "content": "app-id=1512522198"
}]

n8n integration

Use the n8n-nodes-meta-tags-scraper community node (source) to run this Actor directly from an n8n workflow.

See Also

• meta tags and attributes that Google supports - Google developer portal
• <meta>: The metadata element - Mozilla developer portal
• The Open Graph Protocol

Cyber recon toolkit

Part of the Cyber recon toolkit — OSINT, attack-surface recon, and vulnerability-intel data collection for security research:

• Bugcrowd Scraper — Scrapes public Bugcrowd data: disclosed submissions, program directory, leaderboard.
• HackerOne Scraper — Scrapes public HackerOne data: Hacktivity disclosed reports, program directory, researcher leaderboard.
• Intigriti Scraper — Scrapes public Intigriti data: bug bounty/VDP program directory, researcher leaderboard and profiles.
• CVE Scraper — CVE.org vulnerability info scraper.
• TLSX - TLS Data Gathering and Analysis — Fast, configurable TLS grabber for TLS-based data collection and analysis.
• BGP.HE.NET Recon Scraper — Scrapes ASN, prefix, IP, and DNS recon data from bgp.he.net.
• gau - Get All URLs — Fetch known URLs from Wayback Machine, Common Crawl, AlienVault OTX, and URLScan.
• Git Email Extractor — Shallow-clones git repos and extracts contributor emails from commit history.
• Wappalyzer Technology Lookup — Detect the technology stack behind any website using Wappalyzer.
• Universal Contact Extractor — Extract phone numbers, emails and social media URLs from web pages.
• Web Text Extractor — Extract clean text or Markdown content from web pages.

Did you find this useful?

⭐ Rate this actor on Apify! Your feedback helps other users find it and helps us keep improving it.