Meta Tags Scraper

HTML pages may contain <meta> tags with metadata related to search engine indexing, page snippets and third party software integration. This metadata hidden within web pages is relevant for content performance within search engines and enhances link presentability on social media.

Input

Multiple input URLs are supported.

{
  "urls": [
    {"url": "https://stackoverflow.com/questions/5015483/test-if-an-attribute-is-present-in-a-tag-in-beautifulsoup"},
    {"url": "https://developers.google.com/search/docs/crawling-indexing/special-tags"}
  ]
}

Output

[{
  "url": "https://stackoverflow.com/questions/5015483/test-if-an-attribute-is-present-in-a-tag-in-beautifulsoup",
  "meta_name": "viewport",
  "content": "width=device-width, height=device-height, initial-scale=1.0, minimum-scale=1.0"
},
{
  "url": "https://stackoverflow.com/questions/5015483/test-if-an-attribute-is-present-in-a-tag-in-beautifulsoup",
  "meta_name": "twitter:card",
  "content": "summary"
},
{
  "url": "https://stackoverflow.com/questions/5015483/test-if-an-attribute-is-present-in-a-tag-in-beautifulsoup",
  "meta_name": "twitter:domain",
  "content": "stackoverflow.com"
},
{
  "url": "https://stackoverflow.com/questions/5015483/test-if-an-attribute-is-present-in-a-tag-in-beautifulsoup",
  "meta_name": "twitter:title",
  "content": "Test if an attribute is present in a tag in BeautifulSoup"
},
{
  "url": "https://stackoverflow.com/questions/5015483/test-if-an-attribute-is-present-in-a-tag-in-beautifulsoup",
  "meta_name": "twitter:description",
  "content": "I would like to get all the <script> tags in a document and then process each one based on the presence (or absence) of certain attributes.\n\nE.g., for each <script> tag, if the attribut..."
},
{
  "url": "https://developers.google.com/search/docs/crawling-indexing/special-tags",
  "meta_name": "google-signin-client-id",
  "content": "721724668570-nbkv1cfusk7kk4eni4pjvepaus73b13t.apps.googleusercontent.com"
},
{
  "url": "https://developers.google.com/search/docs/crawling-indexing/special-tags",
  "meta_name": "google-signin-scope",
  "content": "profile email https://www.googleapis.com/auth/developerprofiles https://www.googleapis.com/auth/developerprofiles.award"
},
{
  "url": "https://developers.google.com/search/docs/crawling-indexing/special-tags",
  "meta_name": "theme-color",
  "content": "#fff"
},
{
  "url": "https://developers.google.com/search/docs/crawling-indexing/special-tags",
  "meta_name": "X-UA-Compatible",
  "content": "IE=Edge"
},
{
  "url": "https://developers.google.com/search/docs/crawling-indexing/special-tags",
  "meta_name": "viewport",
  "content": "width=device-width, initial-scale=1"
},
{
  "url": "https://developers.google.com/search/docs/crawling-indexing/special-tags",
  "meta_name": "description",
  "content": "Google supports both page-level meta tags and tag attributes. Explore a list of the special tags that Google Search can support."
},
{
  "url": "https://developers.google.com/search/docs/crawling-indexing/special-tags",
  "meta_name": "twitter:card",
  "content": "summary_large_image"
},
{
  "url": "https://developers.google.com/search/docs/crawling-indexing/special-tags",
  "meta_name": "apple-itunes-app",
  "content": "app-id=1512522198"
}]

See Also

• meta tags and attributes that Google supports - Google developer portal
• <meta>: The metadata element - Mozilla developer portal
• The Open Graph Protocol