Metadata Scraper

A powerful web scraper that extracts various types of structured metadata from web pages, including JSON-LD, Microdata, Open Graph, Twitter Cards, and more. Perfect for SEO analysis, content aggregation, and research purposes.

Features

• 🔍 Comprehensive Metadata Extraction:

  • JSON-LD structured data
  • Microdata structured data (schema.org)
  • Open Graph metadata
  • Twitter Card metadata
  • Website icons/favicons
  • Standard meta tags

• ⚙️ Advanced Configuration:

  • Configurable crawling depth
  • Adjustable concurrency
  • Request limits
  • Proxy support

• 🚀 Robust Performance:

  • Efficient HTML parsing
  • Handles multiple JSON-LD formats
  • Support for various icon formats

Input Parameters

Parameter	Type	Description	Default
startUrls	Array	URLs to start crawling from	(required)
maxRequestsPerCrawl	Integer	Maximum number of pages to crawl	100
maxConcurrency	Integer	Maximum number of pages processed in parallel	10
extractMetaTags	Boolean	Whether to extract all meta tags	true

Output Data Structure

For each crawled page, the scraper outputs a JSON object with the following fields:

Field	Type	Description
url	String	The URL of the crawled page
title	String	The page title
icon	String	URL of the website's icon/favicon
linkedData	Array	JSON-LD structured data found on the page
microdata	Array	Microdata structured data (schema.org) found on the page
openGraph	Object	Open Graph metadata (used by Facebook and other platforms)
twitterCard	Object	Twitter Card metadata
metaTags	Object	Other meta tags from the page (when extractMetaTags is enabled)

Example Use Cases

E-commerce Research

Extract product information, pricing, availability, and reviews from various online stores for competitive analysis or price monitoring.

Content Aggregation

Build a news aggregator or content recommendation engine by extracting article metadata from different sources.

SEO Analysis

Analyze websites' structured data implementation for SEO optimization recommendations.

Social Media Preview Testing

Test how your content will appear when shared on social media platforms by extracting Open Graph and Twitter Card data.

Example Outputs

Medium Article

1{
2	"url": "https://medium.com/coding-beauty/new-google-project-idx-fae1fdd079c7",
3	"title": "This new IDE from Google is an absolute game changer | by Tari Ibaba | Coding Beauty | Mar, 2025 | Medium",
4	"linkedData": [
5		{
6			"@context": "http://schema.org",
7			"@type": "NewsArticle",
8			"image": [
9				"https://miro.medium.com/v2/resize:fit:1200/1*f-1HQQng85tbA7kwgECqoQ.png"
10			],
11			"url": "https://medium.com/coding-beauty/new-google-project-idx-fae1fdd079c7",
12			"dateCreated": "2025-03-11T19:45:26.427Z",
13			"datePublished": "2025-03-11T19:45:26.427Z",
14			"dateModified": "2025-04-15T14:25:56.263Z",
15			"headline": "This new IDE from Google is an absolute game changer",
16			"name": "This new IDE from Google is an absolute game changer",
17			"description": "I was not surprised to see this sort of thing coming from Google — with their deep-seated hatred for local desktop apps. Loading your projects from GitHub and then install dependencies instantly…",
18			"identifier": "fae1fdd079c7",
19			"author": {
20				"@type": "Person",
21				"name": "Tari Ibaba",
22				"url": "https://medium.com/@tariibaba"
23			},
24			"creator": [
25				"Tari Ibaba"
26			],
27			"publisher": {
28				"@type": "Organization",
29				"name": "Coding Beauty",
30				"url": "https://medium.com/coding-beauty",
31				"logo": {
32					"@type": "ImageObject",
33					"width": 272,
34					"height": 60,
35					"url": "https://miro.medium.com/v2/resize:fit:544/7*V1_7XP4snlmqrc_0Njontw.png"
36				}
37			},
38			"mainEntityOfPage": "https://medium.com/coding-beauty/new-google-project-idx-fae1fdd079c7",
39			"isAccessibleForFree": "False",
40			"hasPart": {
41				"@type": "WebPageElement",
42				"isAccessibleForFree": "False",
43				"cssSelector": ".meteredContent"
44			}
45		}
46	],
47	"microdata": [],
48	"openGraph": {
49		"site_name": "Medium",
50		"type": "article",
51		"title": "This new IDE from Google is an absolute game changer",
52		"description": "This new IDE from Google is seriously revolutionary.",
53		"url": "https://medium.com/coding-beauty/new-google-project-idx-fae1fdd079c7",
54		"image": "https://miro.medium.com/v2/resize:fit:1200/1*f-1HQQng85tbA7kwgECqoQ.png"
55	},
56	"twitterCard": {
57		"app:name:iphone": "Medium",
58		"app:id:iphone": "828256236",
59		"site": "@CodingBeautyDev",
60		"app:url:iphone": "medium://p/fae1fdd079c7",
61		"image:src": "https://miro.medium.com/v2/resize:fit:1200/1*f-1HQQng85tbA7kwgECqoQ.png",
62		"card": "summary_large_image",
63		"creator": "@tariibabadev",
64		"label1": "Reading time",
65		"data1": "5 min read",
66		"title": "This new IDE from Google is an absolute game changer",
67		"description": "This new IDE from Google is seriously revolutionary.",
68		"image": "https://miro.medium.com/v2/resize:fit:1200/1*f-1HQQng85tbA7kwgECqoQ.png",
69		"has_large_image": "true"
70	},
71	"metaTags": {
72		"viewport": "width=device-width,minimum-scale=1,initial-scale=1,maximum-scale=1",
73		"theme-color": "#000000",
74		"al:ios:app_name": "Medium",
75		"al:ios:app_store_id": "828256236",
76		"al:android:package": "com.medium.reader",
77		"fb:app_id": "542599432471018",
78		"article:published_time": "2025-04-10T09:50:11.344Z",
79		"title": "This new IDE from Google is an absolute game changer | by Tari Ibaba | Coding Beauty | Mar, 2025 | Medium",
80		"al:android:url": "medium://p/fae1fdd079c7",
81		"al:ios:url": "medium://p/fae1fdd079c7",
82		"al:android:app_name": "Medium",
83		"description": "I was not surprised to see this sort of thing coming from Google — with their deep-seated hatred for local desktop apps. Loading your projects from GitHub and then install dependencies instantly…",
84		"al:web:url": "https://medium.com/coding-beauty/new-google-project-idx-fae1fdd079c7",
85		"article:author": "https://medium.com/@tariibaba",
86		"author": "Tari Ibaba",
87		"robots": "index,noarchive,follow,max-image-preview:large",
88		"referrer": "unsafe-url"
89	},
90	"icon": "https://miro.medium.com/v2/resize:fill:304:304/10fd5c419ac61637245384e7099e131627900034828f4f386bdaa47a74eae156"
91}

YouTube Video

1{
2	"url": "https://www.youtube.com/watch?v=YCgnccJW_O0",
3	"title": "Rust on Vercel | Build and Deploy Blazing Fast Serverless Functions (Full Guide) - YouTube",
4	"linkedData": [],
5	"microdata": [
6		{
7			"@type": "http://schema.org/VideoObject",
8			"url": [
9				"https://www.youtube.com/watch?v=YCgnccJW_O0",
10				"http://www.youtube.com/@Semicolon10",
11				"https://i.ytimg.com/vi/YCgnccJW_O0/maxresdefault.jpg"
12			],
13			"name": [
14				"Rust on Vercel | Build and Deploy Blazing Fast Serverless Functions (Full Guide)",
15				"Semicolon"
16			],
17			"description": "Want to deploy blazing fast serverless functions using Rust? In this video, I'll show you how to run Rust on Vercel with zero hassle. We'll build a simple AP...",
18			"requiresSubscription": "False",
19			"identifier": "YCgnccJW_O0",
20			"duration": "PT12M15S",
21			"position": "1",
22			"thumbnailUrl": "https://i.ytimg.com/vi/YCgnccJW_O0/maxresdefault.jpg",
23			"width": [
24				"1280",
25				"1280"
26			],
27			"height": [
28				"720",
29				"720"
30			],
31			"embedUrl": "https://www.youtube.com/embed/YCgnccJW_O0",
32			"playerType": "HTML5 Flash",
33			"isFamilyFriendly": "true",
34			"regionsAllowed": "AD,AE,AF,AG,AI,AL,AM,AO,AQ,AR,AS,AT,AU,AW,AX,AZ,BA,BB,BD,BE,BF,BG,BH,BI,BJ,BL,BM,BN,BO,BQ,BR,BS,BT,BV,BW,BY,BZ,CA,CC,CD,CF,CG,CH,CI,CK,CL,CM,CN,CO,CR,CU,CV,CW,CX,CY,CZ,DE,DJ,DK,DM,DO,DZ,EC,EE,EG,EH,ER,ES,ET,FI,FJ,FK,FM,FO,FR,GA,GB,GD,GE,GF,GG,GH,GI,GL,GM,GN,GP,GQ,GR,GS,GT,GU,GW,GY,HK,HM,HN,HR,HT,HU,ID,IE,IL,IM,IN,IO,IQ,IR,IS,IT,JE,JM,JO,JP,KE,KG,KH,KI,KM,KN,KP,KR,KW,KY,KZ,LA,LB,LC,LI,LK,LR,LS,LT,LU,LV,LY,MA,MC,MD,ME,MF,MG,MH,MK,ML,MM,MN,MO,MP,MQ,MR,MS,MT,MU,MV,MW,MX,MY,MZ,NA,NC,NE,NF,NG,NI,NL,NO,NP,NR,NU,NZ,OM,PA,PE,PF,PG,PH,PK,PL,PM,PN,PR,PS,PT,PW,PY,QA,RE,RO,RS,RU,RW,SA,SB,SC,SD,SE,SG,SH,SI,SJ,SK,SL,SM,SN,SO,SR,SS,ST,SV,SX,SY,SZ,TC,TD,TF,TG,TH,TJ,TK,TL,TM,TN,TO,TR,TT,TV,TW,TZ,UA,UG,UM,US,UY,UZ,VA,VC,VE,VG,VI,VN,VU,WF,WS,YE,YT,ZA,ZM,ZW",
35			"interactionType": [
36				"https://schema.org/LikeAction",
37				"https://schema.org/WatchAction"
38			],
39			"userInteractionCount": [
40				"28",
41				"525"
42			],
43			"datePublished": "2025-04-18T05:31:11-07:00",
44			"uploadDate": "2025-04-18T05:31:11-07:00",
45			"genre": "Science & Technology"
46		}
47	],
48	"openGraph": {
49		"site_name": "YouTube",
50		"url": "https://www.youtube.com/watch?v=YCgnccJW_O0",
51		"title": "Rust on Vercel | Build and Deploy Blazing Fast Serverless Functions (Full Guide)",
52		"image": "https://i.ytimg.com/vi/YCgnccJW_O0/maxresdefault.jpg",
53		"image:width": "1280",
54		"image:height": "720",
55		"description": "Want to deploy blazing fast serverless functions using Rust? In this video, I'll show you how to run Rust on Vercel with zero hassle. We'll build a simple AP...",
56		"type": "video.other",
57		"video:url": "https://www.youtube.com/embed/YCgnccJW_O0",
58		"video:secure_url": "https://www.youtube.com/embed/YCgnccJW_O0",
59		"video:type": "text/html",
60		"video:width": "1280",
61		"video:height": "720"
62	},
63	"twitterCard": {
64		"card": "player",
65		"site": "@youtube",
66		"url": "https://www.youtube.com/watch?v=YCgnccJW_O0",
67		"title": "Rust on Vercel | Build and Deploy Blazing Fast Serverless Functions (Full Guide)",
68		"description": "Want to deploy blazing fast serverless functions using Rust? In this video, I'll show you how to run Rust on Vercel with zero hassle. We'll build a simple AP...",
69		"image": "https://i.ytimg.com/vi/YCgnccJW_O0/maxresdefault.jpg",
70		"app:name:iphone": "YouTube",
71		"app:id:iphone": "544007664",
72		"app:name:ipad": "YouTube",
73		"app:id:ipad": "544007664",
74		"app:url:iphone": "vnd.youtube://www.youtube.com/watch?v=YCgnccJW_O0&feature=applinks",
75		"app:url:ipad": "vnd.youtube://www.youtube.com/watch?v=YCgnccJW_O0&feature=applinks",
76		"app:name:googleplay": "YouTube",
77		"app:id:googleplay": "com.google.android.youtube",
78		"app:url:googleplay": "https://www.youtube.com/watch?v=YCgnccJW_O0",
79		"player": "https://www.youtube.com/embed/YCgnccJW_O0",
80		"player:width": "1280",
81		"player:height": "720"
82	},
83	"metaTags": {
84		"theme-color": "rgba(255, 255, 255, 0.98)",
85		"title": "Rust on Vercel | Build and Deploy Blazing Fast Serverless Functions (Full Guide)",
86		"description": "Want to deploy blazing fast serverless functions using Rust? In this video, I'll show you how to run Rust on Vercel with zero hassle. We'll build a simple AP...",
87		"keywords": "video, sharing, camera phone, video phone, free, upload",
88		"al:ios:app_store_id": "544007664",
89		"al:ios:app_name": "YouTube",
90		"al:ios:url": "vnd.youtube://www.youtube.com/watch?v=YCgnccJW_O0&feature=applinks",
91		"al:android:url": "vnd.youtube://www.youtube.com/watch?v=YCgnccJW_O0&feature=applinks",
92		"al:web:url": "http://www.youtube.com/watch?v=YCgnccJW_O0&feature=applinks",
93		"al:android:app_name": "YouTube",
94		"al:android:package": "com.google.android.youtube",
95		"fb:app_id": "87741124305"
96	},
97	"icon": "https://www.youtube.com/s/desktop/c722ba88/img/logos/favicon_32x32.png"
98}

Getting Help

If you need help or have questions about using this Actor, please don't hesitate to submit issues.