🎥 Video Link Crawler

The Video Link Crawler is a powerful web crawler designed to extract 🎞️ video links from websites, starting from a specified URL, it navigates through web pages, discovers video content, and compiles the extracted links into a structured dataset. It supports customizable crawling settings and efficient video extraction. Whether you're building a video catalog or analyzing video trends, this tool is your go-to solution for efficient video discovery.

---

📝 Description

The Video Link Crawler specializes in extracting video URLs from various HTML elements, including:

• <video> and <source> tags
• <embed> and <object> tags
• <iframe> tags (e.g., YouTube, Vimeo)
• <a> tags linking to video files (e.g., .mp4, .webm)

With advanced features like proxy support, crawl depth control, and retry mechanisms, it adapts seamlessly to your web scraping needs. Extracted data is stored in an Apify Dataset, ready for analysis or integration into your workflows.

---

✨ Key Features

• Customizable Crawling
  ⬩ Define starting URLs
  ⬩ Restrict crawling using glob patterns
  ⬩ Set maximum crawl depth and request limits

• Video URL Extraction
  ⬩ Extract video links from multiple HTML tags
  ⬩ Convert relative URLs to absolute links for seamless results

• Proxy Support
  ⬩ Ensure anonymity with Apify Proxy
  ⬩ Optionally specify proxy groups

• Error Handling
  ⬩ Detailed logs for failed requests
  ⬩ Handles redirect loops and HTTP 403 errors gracefully

• Data Output
  ⬩ Structured and easy-to-use dataset with video links

---

⚙️ How It Works

• Initialization
  Reads input parameters such as startUrl, glob patterns, and crawl depth to configure the crawler.

• Crawling
  Begins at the startUrl, follows links matching the glob patterns, and stops upon reaching the crawl limits.

• Video Extraction
  Detects video links from supported tags and resolves relative URLs to absolute ones.

• Data Storage
  Extracted URLs and video links are saved to an Apify Dataset.

• Error Logging
  Logs all failed or skipped requests for review and troubleshooting.

---

🔧 Input Configuration

The Video Link Crawler accepts the following input parameters in JSON format:

Field	Type	Required	Description
startUrl	string	Yes	The URL where the crawler begins.
globs	string[]	Yes	Array of glob patterns to restrict crawling to specific domains or paths.
maxRequestsPerCrawl	number	Optional	Maximum number of requests the crawler processes.
maxCrawlDepth	number	Optional	Maximum depth of pages to crawl from the starting URL.
proxyConfig	object	Optional	Proxy configuration for anonymized crawling (e.g., Apify Proxy).

🛡️ Proxy Configuration

• useApifyProxy (boolean): Enable Apify Proxy for anonymized crawling.
• apifyProxyGroups (string[]): Specify proxy groups for more granular control (optional).

---

📝 Example Input

1{
2    "startUrl": "https://example.com",
3    "globs": ["https://example.com/**"],
4    "maxRequestsPerCrawl": 50,
5    "maxCrawlDepth": 3,
6    "proxyConfig": {
7        "useApifyProxy": true,
8        "apifyProxyGroups": ["SHARED"]
9    }
10}

---

📄 Example Output

The output of the Actor is saved to an Apify Dataset in the following structured format:

1[
2    {
3        "url": "https://example.com/page1",
4        "videoLinks": [
5            "https://example.com/videos/video1.mp4",
6            "https://youtube.com/watch?v=abc123"
7        ]
8    },
9    {
10        "url": "https://example.com/page2",
11        "videoLinks": []
12    }
13]

ℹ️ Need Help?

If you have any questions, suggestions, or need assistance, feel free to:

💬 Open an issue in the Issues section.

📧 Contact us: infoweaver7@gmail.com