Web Crawler

This web crawler is designed to provide users with complete flexibility by allowing them to use their own proxies. The scraper collects all pages from the website and returns extracts the MetaData, Title, and Content of the page in MarkDown.

Features

• Extracts MetaData, Title, and Content.
• User can use his/her own proxies.
• Easy-to-use interface with customizable scraping options.

How to Use

1. Input the URL you want to scrape.
2. Provide your proxy details (Proxy URL).
3. Run the scraper to collect the data.
4. Download the results as JSON, CSV, or view them directly in Apify.

Recommended Proxy Providers

• Shifter

  • Reliable residential proxies all over the world.
  • Cheap rates
  • Order Shifter Now
  • Get 10% Off any product, use coupan rigelbytes-YoBB.

• OxyLabs

  • 100M+ Proxies
  • Fastest proxies in the market
  • Real profile, human-like Residential IPs
  • Quality assurance framework for most reliable IPs
  • Get Proxies

🙌 Why Buy Through Our Affiliate Link?

• Exclusive Deals: Some providers may offer special discounts or bonuses when you use our link.
• Support Our Work: Each purchase helps us maintain and improve the tools and services we provide.
• No Extra Cost: You pay the same price, but part of it goes to supporting our efforts.

API Request Examples

Python

1from apify_client import ApifyClient
2
3# Initialize the ApifyClient with your API token
4client = ApifyClient("<YOUR_API_TOKEN>")
5
6# Prepare the Actor input
7run_input = {
8    "url": "https://www.rigelbytes.com/",
9}
10
11# Run the Actor and wait for it to finish
12run = client.actor("rigelbytes/webcrawler").call(run_input=run_input)
13
14# Fetch and print Actor results from the run's dataset (if there are any)
15for item in client.dataset(run["defaultDatasetId"]).iterate_items():
16    print(item)

Node.js

1import { ApifyClient } from 'apify-client';
2
3// Initialize the ApifyClient with API token
4const client = new ApifyClient({
5    token: '<YOUR_API_TOKEN>',
6});
7
8// Prepare Actor input
9const input = {
10    "url": "https://www.rigelbytes.com/",
11};
12
13(async () => {
14    // Run the Actor and wait for it to finish
15    const run = await client.actor("rigelbytes/webcrawler").call(input);
16
17    // Fetch and print Actor results from the run's dataset (if any)
18    console.log('Results from dataset');
19    const { items } = await client.dataset(run.defaultDatasetId).listItems();
20    items.forEach((item) => {
21        console.dir(item);
22    });
23})();

cURL

1# Set API token
2API_TOKEN=<YOUR_API_TOKEN>
3
4# Prepare Actor input
5cat > input.json <<'EOF'
6{"url": "https://www.rigelbytes.com/",
7}
8EOF
9
10# Run the Actor
11curl "https://api.apify.com/v2/acts/rigelbytes/webcrawler/runs?token=$API_TOKEN" \
12  -X POST \
13  -d @input.json \
14  -H 'Content-Type: application/json'

Output

1[
2  {
3    "page_url": "http://www.FITaxPlanning.com/taxcenter2.php",
4    "title": "Placentia, CA Accounting Firm | Tax Center Page | Financial Insight Tax Planning, Inc.",
5    "metadata": "{\"viewport\": \"width=device-width, initial-scale=1.0\", \"description\": \"Take a look at our Tax Center page. Financial Insight Tax Planning, Inc.  is a full service tax, accounting and financial consulting firm located in Placentia, CA.\", \"keywords\": \"QuickBooks, QuickBooks Consultant, Certified QuickBooks Consultant, CPA, Accountant, Tax, Tax Preparation, CA, California, Placentia, Financial Planner, Tax Planner, Financial consultant, Business Taxes, 1031 Exchange, Estate Planning, Real Estate Taxes\"}",
6    "content": "## FITax Planning, Inc.\n\n  * Home\n  * About\n    * Firm Background\n    * Meet Our Team\n    * Security Measures\n    * Privacy Policy\n    * Refer Us\n    * Why Choose a CPA\n    * Why Choose an Enrolled Agent\n    * Employment Opportunities\n  * Services\n    * IRS Tax Problems\n    * Business Services\n    * Financial Planning\n    * Estate Planning\n    * Estate & Trust Tax Services\n    * Tax Preparation and Planning\n  * Newsletters\n    * Previous Newsletters\n    * This Month's Newsletter\n  * Financial Guides\n    * Life Events\n    * Financial Calculators\n    * Business Strategies\n    * Investment Strategies\n    * Tax Strategies for Individuals\n    * Tax Strategies for Business Owners\n    * Frequently Asked Questions (FAQs)\n  * Tax Center\n    * Tax Rates\n    * Tax Due Dates\n    * Tax Calculators\n    * IRS Withholding Calculator\n    * Tax Glossary & More\n    * Free Tax Organizer\n    * Track Your Refund\n    * State Tax Forms\n    * IRS Tax Forms and Publications\n    * Record Retention Guide\n  * Resources\n    * Appointment Checklist\n    * Online Tax Organizer\n    * Financial Calculators\n    * Solution Center\n    * Client Portal\n    * Secure Send\n  * Contact\n\n##  Call Us: (888) 912-7526, ext. 5\n\nSet Up a Free Consultation Search\n\n  * Home\n  * About\n    * Firm Background\n    * Meet Our Team\n    * Security Measures\n    * Privacy Policy\n    * Refer Us\n    * Why Choose a CPA\n    * Why Choose an Enrolled Agent\n    * Employment Opportunities\n  * Services\n    * IRS Tax Problems\n    * Business Services\n    * Financial Planning\n    * Estate Planning\n    * Estate & Trust Tax Services\n    * Tax Preparation and Planning\n  * Newsletters\n    * Previous Newsletters\n    * This Month's Newsletter\n  * Financial Guides\n    * Life Events\n    * Financial Calculators\n    * Business Strategies\n    * Investment Strategies\n    * Tax Strategies for Individuals\n    * Tax Strategies for Business Owners\n    * Frequently Asked Questions (FAQs)\n  * Tax Center\n    * Tax Rates\n    * Tax Due Dates\n    * Tax Calculators\n    * IRS Withholding Calculator\n    * Tax Glossary & More\n    * Free Tax Organizer\n    * Track Your Refund\n    * State Tax Forms\n    * IRS Tax Forms and Publications\n    * Record Retention Guide\n  * Resources\n    * Appointment Checklist\n    * Online Tax Organizer\n    * Financial Calculators\n    * Solution Center\n    * Client Portal\n    * Secure Send\n  * Contact\n\n# Tax Center\n\n|\n\n# Tax Center  \n  \n---  \n|  \n---  \n  \n\n##  Days until April 15\n\nDeadline for Individuals and C-Corporations\n\n##  Days until March 15\n\nDeadline for S-Corporations and Partnerships\n\n#  Tax Center\n\n##  Tax tools to help you reach your tax planning goals\n\n  \n  \n\n**Tax Rates**  \nTax rates change every year. Take a look at this year's tax rates.  |    |  **Tax Due Dates**   \nPlease note the following tax due dates on your calendar, and come back often\nto keep up with the changes.  \n---|---|---  \n**Tax Calculators**  \nOur advance and simple to use calculators will help you better plan out your taxes and make decisions.  |    |  **IRS Withholding Calculator**   \nUse this calculator to help you figure out how to avoid having too much or too\nlittle Federal income tax withheld from your pay.  \n**Tax Glossary& More**  \nPrimary resources for tax glossaries and learning about taxes.  |    |  **Free Tax Organizer**   \nDon't miss any important deductions! Help us uncover hidden tax deductions by\nfilling out this form.  \n**Track Your Refund**  \nFind out when you'll receive your federal and state refund.  |    |  **State Tax Forms**   \nQuickly print the tax forms you need from any State in the country.  \n**Tax Forms& Publications**  \nQuickly view and print any IRS tax form or publication. Saves a trip to the post office.  |    |  **Record Retention Guide**   \nUse this guide to determine how long you need to keep your tax and other\nfinancial records.  \n  \n## Free Newsletter\n\nStay current and receive our informative monthly newsletter.\n\n## Latest News\n\n## Connect With Us\n\n  \nFacebook |    \nTwitter |    \nLinkedin |    \nGoogle+  \n---|---|---|---  \n  \n## Quick Links\n\nFinancial Calculators Send Us A File Track Your Refund IRS Tax Forms State Tax\nForms Tax Due Dates\n\n## Client Login\n\n  1. E-Mail:\n\n  2. Password:\n\n  3. \n\n##  Contact Us\n\nFinancial Insight Tax Planning, Inc.  \n550 W Orangethorpe Ave., Suite T  \nPlacentia, CA  92870  \nPhone: (888) 912-7526, ext. 5  \nFax: (714) 988-2178  \nMonday-Friday: 9am – 5pm  \nSaturday & Sunday: by appointment  \noffice@fitaxplanning.com\n\nLogin   Search   Site Map   Privacy Policy   Disclaimer  \n\n"
7  },
8  {
9    "page_url": "http://www.FITaxPlanning.com/investment-strategies.php",
10    "title": "Placentia, CA Accounting Firm | Investment Strategies Page | Financial Insight Tax Planning, Inc.",
11    "metadata": "{\"viewport\": \"width=device-width, initial-scale=1.0\", \"description\": \"Take a look at our Investment Strategies page. Financial Insight Tax Planning, Inc.  is a full service tax, accounting and financial consulting firm located in Placentia, CA.\", \"keywords\": \"QuickBooks, QuickBooks Consultant, Certified QuickBooks Consultant, CPA, Accountant, Tax, Tax Preparation, CA, California, Placentia, Financial Planner, Tax Planner, Financial consultant, Business Taxes, 1031 Exchange, Estate Planning, Real Estate Taxes\"}",
12    "content": "## FITax Planning, Inc.\n\n  * Home\n  * About\n    * Firm Background\n    * Meet Our Team\n    * Security Measures\n    * Privacy Policy\n    * Refer Us\n    * Why Choose a CPA\n    * Why Choose an Enrolled Agent\n    * Employment Opportunities\n  * Services\n    * IRS Tax Problems\n    * Business Services\n    * Financial Planning\n    * Estate Planning\n    * Estate & Trust Tax Services\n    * Tax Preparation and Planning\n  * Newsletters\n    * Previous Newsletters\n    * This Month's Newsletter\n  * Financial Guides\n    * Life Events\n    * Financial Calculators\n    * Business Strategies\n    * Investment Strategies\n    * Tax Strategies for Individuals\n    * Tax Strategies for Business Owners\n    * Frequently Asked Questions (FAQs)\n  * Tax Center\n    * Tax Rates\n    * Tax Due Dates\n    * Tax Calculators\n    * IRS Withholding Calculator\n    * Tax Glossary & More\n    * Free Tax Organizer\n    * Track Your Refund\n    * State Tax Forms\n    * IRS Tax Forms and Publications\n    * Record Retention Guide\n  * Resources\n    * Appointment Checklist\n    * Online Tax Organizer\n    * Financial Calculators\n    * Solution Center\n    * Client Portal\n    * Secure Send\n  * Contact\n\n##  Call Us: (888) 912-7526, ext. 5\n\nSet Up a Free Consultation Search\n\n  * Home\n  * About\n    * Firm Background\n    * Meet Our Team\n    * Security Measures\n    * Privacy Policy\n    * Refer Us\n    * Why Choose a CPA\n    * Why Choose an Enrolled Agent\n    * Employment Opportunities\n  * Services\n    * IRS Tax Problems\n    * Business Services\n    * Financial Planning\n    * Estate Planning\n    * Estate & Trust Tax Services\n    * Tax Preparation and Planning\n  * Newsletters\n    * Previous Newsletters\n    * This Month's Newsletter\n  * Financial Guides\n    * Life Events\n    * Financial Calculators\n    * Business Strategies\n    * Investment Strategies\n    * Tax Strategies for Individuals\n    * Tax Strategies for Business Owners\n    * Frequently Asked Questions (FAQs)\n  * Tax Center\n    * Tax Rates\n    * Tax Due Dates\n    * Tax Calculators\n    * IRS Withholding Calculator\n    * Tax Glossary & More\n    * Free Tax Organizer\n    * Track Your Refund\n    * State Tax Forms\n    * IRS Tax Forms and Publications\n    * Record Retention Guide\n  * Resources\n    * Appointment Checklist\n    * Online Tax Organizer\n    * Financial Calculators\n    * Solution Center\n    * Client Portal\n    * Secure Send\n  * Contact\n\n# Investment Strategies\n\n  \n  \nWhy leave your money sitting in a bank account earning for other people when\nyou can put it to work for yourself? This section is dedicated to tested and\nproven investment strategies and concepts.  \n  \n\n## Financial Guides\n\n  \n\n* * *\n\n| **Investment Basics: What You Should Know**  \n**Asset Allocation: How To Diversify For Maximum Return**  \n**Buying On Margin: How It Works and What To Watch Out For**  \n**Financial Planning Checklist**  \n**Investment Clubs: What You Need To Know**  \n**Swap Tactic Lets You Defer Capital Gains Tax**  \n**Changing Jobs? Don't Forget Your 401(k)**  \n**How Brokers Execute Trades: What Every Investor Should Know**  \n**Investing In Mutual Funds: The Time-Tested Guidelines**  \n**Mutual Fund Taxation: How To Cut The Tax Bite**  \n**Penny Stocks: How To Investigate Them and Avoid the Traps**  \n**Annuities: How They Work and When You Should Use Them**  \n**10 Retirement Saving Tips**  \n**Your Retirement Plan: How To Get Started**  \n**Roth IRAs: How They Work and How To Use Them**  \n  \n---|---  \n  \n  \n\n## Frequently Asked Questions\n\n  \n\n* * *\n\n| **Investment Options: Frequently Asked Questions**  \n**Mutual Funds: Frequently Asked Questions**  \n**Stocks: Frequently Asked Questions**  \n**Bonds: Frequently Asked Questions**  \n**Annuities: Frequently Asked Questions**  \n**Retirement Assets: Frequently Asked Questions**  \n**Developing a Financial Plan: Frequently Asked Questions**  \n**Traditional vs Roth IRAs: Frequently Asked Questions**  \n  \n---|---  \n  \n  \n\n## Financial Calculators\n\n  \n\n* * *\n\n| **Am I saving enough for my retirement?**  \n**How Much Can I Contribute To An IRA?**  \n**Should I Convert To A Roth IRA?**  \n**What are my projected required minimum distributions?**  \n**How Do Taxes and Inflation Impact My Investment Return?**  \n**What are my projected required minimum distributions?**  \n**Becoming A Millionaire**  \n  \n---|---  \n  \n  \n\n#  Investment Strategies\n\n##  Put your money to work for yourself.\n\n## Free Newsletter\n\nStay current and receive our informative monthly newsletter.\n\n## Latest News\n\n## Connect With Us\n\n  \nFacebook |    \nTwitter |    \nLinkedin |    \nGoogle+  \n---|---|---|---  \n  \n## Quick Links\n\nFinancial Calculators Send Us A File Track Your Refund IRS Tax Forms State Tax\nForms Tax Due Dates\n\n## Client Login\n\n  1. E-Mail:\n\n  2. Password:\n\n  3. \n\n##  Contact Us\n\nFinancial Insight Tax Planning, Inc.  \n550 W Orangethorpe Ave., Suite T  \nPlacentia, CA  92870  \nPhone: (888) 912-7526, ext. 5  \nFax: (714) 988-2178  \nMonday-Friday: 9am – 5pm  \nSaturday & Sunday: by appointment  \noffice@fitaxplanning.com\n\nLogin   Search   Site Map   Privacy Policy   Disclaimer  \n\n"
13  },
14  {
15    "page_url": "http://www.FITaxPlanning.com/reports.php",
16    "title": "Placentia, CA Accounting Firm | Financial Guides Page | Financial Insight Tax Planning, Inc.",
17    "metadata": "{\"viewport\": \"width=device-width, initial-scale=1.0\", \"description\": \"Take a look at our Financial Guides page. Financial Insight Tax Planning, Inc.  is a full service tax, accounting and financial consulting firm located in Placentia, CA.\", \"keywords\": \"QuickBooks, QuickBooks Consultant, Certified QuickBooks Consultant, CPA, Accountant, Tax, Tax Preparation, CA, California, Placentia, Financial Planner, Tax Planner, Financial consultant, Business Taxes, 1031 Exchange, Estate Planning, Real Estate Taxes\"}",
18    "content": "## FITax Planning, Inc.\n\n  * Home\n  * About\n    * Firm Background\n    * Meet Our Team\n    * Security Measures\n    * Privacy Policy\n    * Refer Us\n    * Why Choose a CPA\n    * Why Choose an Enrolled Agent\n    * Employment Opportunities\n  * Services\n    * IRS Tax Problems\n    * Business Services\n    * Financial Planning\n    * Estate Planning\n    * Estate & Trust Tax Services\n    * Tax Preparation and Planning\n  * Newsletters\n    * Previous Newsletters\n    * This Month's Newsletter\n  * Financial Guides\n    * Life Events\n    * Financial Calculators\n    * Business Strategies\n    * Investment Strategies\n    * Tax Strategies for Individuals\n    * Tax Strategies for Business Owners\n    * Frequently Asked Questions (FAQs)\n  * Tax Center\n    * Tax Rates\n    * Tax Due Dates\n    * Tax Calculators\n    * IRS Withholding Calculator\n    * Tax Glossary & More\n    * Free Tax Organizer\n    * Track Your Refund\n    * State Tax Forms\n    * IRS Tax Forms and Publications\n    * Record Retention Guide\n  * Resources\n    * Appointment Checklist\n    * Online Tax Organizer\n    * Financial Calculators\n    * Solution Center\n    * Client Portal\n    * Secure Send\n  * Contact\n\n##  Call Us: (888) 912-7526, ext. 5\n\nSet Up a Free Consultation Search\n\n  * Home\n  * About\n    * Firm Background\n    * Meet Our Team\n    * Security Measures\n    * Privacy Policy\n    * Refer Us\n    * Why Choose a CPA\n    * Why Choose an Enrolled Agent\n    * Employment Opportunities\n  * Services\n    * IRS Tax Problems\n    * Business Services\n    * Financial Planning\n    * Estate Planning\n    * Estate & Trust Tax Services\n    * Tax Preparation and Planning\n  * Newsletters\n    * Previous Newsletters\n    * This Month's Newsletter\n  * Financial Guides\n    * Life Events\n    * Financial Calculators\n    * Business Strategies\n    * Investment Strategies\n    * Tax Strategies for Individuals\n    * Tax Strategies for Business Owners\n    * Frequently Asked Questions (FAQs)\n  * Tax Center\n    * Tax Rates\n    * Tax Due Dates\n    * Tax Calculators\n    * IRS Withholding Calculator\n    * Tax Glossary & More\n    * Free Tax Organizer\n    * Track Your Refund\n    * State Tax Forms\n    * IRS Tax Forms and Publications\n    * Record Retention Guide\n  * Resources\n    * Appointment Checklist\n    * Online Tax Organizer\n    * Financial Calculators\n    * Solution Center\n    * Client Portal\n    * Secure Send\n  * Contact\n\n# Financial Guides\n\n|\n\n# Financial Guides  \n  \n---|---  \n|  \n---  \n  \n\n#  Financial Guides\n\n##  Click on any of the categories below to find practical, unbiased\ninformation to help you reach your financial goals.\n\n  \n  \n\n**Life Events**  \nLife is full of changes, and surprises. Use these guides to help you make sound financial decisions.  |    |  **Financial Calculators**   \nPlease try our wide variety of interactive financial calculators.  \n---|---|---  \n**Business Strategies**  \nLearn how to start, run and eventually sell a successful business.  |    |  **Investment Strategies**   \nWhy leave your money sitting in a bank account earning for other people when\nyou can put it work for yourself? This section is dedicated to tested and\nproven investment strategies and concepts.  \n**Tax Strategies for Individuals**  \nLearn how to lower your taxes with these legitimate tax saving strategies.  |    |  **Tax Strategies for Business Owners**   \nFind perfectly legal tax deductions that you may be missing.  \n**Frequently Asked Questions (FAQs)**  \nOver 500 everyday financial questions answered.  |    |     \n  \nFI Tax Planning  \n550 W Orangethorpe Ave  \nPlacentia, CA, 92870  \n**Phone: (714) 231-5807**  \noffice@fitaxplanning.com  \n  \n\n## Free Newsletter\n\nStay current and receive our informative monthly newsletter.\n\n## Latest News\n\n## Connect With Us\n\n  \nFacebook |    \nTwitter |    \nLinkedin |    \nGoogle+  \n---|---|---|---  \n  \n## Quick Links\n\nFinancial Calculators Send Us A File Track Your Refund IRS Tax Forms State Tax\nForms Tax Due Dates\n\n## Client Login\n\n  1. E-Mail:\n\n  2. Password:\n\n  3. \n\n##  Contact Us\n\nFinancial Insight Tax Planning, Inc.  \n550 W Orangethorpe Ave., Suite T  \nPlacentia, CA  92870  \nPhone: (888) 912-7526, ext. 5  \nFax: (714) 988-2178  \nMonday-Friday: 9am – 5pm  \nSaturday & Sunday: by appointment  \noffice@fitaxplanning.com\n\nLogin   Search   Site Map   Privacy Policy   Disclaimer  \n\n"
19  }
20]

Requirements

• An Apify account
• Website URL

Understanding Proxies:

When scraping data or browsing anonymously, proxies are essential. They act as intermediaries, masking your original IP address and allowing you to send requests from another location.

Why Use Proxies?

• Avoid IP Blocks: By routing requests through proxies, you prevent the target website from recognizing your IP as a scraper or spammer.
• Access Geo-restricted Content: Proxies let you access content or websites restricted by location.
• Enhance Anonymity: Hide your actual IP, ensuring privacy while scraping or browsing.

Types of Proxies

1. Residential Proxies
   • Real IP addresses provided by ISPs to home users.
   • They mimic regular users, making them harder to detect.
   • Best for: Long-term, undetectable scraping, and avoiding blocks.
2. Data Center Proxies
   • IP addresses from servers in data centers.
   • Faster and cheaper than residential proxies but easier to detect and block.
   • Best for: High-speed scraping, but with a higher risk of detection.
3. Mobile Proxies
   • IPs provided by mobile carriers (3G/4G/5G networks).
   • Very difficult to detect, as they appear as regular mobile users.
   • Best for: Mobile-related scraping or avoiding sophisticated blocks.

Rotating Proxies vs. Straight Proxies

• Rotating Proxies: Every request you send goes through a different proxy, making it harder for websites to detect patterns.
• Straight Proxies: All requests are sent through the same proxy, making it easier to track your IP.

About Rigel Bytes

Rigel Bytes specializes in web scraping, automation, and data analytics. We help businesses extract and leverage valuable data for informed decision-making.

Contact Us

Ready to unlock the power of data? Reach out to us at (contact@rigelbytes.com) or book an appointment with us to learn more about how we can help you achieve your data goals.