You can access the Smart AI Web Scraper programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

$echo '{
<  "startUrl": "https://www.scrapethissite.com/pages/",
<  "actions": [
<    {
<      "action": "Click heading: Hockey Teams: Forms, Searching and Pagination",
<      "waitAfterAction": 0.5,
<      "waitBeforeAction": 0.5
<    }
<  ],
<  "fields": [
<    {
<      "dataType": "text",
<      "fieldName": "Team Name"
<    },
<    {
<      "dataType": "number",
<      "fieldName": "Year"
<    },
<    {
<      "dataType": "number",
<      "fieldName": "Wins"
<    },
<    {
<      "dataType": "number",
<      "fieldName": "Losses"
<    },
<    {
<      "dataType": "number",
<      "fieldName": "Win %"
<    },
<    {
<      "dataType": "number",
<      "fieldName": "Goals Against"
<    },
<    {
<      "dataType": "number",
<      "fieldName": "Goals For"
<    }
<  ],
<  "proxyConfiguration": {
<    "useApifyProxy": false
<  }
<}' |
<apify call cockroachapi/smart-ai-web-scraper --silent --output-dataset

Smart AI Web Scraper API through CLI

The Apify CLI is the official tool that allows you to use Smart AI Web Scraper locally, providing convenience functions and automatic retries on errors.

Install Apify CLI

Using installation script (macOS/Linux):

$curl -fsSL https://apify.com/install-cli.sh | bash

Using installation script (Windows):

$irm https://apify.com/install-cli.ps1 | iex

Using Homebrew:

$brew install apify-cli

Using npm:

$npm install -g apify-cli

Other API clients include:

Smart AI Web Scraper API in Python

Smart AI Web Scraper API in JavaScript

Smart AI Web Scraper OpenAPI definition

Smart AI Web Scraper API

Dynamic Web Scraper

josejet/dynamic-web-scraper

Dynamic Web Scraper is an Apify Actor that gathers information online by simulating user browsing behavior on the web. It reduces the time and amount of scraped web pages by using a model (ChatGPT) to make decisions regarding browser navigation and results evaluation.

Pepa J

353

🧠 RAG Web Browser — Web Content for AI & LLMs

nexgendata/rag-web-browser

Web browser for RAG pipelines and AI agents. Search Google, scrape top results, return clean Markdown. Feed your LLM with real-time web data. Works with Claude, GPT, LangChain, CrewAI. No API key needed.

NexGenData

AI Web Crawler

hounderd/ai-web-crawler

Crawl websites and extract clean, LLM-ready markdown content with stealth browser rendering, anti-bot hardening, smart content filtering, and structured metadata extraction. Built for RAG pipelines, AI agents, and data workflows.

Hounderd

RAG-Ready Web Scraper & Smart Chunker for AI Knowledge Bases

adinfosys-labs/rag-ready-web-scraper-smart-chunker-for-ai-knowledge-bases

RAG-ready web scraper that collects, cleans, deduplicates, filters, and chunks web content into structured datasets for AI pipelines. Generates high-quality knowledge-base data optimized for LLMs, embeddings, and vector databases

Artashes Arakelyan

Smart Business Lead Collector – AI Contact & Company Scraper

vanagha/smart-business-lead-collector---ai-contact-company-scraper

Collect verified business emails, phones, and company summaries with AI. This smart scraper uses LlamaIndex to find and deduplicate contact info from any website. Fast, tested, and free for a limited time!

Van agha

SOY Smart Web Data Extractor

xavier.hc/soy-smart-data-extractor

Extract structured data from public web pages, including titles, descriptions, emails, phone numbers, social links, text previews, status codes, and URLs. Useful for lead research, directory building, monitoring, and automation workflows.

Xavier Hernandez

RAG Web Browser

apify/rag-web-browser

Web search and fetch tool for AI agents and RAG pipelines. It queries Google Search, scrapes the top N pages using a full web browser, and returns their content as clean Markdown for further processing by an LLM. Can also fetch individual URLs.

Apify

166K

4.7

Web Content Scraper

anyxsolutions/web-content-scraper

Fetches a list of web pages and extracts clean article text, markdown, and publication metadata.

Anyx Solutions

AI-Powered Smart Web Scraper

cloud9_ai/ai-web-scraper

Intelligent content extraction from any website using Crawlee + AI. Auto-detects structure, adapts to layout changes, handles JavaScript rendering. No custom code needed. Extract articles, products, listings from 1000s of pages.