# Website Content Vector Retriever (`hamza.alwan/website-content-vector-retriever`) Actor - **URL**: https://apify.com/hamza.alwan/website-content-vector-retriever.md - **Developed by:** [Hamza Alwan](https://apify.com/hamza.alwan) (community) - **Categories:** AI - **Stats:** 26 total users, 0 monthly users, 0.0% runs succeeded, 5 bookmarks - **User rating**: No ratings yet ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## Website Content Vector Retriever ### Description This Actor is primarily designed for use with the [Chat with website](https://chat.openai.com/g/g-FvHl3aAOB-chat-with-website) GPT, but it can also be used for other purposes. 1. **Initiate the Process**: Start by providing a `URL` and a `Query`. These are your primary inputs, but there are additional optional settings available for exploration. 2. **Web Crawling**: The system efficiently crawls the specified `URL` using the reliable [Website Content Crawler](https://apify.com/apify/website-content-crawler) Actor. 3. **Data Transformation**: The results from the Website Content Crawler are transformed into a query-friendly format using OpenAI's embedding function. 4. **Data Storage**: The transformed results are stored in a dedicated Chroma database for efficient storage and quick retrieval. 5. **Query Execution**: The Actor queries the transformed results within the Chroma database. 6. **Results**: The Actor returns relevant results sorted by relevance into your dataset, prioritizing the most important information. 7. **Chat with Website GPT**: The [Chat with website](https://chat.openai.com/g/g-FvHl3aAOB-chat-with-website) GPT reads the results and responds to the user based on the `Query`. ### Example Results ```json { "request": { "url": "https://www.cnn.com/", "query": "top stories" }, "status": "Website crawled completely, and relevant content found.", "results": [ { "id": "b6922816-89f4-11ee-9146-4640c8bcfb86", "text": "This Moroccan startup is growing crops in the desert \nWarren Buffett donates $870 million to charities ahead of Thanksgiving \nWhat’s open and closed on Thanksgiving Day 2023 \nJake Tapper reveals challenges of covering war, why he feels news outlets ‘censor too much’ and what has left him ‘shocked’ \nChicken wings are on Popeyes menu for good \nCEO of Chinese online game streaming firm arrested amid executive crackdown \nWhat is Binance, why is it in so much trouble, and what does it mean for crypto? \nAd Feedback\nQuote Search\nMarket Movers \nACTIVES GAINERS LOSERS \n$ Price % Change \nAd Feedback", "metadata": { "url": "https://edition.cnn.com/business" }, "score": 0.4413832212525754 }, { "id": "b69226f4-89f4-11ee-9146-4640c8bcfb86", "text": "Liberia’s President George Weah concedes victory after tight run-off election \n‘They called me a slave’: Witness testimony exposes alleged RSF-led campaign to enslave men and women in Sudan \nDisturbing videos emerge showing atrocities against African ethnic groups in Darfur \nMass visa cancellations at Saudi airport puts relations with Nigeria in spotlight \nAmericas \nShow all\nGetty Images \nAs Argentina heads to the polls, will a plan to ditch the peso for the dollar be a vote-winner? \nMexico’s first openly non-binary magistrate and prominent LGBTQ activist found dead \nProtests against copper mine deal turn deadly in Panama \nPablo Escobar’s ‘cocaine hippos’ face Colombian government cull \nAsia \nShow all\nCaroline Chia/Reuters/File \nAustralian prime minister accuses Chinese navy of ‘dangerous’ conduct \nCivilians caught in the crossfire as fighting escalates between Myanmar military and armed group \nRenewed fighting in Myanmar has displaced 26,000 people since Monday, UN says", "metadata": { "url": "https://edition.cnn.com/world" }, "score": 0.40209810318097283 }, { "id": "b6922726-89f4-11ee-9146-4640c8bcfb86", "text": "Look of the week: Emily Ratajkowski and the ultra-cinched puffer \nRemember when Angelina Jolie’s thigh-high slit dress kickstarted the ‘Angelina Effect’? \nAnimatronic model of E.T.’s head expected to fetch up to $1 million at auction \nThe Iranian artist using painting to honor her lost homeland \nWeather \nShow all\nCNN Weather \nDisruptive Thanksgiving-week storm affecting much of the East as holiday travel rush begins \nSeveral tornadoes reported as dangerous storms fire up, threatening 20 million from Texas to the Southeast \nSignificant Thanksgiving-week storm will disrupt travel with rain, snow and severe storms \nSnow and rain to bring weather woes to millions of Thanksgiving travelers \nMore of the latest stories\n•Video \n2-year-old dies after boat capsizes near Lampedusa \n•Video \nFamilies of hostages held by Hamas Await news of a possible release of their loved ones \nReuters \n•Video \nVideo shows Israeli soldiers forcefully arresting Palestinian man in his home \n•Video", "metadata": { "url": "https://edition.cnn.com/world" }, "score": 0.3971370513809591 } ] } ``` ### Detailed Description Here's a detailed breakdown of how the Actor works: 1. **Input**: Provide a `URL` and a `Query`. These are your primary inputs, but there are additional optional settings to explore. 2. **Unique ID Generation**: Generate a unique ID from the URL for the key-value store. This ID will be used to store the final results of the Chroma database for querying and caching purposes. 3. **URL Check**: Check if the URL has been previously scraped and if the results are saved in the key-value store. - If the URL has been previously scraped, retrieve the data from the Chroma database, query it, and return the results to the user. 4. **Scraping Check**: Verify if the URL is currently undergoing scraping by another instance of the Actor, as it could be a lengthy process. - If the URL is presently under scraping, log the following results to the dataset: ```json { "request": { "url": "url", "query": "query" }, "status": "The website '{url}' is currently undergoing crawling. Expect results shortly; they will be cached for faster access in the future.", "results": [] } ``` - This information will be processed by ChatGPT, and the user will receive an appropriate response. - Refer to the [notes](#notes) for additional details; a webhook is already configured. 5. **Website Content Crawler**: If the URL hasn't been scraped before, utilize the [Website Content Crawler](https://apify.com/apify/website-content-crawler) Actor to scrape the input URL. 6. **Data Transformation**: Once the Actor has finished scraping the URL, the results will be converted by OpenAI's embedding function into a format that enhances the ease of querying of results. 7. **Data Storage**: The transformed results are then saved in a dedicated Chroma database, which will also be cashed in the key-value store. 8. **Query Execution**: The Actor then queries the transformed results within the Chroma database using the provided `Query` in the input. 9. **Results**: The Actor then returns relevant results sorted by relevance into your dataset, [example](#example-results). 10. **Chat with Website GPT**: The results will be read by ChatGPT (For the Chat with website GPT) and will respond to the user in accordance to his question (Query). ### Notes 1. This Actor is primarily designed for use with ChatGPT, which has a timeout of 45 seconds. Therefore, we've set a 42-second limit for the Actor to complete its process. If the Actor surpasses this limit, the following steps will be taken: - If the process exceeds 42 seconds, record the following results in the dataset: ```json { "request": { "url": "url", "query": "query" }, "status": "Currently crawling the website '{url}', but it may take a minute. Results will be cached for faster access in the future.", "results": [] } ``` - ChatGPT will interpret this information and advise the user to try again later. - A webhook is configured to wait for the completion of the [Website Content Crawler](https://apify.com/apify/website-content-crawler) Actor. Once completed, it will trigger a callback to the current Actor with the same input and the run ID. This 2. By default each URLs data is cached for 30 days, but this can be changed in the input settings. # Actor input Schema ## `url` (type: `string`): A URL of a website where to fetch the web pages from. The URL can be a top-level domain like https://example.com, a subdirectory https://example.com/some-directory/, or a specific page https://example.com/some-directory/page.html. ## `query` (type: `string`): Text query to look for on the website. ## `openaiApiKey` (type: `string`): OpenAI API key to generate vector embeddings. ## `maxCrawlPages` (type: `integer`): The maximum number pages to crawl. It includes the start URLs, pagination pages, pages with no content, etc. The crawler will automatically finish after reaching this number. This setting is useful to prevent accidental crawler runaway. ## `maxResults` (type: `integer`): The maximum number of resulting web pages to store. The crawler will automatically finish after reaching this number. This setting is useful to prevent accidental crawler runaway. If both **Max page** and **Max results** are defined, then the crawler will finish when the first limit is reached. Note that the crawler skips pages with the canonical URL of a page that has already been crawled, hence it might crawl more pages than there are results. ## `chunkSize` (type: `integer`): The maximum size of each chunk in characters or tokens. ## `chunkOverlap` (type: `integer`): The number of overlapping characters or tokens between consecutive chunks. ## `crawlerType` (type: `string`):
Select the crawling engine:
## `maxCacheAge` (type: `integer`): The maximum age of cached records, in days, to consider them useful. By default, set to 30. ## `cacheKeyValueStoreName` (type: `string`): Name of the key-value store where the actor will store its cache. ## `indexWebsiteContentCrawledRunId` (type: `string`): When set, it invokes a special mode in which we index the crawl to the database, rather than answer user queries. This is invoked using webhook from Website Content Crawler, scheduled by this actor. Note that the URL and Query need to also be passed, but the URL can be dummy, as they it's ignored. ## `forceRescrape` (type: `boolean`): When set to true, the actor will ignore the cache and scrape the website again. ## Actor input object example ```json { "maxCrawlPages": 1000, "maxResults": 9999999, "chunkSize": 1000, "chunkOverlap": 200, "crawlerType": "cheerio", "maxCacheAge": 30, "cacheKeyValueStoreName": "website-content-vector-retriever-cache" } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = {}; // Run the Actor and wait for it to finish const run = await client.actor("hamza.alwan/website-content-vector-retriever").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = {} # Run the Actor and wait for it to finish run = client.actor("hamza.alwan/website-content-vector-retriever").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{}' | apify call hamza.alwan/website-content-vector-retriever --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,hamza.alwan/website-content-vector-retriever" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/9xSkC59oEysDn1XVX/builds/ly3L9DkELReuUnNfY/openapi.json