Website Content Crawler
Pricing
Pay per usage
Website Content Crawler
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.
4.3 (43)
Pricing
Pay per usage
1.6K
66K
8.6K
Issues response
6.8 days
Last modified
6 days ago
Add Full File Name to the Key-Value-Stores
ClosedWhen running "apfy/website-content-crawler" with option: "saveFiles: true" PowerSchool would like to request that you add the full file name to your "key-value-stores". Currently the "key" is sort of like the full file name but not quite. What we're looking for is the full name along with the full URL of the file itself. The "key" in the key-value store is NOT the file's real name, it's similar but different, for one thing, it places a random 7 character string in front of the name, secondly, it html-encodes the file name AND strips off the '%' which effectively replaces things like spaces in the name (which would be "%20") with just a "20" so it is really difficult if not impossible to reconstitute the files real name. It is nearly impossible to know if a "20" represents a "%20" vs a number 20 like in year 2020, or 2021 etc.. Here is an example response from "https://api.apify.com/v2/key-value-stores/{{storeId}}/keys?token=apify_api_xxxx" { "key": "4c55c60-3-D20Curriculum202021.pdf", "size": 300429 }, Here is the full file name in the CSV Extract: https://www.dvsd.org/cms/lib/PA01001022/Centricity/Domain/1111/3-D%20Curriculum%202021.pdf
We'd like to ask if you could add the full file name to the key-value-stores
Hello and thank you for your interest in Website Content Crawler! Unfortunately, it is not possible to keep the original file name in the key value store key. However, you should be able to find them in the dataset. Apart from entries for each page, there should also be an entry for each downloaded file. The title attribute of the metadata object will contain the original filename.
CtrlAltElite
Could you describe the process of how we would be able to take the key from the key-value-store and match it to the original file name value that is stored in the title attribute of the metadata object. The values are significantly different...
CtrlAltElite
Jan ~ Please describe the process of how we would be able to take the key from the key-value-store and match it to the original file name value that is stored in the url attribute of the downloaded export file. The values are significantly different.
Here is an example response from "https://api.apify.com/v2/key-value-stores/{{storeId}}/keys?token=apify_api_xxxx"
{ "key": "4c55c60-3-D20Curriculum202021.pdf", "size": 300429 }
Here is the full file name in the CSV Extract: https://www.dvsd.org/cms/lib/PA01001022/Centricity/Domain/1111/3-D%20Curriculum%202021.pdf
How do you think we should match these two values?
Hi,
Apologies for the delayed response. When accessing the run dataset at https://api.apify.com/v2/datasets/zJiUfMvNc7eEqwg6l/items, the JSON records containing saved HTML content include an htmlUrl key. This key links to the key-value-store entry with the exported HTML file, such as https://api.apify.com/v2/key-value-stores/Y4ACP5QqXOywliqnL/records/https---blog-apify-com-how-to-build-an-ai-agent--95slqg.html.
You can export the dataset directly from the Console in the Storage section of the Actor run tab or via the API/Apify client (see https://docs.apify.com/platform/storage/dataset). To access the file, use an HTTP client or extract the key-value-store ID (e.g., Y4ACP5QqXOywliqnL) and record key (e.g., https---blog-apify-com-how-to-build-an-ai-agent--95slqg.html) from the htmlUrl and access via Apify client (https://docs.apify.com/platform/storage/key-value-store) .
Jakub
I believe the responses by Jakub and Jan already answer the question.
Each downloaded file is represented by a Dataset record containing a fileUrl field. This leads to the Key-Value store storing the actual file data. The file name (as your web browser would download it in your Downloads folder) is in the Dataset record, under metadata.title.
Note that the KVS doesn't contain only the downloaded files - it has internal data for the crawler, the Actor input, or HTML files for debugging (with debugMode: true). Because of this, it's safer to iterate over the Dataset primarily and only accessing the KVS through the links stored in the fileUrl dataset fields. This way, you'll also be able to match the file names (metadata.title) with the actual binary file data.
I'll close this issue now, but feel free to ask additional questions, if you have any. Cheers!
