Website Content Crawler
Pricing
Pay per usage
Website Content Crawler
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.
4.6 (38)
Pricing
Pay per usage
1224
Monthly users
6.4k
Runs succeeded
>99%
Response time
4.6 days
Last modified
a day ago
Add Full File Name to the Key-Value-Stores
Open
When running "apfy/website-content-crawler" with option: "saveFiles: true" PowerSchool would like to request that you add the full file name to your "key-value-stores". Currently the "key" is sort of like the full file name but not quite. What we're looking for is the full name along with the full URL of the file itself. The "key" in the key-value store is NOT the file's real name, it's similar but different, for one thing, it places a random 7 character string in front of the name, secondly, it html-encodes the file name AND strips off the '%' which effectively replaces things like spaces in the name (which would be "%20") with just a "20" so it is really difficult if not impossible to reconstitute the files real name. It is nearly impossible to know if a "20" represents a "%20" vs a number 20 like in year 2020, or 2021 etc.. Here is an example response from "https://api.apify.com/v2/key-value-stores/{{storeId}}/keys?token=apify_api_xxxx" { "key": "4c55c60-3-D20Curriculum202021.pdf", "size": 300429 }, Here is the full file name in the CSV Extract: https://www.dvsd.org/cms/lib/PA01001022/Centricity/Domain/1111/3-D%20Curriculum%202021.pdf
We'd like to ask if you could add the full file name to the key-value-stores
Hello and thank you for your interest in Website Content Crawler! Unfortunately, it is not possible to keep the original file name in the key value store key. However, you should be able to find them in the dataset. Apart from entries for each page, there should also be an entry for each downloaded file. The title attribute of the metadata object will contain the original filename.
CtrlAltElite
Could you describe the process of how we would be able to take the key from the key-value-store and match it to the original file name value that is stored in the title attribute of the metadata object. The values are significantly different...
CtrlAltElite
Jan ~ Please describe the process of how we would be able to take the key from the key-value-store and match it to the original file name value that is stored in the url attribute of the downloaded export file. The values are significantly different.
Here is an example response from "https://api.apify.com/v2/key-value-stores/{{storeId}}/keys?token=apify_api_xxxx"
{ "key": "4c55c60-3-D20Curriculum202021.pdf", "size": 300429 }
Here is the full file name in the CSV Extract: https://www.dvsd.org/cms/lib/PA01001022/Centricity/Domain/1111/3-D%20Curriculum%202021.pdf
How do you think we should match these two values?
Pricing
Pricing model
Pay per usageThis Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage.
