Website Content Crawler
No credit card required
Website Content Crawler
No credit card required
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.
Do you want to learn more about this Actor?
Get a demowhen exporting the results of a scrape, the markdown has broken hyperlinks such as:
[
Admin & Relations
](/careers_category/admin-relations)
is it possible to remove the unwanted space after [ and before ] so that the markdown is as expected?
Hello, and thank you for your interest in the Actor!
Website Content Crawler uses the Turndown library to convert from HTML to markdown. I have verified that the library behaves like you describe:
1import TurndownService from 'turndown'; 2const str = `<a href="/careers/career-fields/admin-relations" class="promo-box tab-focus" style="background-image: url("https://content.nationalguard.com/sites/default/files/adminrelations2023.jpg");"><div class="inner"><h4 class="promo-title">Admin & Relations</h4> <h6> 3 Learn more. 4 <i class="fas fa-chevron-right" aria-hidden="true"></i></h6></div> <div class="overlay"></div></a>`; 5const td = TurndownService({headingStyle: 'atx', codeBlockStyle: 'fenced'}); 6td.turndown(str); // returns broken markdown link
I'm afraid that there is nothing we can do on our end. You can open an issue with turndown (https://github.com/mixmark-io/turndown), or you could remove the offending links completely with the removeElementsCssSelector
option. Or you could ignore the markdown produced by Website Content Crawler and perform the conversion using a different tool after you download the dataset (using the HTML as input - you need to enable that with the "Save HTML" option).
Hope that helps!
Actor Metrics
3.9k monthly users
-
708 stars
>99% runs succeeded
2.1 days response time
Created in Mar 2023
Modified 17 days ago