API to turn entire websites into Markdown

Hacker News

While building mendable - we found that feeding LLMs well-structured markdown improved accuracy. We also found it surprisingly hard. We found some great tools online, but none reliably handled the entire process. We wanted an API that took a URL, crawled the pages in the URL, and gave us an easy-to-use, up-to-date markdown we could feed into our index. So, we released an open-source repo and an API that crawls and turns entire websites into a markdown with just a few lines of code The API handles: - Crawling without consistent sitemaps - Infra to handle running many crawling jobs - Proxying, hosting headless browsers at scale - Conversion to clean markdown - Caching - Handling images, videos (soon), and tables(soon) - LLM extraction (soon) It is open source, and we also offer an easy-to-use API that starts free. It has built-in loaders for both @llama_index and @langchain. Excited to see people try it

Visit product View on Hacker NewsApr 16, 2024

Share card

Actual performance

23points

4comments

Made the leaderboard

Launch Intel predictions

Analyze your own launch →

Product HuntOn track for Day 1 leaderboard · Strong signals: code, open · Missing: mac, agents, macos

86%86% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

Hacker NewsStrong engagement from HN community · Strong signals: excited, open source, llama · Missing: https docs, just released, exist

74%74% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

Indie HackersFits the IH revenue-focused audience · Missing: supports, reddit linkedin, podcasting

73%73% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

TrustMRRLess likely to generate early MRR · Strong signals: video · Missing: mobile apps, ios, personal

38%38% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

AppSumoMay struggle as an AppSumo deal · Strong signals: host, soon · Missing: plus, platform, intuitive

34%34% predicted probability of success on AppSumo, based on ML models trained on real launch data.

Acquire.comPre-revenue stage for this audience · Missing: arr, mrr, revenue

19%19% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Missing: web3, chat, crypto

0%0% predicted probability of success on BetaList, based on ML models trained on real launch data.

Correct prediction on native model

Similar products

Turn Any JSDoc, Javadoc or PHPDoc into Markdown70%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Turn Any JSDoc, Javadoc or PHPDoc into Markdown

Hacker News3

I made a GitHub Flavored Markdown API69%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

I made a GitHub Flavored Markdown API

Hacker News12

ApiFrom37%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Extract websites data and turn into an API

Indie Hackers

Create documentation websites from Markdown68%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Create documentation websites from Markdown

Hacker News4

Upsite.markdown – Beautiful websites with a few lines of markdown74%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Upsite.markdown – Beautiful websites with a few lines of markdown

Hacker News4

Publish websites from your markdown notes.80%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Publish websites from your markdown notes.

Hacker News10

Godown.vim – Markdown Previewer in Go59%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Godown.vim – Markdown Previewer in Go

Hacker News1

Extending Markdown with Pandoc+Panflute62%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Extending Markdown with Pandoc+Panflute

Hacker News4

Hyperfiddle Markdown62%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Hyperfiddle Markdown

Hacker News1

Markdown codeblocks preprocessor62%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Markdown codeblocks preprocessor

Hacker News2

Browse all products Analyze your launch