ColiVara – State of the Art RAG API with Vision Models

Hacker News

we have been working on ColiVara and wanted to show it to the community. ColiVara is an api-first implementation of the ColPali paper using ColQwen2 as the LLM model. It works exactly like RAG from the end-user standpoint - but using vision models instead of chunking and text-processing for documents. Why should anyone working with RAG care? ColPali makes information retrieval from visual document types - like PDFs - better. Colivara is a suite of services that allows you to store, search, and retrieve documents based on their visual embedding built on top of ColPali. (We are not affiliated with the ColPali team in anyway, although we are big fans of their work!) Information retrieval from PDFs is hard because they contain various components: Text, images, tables, different headings, captions, complex layouts, etc. For this, parsing PDFs currently requires multiple complex steps: 1. OCR 2. Layout recognition 3. Figure captioning 4. Chunking 5. Embedding Not only are these steps complex and time-consuming, but they are also prone to error. This is where ColPali comes into play. But what is ColPali? ColPali combines: • Col -> the contextualized late interaction mechanism introduced in ColBERT • Pali -> with a Vision Language Model (VLM), in this case, PaliGemma (note - both us and the ColPali team moved from PaliGemma to use Qwen models) And how does it work? During indexing, the complex PDF parsing steps are replaced by using "screenshots" of the PDF pages directly. These screenshots are then embedded with the VLM. At inference time, the query is embedded and matched with a late interaction mechanism to retrieve the most similar document pages. Ok - so what exactly ColiVara does? ColiVara is an API (with a Python SDK) that makes this whole process easy and viable for production workloads. With 1-line of code - you get a SOTA retrieval in your RAG system. We optimized how the embeddings are stored (using pgVector and halfvecs) as well as re-implemented the scoring to happen in Postgres, similar to and building on pgVector work with Cosine Similarity. All what the user have to do is: 1. Upsert a document to ColiVara to index it 2. At query time - perform a search and get the top-k pages We support advanced filtering based on arbitrary document and collection metadata as well. So, we support re-ranking use cases and hybrid search. State of the art? We started this whole journey when we tried to do RAG over clinical trials and medical literature. We simply had too many failures and up to 30% of the paper was lost or malformed. This is just not our experience, in the ColPali paper - on average ColPali outperformed Unstructured + BM25 + captioning by 15+ points. ColiVara with its optimizations is is 20+ points. We used NCDG@5 - which is similar to Recall but more demanding, as it measure not just if the right results are returned, but if they returned in the correct order. You can see our full eval results here: https://github.com/tjmlabs/ColiVara-eval If this sounds like something you could use, check it out on GitHub: https://github.com/tjmlabs/ColiVara It’s fair-source with an FSL license (similar to Sentry), and we’d love to hear how you’d use it or any feedback you might have. Additionally - our eval repo is public and we continuously run against major releases. You are welcome to run the evals independently: https://github.com/tjmlabs/ColiVara-eval

Visit product View on Hacker NewsNov 16, 2024

Share card

Actual performance

10points

Made the leaderboard

Launch Intel predictions

Analyze your own launch →

Product HuntOn track for Day 1 leaderboard · Strong signals: model, user, models · Missing: mac, agents, macos

93%93% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

Indie HackersFits the IH revenue-focused audience · Strong signals: started · Missing: supports, reddit linkedin, podcasting

83%83% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

Hacker NewsStrong engagement from HN community · Strong signals: io · Missing: https docs, excited, just released

75%75% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

AppSumoMay struggle as an AppSumo deal · Missing: plus, platform, intuitive

45%45% predicted probability of success on AppSumo, based on ML models trained on real launch data.

TrustMRRLess likely to generate early MRR · Strong signals: way · Missing: mobile apps, ios, personal

30%30% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

Acquire.comPre-revenue stage for this audience · Missing: arr, mrr, revenue

11%11% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Strong signals: introduce · Missing: web3, chat, crypto

0%0% predicted probability of success on BetaList, based on ML models trained on real launch data.

Correct prediction on native model

Similar products

Last System State Before OOM49%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Last System State Before OOM

Hacker News1

SignalCend – API that resolves conflicting IoT device state in 47ms54%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

SignalCend – API that resolves conflicting IoT device state in 47ms

Hacker News5

Vision-Based, Vectorless RAG for Long Douments66%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Vision-Based, Vectorless RAG for Long Douments

Hacker News6

Vectorless RAG48%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Vectorless RAG

Hacker News11

PageIndex – Vectorless RAG66%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

PageIndex – Vectorless RAG

Hacker News192

Preprocess65%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Preprocess maximises RAG performances

Product Hunt+106API

Backprop – a simple library to use and finetune state-of-the-art models59%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Backprop – a simple library to use and finetune state-of-the-art models

Hacker News5

Quickai – Quickly experiment with state-of-the-art ML models63%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Quickai – Quickly experiment with state-of-the-art ML models

Hacker News2

Backprop – a library to easily finetune and use state-of-the-art models59%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Backprop – a library to easily finetune and use state-of-the-art models

Hacker News4

A state of the art retrieval API on both text and visual documents69%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

A state of the art retrieval API on both text and visual documents

Hacker News1

Browse all products Analyze your launch