Open-source ETL framework to sync data from SaaS tools to vector stores

Hacker News

Hey hacker news, we launched a few weeks ago as a GPT-powered chatbot for developer docs, and quickly realized that the value of what we’re doing isn’t the chatbot itself. Rather, it’s the time we save developers by automating the extraction of data from their SaaS tools (Github, Zendesk, Salesforce, etc) and helping transform it to contextually relevant chunks that fit into GPT’s context window. A lot of companies are building prototypes with GPT right now and they’re all using some combination of Langchain/Llama Index + Weaviate/Pinecone + GPT3.5/GPT4 as their stack for retrieval augmented generation (RAG). This works great for prototypes, but what we learned was that as you scale your RAG app to more users and ingest more sources of content, it becomes a real pain to manage your data pipelines. For example, if you want to ingest your developer docs, process it into chunks of <500 tokens, and add those chunks to a vector store, you can build a prototype with Langchain fairly quickly. However, if you want to deploy it to customers like we did for BentoML ([ https://www.bentoml.com/](https://www.bentoml.com/) ) you’ll quickly realize that a naive chunking method that splits by character/token leads to poor results, and that “delete and re-vectorize everything” when the source docs change doesn’t scale as a data synchronization strategy. We took the code we used to build chatbots for our early customers and turned it into an open source framework to rapidly build new data Connectors and Chunkers. This way developers can use community built Connectors and Chunkers to start running vector searches on data from any source in a matter of minutes, or write their own in a matter of hours. Here’s a video demo: [ https://youtu.be/I2V3Cu8L6wk](https://youtu.be/I2V3Cu8L6wk) The repo has instructions on how to get started and set up API endpoints to load, chunk, and vectorize data quickly. Right now it only works with websites and Github repos, but we’ll be adding Zendesk, Google Drive, and Confluence integrations soon too.

Visit product View on Hacker NewsMar 30, 2023

Share card

Actual performance

63points

8comments

Made the leaderboard

Launch Intel predictions

Analyze your own launch →

Product HuntOn track for Day 1 leaderboard · Strong signals: google, user, new · Missing: mac, agents, macos

97%97% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

Indie HackersFits the IH revenue-focused audience · Strong signals: started · Missing: supports, reddit linkedin, podcasting

91%91% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

Hacker NewsStrong engagement from HN community · Strong signals: open source, llama, hacker news · Missing: https docs, excited, just released

80%80% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

TrustMRRFits verified-revenue profile · Strong signals: video, google, users · Missing: mobile apps, ios, personal

53%53% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

AppSumoMay struggle as an AppSumo deal · Strong signals: soon, users · Missing: plus, platform, intuitive

32%32% predicted probability of success on AppSumo, based on ML models trained on real launch data.

Acquire.comPre-revenue stage for this audience · Strong signals: saas · Missing: arr, mrr, revenue

18%18% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Strong signals: chat · Missing: web3, crypto, cryptocurrency

0%0% predicted probability of success on BetaList, based on ML models trained on real launch data.

Correct prediction on native model

Similar products

ETL Framework for Vector Databases (Open Source)65%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

ETL Framework for Vector Databases (Open Source)

Hacker News2

Duckle was benchmarked against 4 open source ELT/ETL tools71%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Duckle was benchmarked against 4 open source ELT/ETL tools

Hacker News2

DataDuck ETL – Ruby framework for setting up a Redshift data warehouse62%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

DataDuck ETL – Ruby framework for setting up a Redshift data warehouse

Hacker News7

VDP – open-source unstructured visual data ETL80%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

VDP – open-source unstructured visual data ETL

Hacker News11

We made an open-source visual data ETL74%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

We made an open-source visual data ETL

Hacker News8

BemiDB – Open-source data warehouse with zero-ETL82%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

BemiDB – Open-source data warehouse with zero-ETL

Hacker News13

VDP34%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Open-source unstructured data ETL infrastructure

Indie Hackers3ai

No-Code ETL Framework for Vector Databases65%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

No-Code ETL Framework for Vector Databases

Hacker News11

Open-Source Plaid for Ticketing SaaS Tools66%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Open-Source Plaid for Ticketing SaaS Tools

Hacker News4

Singer – Simple, Composable Open Source ETL71%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Singer – Simple, Composable Open Source ETL

Hacker News91

Browse all products Analyze your launch