We unified LLMs, vector memory, ranking, pruning models in one process

Hacker News

Share card

We unified LLMs, vector memory, ranking, pruning models in one process

There is a lot of latency involved shuffling data for modern and complex ML systems in production. In our experience these costs dominate end-to-end user latency, rather than actual model or ANN algorithms, which unfortunately limits what is achievable for interactive applications. We've extended Postgres w/ open source models from Huggingface, as well as vector search, and classical ML algos, so that everything can happen in the same process. It's significantly faster and cheaper, which leaves a large latency budget available to expand model and algorithm complexity. In addition open source models have already surpassed OpenAI's text-embedding-ada-002 in quality, not just speed. [1] Here is a series of posts explaining how to accomplish the complexity involved in a typical ML powered application, as a single SQL query, that runs in a single process with memory shared between models and feature indexes, including learned embeddings and reranking models. - Generating LLM embeddings with open source models in the database[2] - Tuning vector recall [3] - Personalize embedding results with application data [4] This allows a single SQL query to accomplish what would normally be an entire application w/ several model services and databases e.g. for a modern chatbot built across various services and databases -> application sends user input data to embedding service <- embedding model generates a vector to send back to application -> application sends vector to vector database <- vector database returns associated metadata found via ANN -> application sends metadata for reranking <- reranking model prunes less helpful context -> application sends finished prompt w/ context to generative model <- model produces final output -> application streams response to user [1]: https://huggingface.co/spaces/mteb/leaderboard [2]: https://postgresml.org/blog/generating-llm-embeddings-with-open-source-models-in-postgresml [3]: https://postgresml.org/blog/tuning-vector-recall-while-generating-query-embeddings-in-the-database [4]: https://postgresml.org/blog/personalize-embedding-vector-search-results-with-huggingface-and-pgvector Github: https://github.com/postgresml/postgresml