Zero downtime embedding model upgrades

Hacker News

Share card

Zero downtime embedding model upgrades

People use embedding models all the time for rag/semantic retrieval. However, when a newer, more desireable model comes out, there is an expensive (both in time and computational) cost of re-embedding every document in the database. However, I figured out an interesting way to forgo that upfront embedding cost. algo: old model/index -> retrieve top-K docs -> score those docs with the new model -> cache/materialize the new embeddings so instead of rebuilding the entire vector store upfront, the old index keeps getting retrieved from, while the new model reranks those candidates. This works surprisingly well for some model pairs, (i tested 63 source-> target migrations on h100s, on upto 1M documents). For example, on a 1M document Natural Questions dataset, native Qwen3-Embedding-8B: 0.6812 nDCG@10 Qwen3-4B -> Qwen3-8B, K=50: 0.6816 Qwen3-0.6B -> Qwen3-8B, K=50: 0.6638 MiniLM -> Qwen3-8B, K=50: 0.6486 (the hard part is determining k, I held the k constant above to give some sense of migratability). You can install it with pip pip install embedflow and the code is on github https://github.com/arnsri33/embedflow