Build, test, and deploy your own LLM router

Hacker News

Share card

Build, test, and deploy your own LLM router

We created an LLM router to improve quality and reduce costs for LLM applications. We started this project some time ago and used the router internally due to our own experience developing LLM solutions. We realized that there were many challenges when working on projects with LLMs: First, the quality of the model. Every week, new models are released with improved capabilities, so the projects we built months ago become somewhat obsolete (and changing the model can be a pain). Second, costs. As we scaled the solutions, we realized that cost is a limiting factor for real use cases. Also, every 3 months, there are drastic changes in prices, so many ongoing LLM solutions today are probably paying more than they need to. Third, why should we use only one model? Using a multi-model approach offers a much better opportunity to get the best out of every model, reduce costs, and more. With that in mind, we started mapping models to their best performance in tasks (like coding, creative writing, etc.) and also mapping them by topics (Finance, Tech, Marketing, etc.). We included other factors like cost and latency to create our optimal solution. Today, the research we conducted became what is now the first version of our router, which redirects prompts to the model selected by the user based on the difficulty of each prompt. Try it out and let us know what you think! You can create a custom router, test it in the chat interface, and later, once you have some conversations, create evaluations to compare the router’s performance with a single-model approach. This is the first version of the project! We have tons of other ideas/prototypes that we are adding to the platform in the short term (new types of routers, automatic model selection based on sample prompts, model usage suggestions, calibration metrics, and more). We are keen to receive feedback from the community.