Ranking LLMs by Usage over Time

Hacker News

Share card

Ranking LLMs by Usage over Time

Hey HN, We've been building a multi-model API for LLMs called OpenRouter since we launched the Window AI extension in April this year [0]. The events of the last week make it clear the LLM landscape is unpredictable, so LLM aggregators are picking up interest. Unlike others, we're making a router on top of a public, explorable dataset. We recently built a way to rank and visualize this data, including the token counts we see going to and from different models, both open-sourced (like Llama, Mistral, finetunes, and variants) and closed-sourced (OpenAI, Anthropic). The API supports - 50+ different models: [0] - Consolidated payments for all models - OAuth, so users can pay for usage directly - Upstream latency/throughput tracking - Multiple providers per model, for redundancy (downtime happens!) - Prompt compression, so you don't have to worry as much about context length Docs: [1] We support 11 model hosts, including our own, based on vLLM, which we've just open-sourced: [2] Some users have opted-in to sharing their prompts, which will soon allow us to show which models are best for different tasks. Let us know your feedback, and if you've worked on a similar problem before! Alex and Louis [0] https://news.ycombinator.com/item?id=35481760 [1] https://openrouter/models [2] https://openrouter.ai/docs [3] https://github.com/OpenRouterTeam/openrouter-runner