Ma
Made a batching LLM API for a project. Mistral 200 tk/s on RTX 3090
Hacker NewsMade a batching LLM API for a project. Mistral 200 tk/s on RTX 3090
I was running into an issue with a vLLM bug that affected multiple GPUs and I needed a stand-in while that bug was getting fixed that used the same API format but had better performance than the API on text-generation-webui. It's very rough. I'm not a coder by trade. But it's very fast once you have many simultaneous connections.
Share cardActual performance
3points
Did not reach leaderboard
Launch Intel predictions
Analyze your own launch →
Product HuntOn track for Day 1 leaderboard · Strong signals: code · Missing: mac, agents, macos71%71% predicted probability of success on Product Hunt, based on ML models trained on real launch data.
best fitHighest predicted score across all platforms for this description.
Indie HackersFits the IH revenue-focused audience · Missing: supports, reddit linkedin, podcasting61%61% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.
TrustMRRFits verified-revenue profile · Missing: mobile apps, ios, personal56%56% predicted probability of success on TrustMRR, based on ML models trained on real launch data.
Hacker NewsStrong engagement from HN community · Strong signals: io · Missing: https docs, excited, just released51%51% predicted probability of success on Hacker News, based on ML models trained on real launch data.
nativeThis product was originally launched on this platform.
AppSumoMay struggle as an AppSumo deal · Missing: plus, platform, intuitive39%39% predicted probability of success on AppSumo, based on ML models trained on real launch data.
Acquire.comPre-revenue stage for this audience · Missing: arr, mrr, revenue21%21% predicted probability of success on Acquire.com, based on ML models trained on real launch data.
BetaListMay not resonate with beta-testers · Missing: web3, chat, crypto4%4% predicted probability of success on BetaList, based on ML models trained on real launch data.
Incorrect prediction on native model
Similar products
Da
Daily ETF holdings for 2,200+ ETFs as one API53%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.
Daily ETF holdings for 2,200+ ETFs as one API
Hacker News3C+
C++ LMAX Disruptor < 200 LOC45%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.
C++ LMAX Disruptor < 200 LOC
Hacker News3Матрац на ліжко 160 на 20031%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.
Двоспальний матрац: інженерний компроміс між двома тілами
Indie Hackers20
200 AI Conferences in 201844%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.
200 AI Conferences in 2018
Hacker News2
LLM API84%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.
Use any LLM with just one API
Product Hunt+7Mistral61%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.
Kickstart your project with pre-made TailwindCSS blocks
Indie Hackers7$772/modesignLe
Less‑filtered LLM chat and API50%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.
Less‑filtered LLM chat and API
Hacker News1Tu
Turnkey API monetization and consumption management – Project X40%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.
Turnkey API monetization and consumption management – Project X
Hacker News1My
MyGPT a toy LLM which can be trained on Project Gutenberg and dad jokes60%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.
MyGPT a toy LLM which can be trained on Project Gutenberg and dad jokes
Hacker News25Project X53%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.
Turnkey monetization for API developers.
Indie Hackers2apis