a Rust-based multimodal inference server

Hacker News

We built a production-grade multimodal inference server in Rust for serving vision–language models (image + text → streamed text). The goal was to explore what a Rust-native control plane looks like for modern multimodal inference: continuous batching, KV-aware admission control, predictable behavior under load, and proper streaming semantics. The system exposes an OpenAI-compatible API, supports multi-image inputs, and is designed to degrade gracefully under overload rather than OOM or stall. It’s organized as a single monorepo with a gateway, GPU workers, scheduler, and pluggable engine adapters. We’ve also included a benchmark suite focused on real-world scenarios (TTFT, cancellation, overload, fairness) rather than synthetic tokens/sec numbers. Would love feedback from folks building or operating inference infrastructure.

Visit product View on Hacker NewsJan 26, 2026

Share card

Actual performance

1points

Did not reach leaderboard

Launch Intel predictions

Analyze your own launch →

Product HuntOn track for Day 1 leaderboard · Strong signals: model, models, openai · Missing: mac, agents, macos

87%87% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

Indie HackersFits the IH revenue-focused audience · Strong signals: supports, ios, compatible · Missing: reddit linkedin, podcasting, created

87%87% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

Hacker NewsStrong engagement from HN community · Strong signals: io · Missing: https docs, excited, just released

70%70% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

TrustMRRFits verified-revenue profile · Strong signals: ios, way · Missing: mobile apps, personal, entrepreneurs

61%61% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

AppSumoMay struggle as an AppSumo deal · Missing: plus, platform, intuitive

36%36% predicted probability of success on AppSumo, based on ML models trained on real launch data.

Acquire.comPre-revenue stage for this audience · Missing: arr, mrr, revenue

19%19% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Missing: web3, chat, crypto

1%1% predicted probability of success on BetaList, based on ML models trained on real launch data.

Incorrect prediction on native model

Similar products

Rust Powered Inference, Ingestion and Indexing with EmbedAnything74%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Rust Powered Inference, Ingestion and Indexing with EmbedAnything

Hacker News1

Mighty Inference Server64%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Mighty Inference Server

Hacker News6

VLM Inference Engine in Rust62%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

VLM Inference Engine in Rust

Hacker News1

NOS – A fast, and ergonomic PyTorch inference server69%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

NOS – A fast, and ergonomic PyTorch inference server

Hacker News3

Orkhon – ML Inference Framework and Server Runtime (Rust and Python)67%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Orkhon – ML Inference Framework and Server Runtime (Rust and Python)

Hacker News1

Rooster – Personal Web Server with Rust60%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Rooster – Personal Web Server with Rust

Hacker News10

MetricWhiz a Rust based TUI for NLP binary classification analysis58%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

MetricWhiz a Rust based TUI for NLP binary classification analysis

Hacker News1

An example GraphQL server written in Rust80%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

An example GraphQL server written in Rust

Hacker News74

Warp, a Rust-based terminal83%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Warp, a Rust-based terminal

Hacker News946

Godot and Rust based multiplexer (terminal panes and more)68%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Godot and Rust based multiplexer (terminal panes and more)

Hacker News97

Browse all products Analyze your launch