Why is ML inference still so ad-hoc in practice?

Hacker News

Every place I’ve seen run more than a couple of ML models in production ends up with a mess of bespoke inference services: different APIs, different auth, different logging, half-working dashboards, and tribal knowledge holding it all together. I’ve been building a small side project that tries to standardize just the serving part — a single gateway in front of heterogeneous models (local, managed cloud, different teams) that handles inference APIs, versioning/rollback, auth, basic metrics, and health checks. No training, no AutoML, no “end-to-end MLOps platform”. Before I sink more time into it, I’m trying to figure out whether this is: a real gap people quietly paper over with internal glue, or something that sounds useful but collapses under real-world constraints. For people actually running ML in prod: Do you already have an internal inference layer like this? Where does inference usually go wrong (deployments, versioning, debugging, compliance)? At what scale does it stop being worth abstracting at all? Not announcing anything — genuinely curious whether this resonates or if I’m just rediscovering why everyone rolls their own.

Visit product View on Hacker NewsDec 26, 2025

Share card

Actual performance

4points

Did not reach leaderboard

Launch Intel predictions

Analyze your own launch →

Product HuntOn track for Day 1 leaderboard · Strong signals: model, models, single · Missing: mac, agents, macos

93%93% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

Indie HackersFits the IH revenue-focused audience · Missing: supports, reddit linkedin, podcasting

76%76% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

Hacker NewsStrong engagement from HN community · Strong signals: ide, io · Missing: https docs, excited, just released

66%66% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

TrustMRRLess likely to generate early MRR · Strong signals: way · Missing: mobile apps, ios, personal

47%47% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

AppSumoMay struggle as an AppSumo deal · Strong signals: platform · Missing: plus, intuitive, reviews

27%27% predicted probability of success on AppSumo, based on ML models trained on real launch data.

Acquire.comPre-revenue stage for this audience · Strong signals: training · Missing: arr, mrr, revenue

22%22% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Missing: web3, chat, crypto

0%0% predicted probability of success on BetaList, based on ML models trained on real launch data.

Incorrect prediction on native model

Similar products

Graphsignal – ML profiler to speed up training and inference65%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Graphsignal – ML profiler to speed up training and inference

Hacker News16

Weed–Minimalist AI/ML inference and backprogation in the style of Qrack39%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Weed–Minimalist AI/ML inference and backprogation in the style of Qrack

Hacker News1

Pyinfer – A tool to benchmark inference statistics for any ML model50%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Pyinfer – A tool to benchmark inference statistics for any ML model

Hacker News1

Cellulose – a tool to improve inference performance of ML models69%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Cellulose – a tool to improve inference performance of ML models

Hacker News11

Inference GUIs for 12 SoTA ML models64%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Inference GUIs for 12 SoTA ML models

Hacker News9

TQNN55%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Fault-tolerant inference for noisy and imperfect data.

Indie Hackers1ai

The cheapest ML inference API on A100 GPUs for your apps60%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

The cheapest ML inference API on A100 GPUs for your apps

Hacker News3

We just launched MegaAI. It's a 4k30fps, 4W, 4TOPS inference powerhouse69%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

We just launched MegaAI. It's a 4k30fps, 4W, 4TOPS inference powerhouse

Hacker News3

An annotation tool for ML and NLP63%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

An annotation tool for ML and NLP

Hacker News76

An annotation tool for ML and NLP63%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

An annotation tool for ML and NLP

Hacker News5

Browse all products Analyze your launch