Saving Money Deploying Open Source AI at Scale with Kubernetes

Hacker News

Hi HN: I wanted to share this piece I wrote on how I saved our small startup 10s of thousands of dollars every month by lifting and shifting or AI data-pipelines from using OpenAI's API to a vLLM deployment ontop of Kubernetes running on a few nodes with T4 GPUs. I haven't seen alot on the "AI-DevOps" or infrastructure side of actually running an at-scale AI service. Many of the AI inference engines that offer an OpenAI compatible API (like vLLM, llama.cpp, etc.) make it very approachable and cost effective. Today, this vLLM AI service handles all of our batching micro-services which scrape for content to generate text on over 40,000+ repos on GitHub. I'm happy to answer any / all questions you might have!

Visit product View on Hacker NewsMay 14, 2024

Share card

Actual performance

3points

2comments

Did not reach leaderboard

Launch Intel predictions

Analyze your own launch →

Product HuntOn track for Day 1 leaderboard · Strong signals: openai, using, open · Missing: mac, agents, macos

89%89% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

Hacker NewsStrong engagement from HN community · Strong signals: open source, llama, ide · Missing: https docs, excited, just released

69%69% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

Indie HackersFits the IH revenue-focused audience · Strong signals: compatible · Missing: supports, reddit linkedin, podcasting

60%60% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

TrustMRRFits verified-revenue profile · Strong signals: month · Missing: mobile apps, ios, personal

57%57% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

AppSumoMay struggle as an AppSumo deal · Missing: plus, platform, intuitive

26%26% predicted probability of success on AppSumo, based on ML models trained on real launch data.

Acquire.comPre-revenue stage for this audience · Missing: arr, mrr, revenue

22%22% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Missing: web3, chat, crypto

4%4% predicted probability of success on BetaList, based on ML models trained on real launch data.

Incorrect prediction on native model

Similar products

Open-source AI assistant for Kubernetes troubleshooting45%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Open-source AI assistant for Kubernetes troubleshooting

Hacker News1

Bakery By Bagel68%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Monetizable open source AI

Product Hunt+190Open Source

Extrapolate – Open-Source AI Aging App56%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Extrapolate – Open-Source AI Aging App

Hacker News5

I made an open-source AI Headshot Generator57%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

I made an open-source AI Headshot Generator

Hacker News3

I built an open-source AI system for drones69%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

I built an open-source AI system for drones

Hacker News5

InfiniteGpu, An open-source AI computational network63%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

InfiniteGpu, An open-source AI computational network

Hacker News2

Playcall80%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

The open-source AI alternative to Gong

Product Hunt+96Sales

CopilotTextarea/> = an open-source, AI-infused react <textarea>54%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

CopilotTextarea/> = an open-source, AI-infused react <textarea>

Hacker News2

The Internet's Open Source AI Paywall52%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

The Internet's Open Source AI Paywall

Hacker News13

Xalgorix34%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Open-source AI pentester with exploit-verified findings

Indie Hackers

Browse all products Analyze your launch