Tune LLaMa3.1 on Google Cloud TPUs

Hacker News

Hey HN, we wanted to share our repo where we fine-tuned Llama 3.1 on Google TPUs. We’re building AI infra to fine-tune and serve LLMs on non-NVIDIA GPUs (TPUs, Trainium, AMD GPUs). The problem: Right now, 90% of LLM workloads run on NVIDIA GPUs, but there are equally powerful and more cost-effective alternatives out there. For example, training and serving Llama 3.1 on Google TPUs is about 30% cheaper than NVIDIA GPUs. But developer tooling for non-NVIDIA chipsets is lacking. We felt this pain ourselves. We initially tried using PyTorch XLA to train Llama 3.1 on TPUs, but it was rough: xla integration with pytorch is clunky, missing libraries (bitsandbytes didn't work), and cryptic HuggingFace errors. We then took a different route and translated Llama 3.1 from PyTorch to JAX. Now, it’s running smoothly on TPUs! We still have challenges ahead, there is no good LoRA library in JAX, but this feels like the right path forward. Here's a demo ( https://dub.sh/felafax-demo ) of our managed solution. Would love your thoughts on our repo and vision as we keep chugging along!

Visit product View on Hacker NewsSep 11, 2024

Share card

Actual performance

189points

52comments

Made the leaderboard

Launch Intel predictions

Analyze your own launch →

Product HuntOn track for Day 1 leaderboard · Strong signals: google, using · Missing: mac, agents, macos

91%91% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

Indie HackersFits the IH revenue-focused audience · Missing: supports, reddit linkedin, podcasting

81%81% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

Hacker NewsStrong engagement from HN community · Strong signals: llama, io · Missing: https docs, excited, just released

68%68% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

TrustMRRLess likely to generate early MRR · Strong signals: google · Missing: mobile apps, ios, personal

50%50% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

AppSumoMay struggle as an AppSumo deal · Missing: plus, platform, intuitive

36%36% predicted probability of success on AppSumo, based on ML models trained on real launch data.

Acquire.comPre-revenue stage for this audience · Strong signals: training · Missing: arr, mrr, revenue

18%18% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Missing: web3, chat, crypto

0%0% predicted probability of success on BetaList, based on ML models trained on real launch data.

Correct prediction on native model

Similar products

SingChord23%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Sing a tune, get some chords.

Indie Hackerscommitment-side-project

What’s Spinning13%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Stay in Tune

Indie Hackers1analytics

Unofficial FAQ for Google Cloud Run50%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Unofficial FAQ for Google Cloud Run

Hacker News11

Datacol – Get Private Heroku on Google Cloud45%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Datacol – Get Private Heroku on Google Cloud

Hacker News16

Solace63%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Your Mac's appearance, in tune with the world around you.

Indie Hackers1$500/moproductivity

Solace55%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Your Mac's appearance, in tune with the world around you.

Indie Hackers1$300/moproductivity

Solace90%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Your Mac's appearance, in tune with the world around you.

Product Hunt+171Mac

Google Cloud Dataflow example project released56%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Google Cloud Dataflow example project released

Hacker News2

turn on, tune in, drop out. .mixtape 1.346%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

turn on, tune in, drop out. .mixtape 1.3

Hacker News2

Tune into Craterland – The Next Generation of Local32%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Tune into Craterland – The Next Generation of Local

Hacker News3

Browse all products Analyze your launch