Deep learning framework from scratch, trains GPT-2 in 3 days

Hacker News

I wanted to share Tricycle ( https://github.com/bclarkson-code/Tricycle ), a deep learning framework I built completely from scratch from Autograd to a GPT. I wanted a library that is fast and feature rich enough to train actual models while being simple enough that anyone with a bit of python experience can understand what is going on. The biggest milestone so far is training GPT-2 (124M) on 2.3B tokens in just under 3 days on my GPU (RTX 3090). So far, I've added the following to Tricycle: - An automatic differentiation engine - General matrix operations with einsum - Standard network layers (Dense, ReLU, GeLU etc) - Transformer blocks (MultiHeadSelfAttention and MLP blocks) - Optimisers (SGD, AdamW) - GPT-2 - etc The project is still under active development, I'm in the process of adding mixed precision and multi-gpu support with the goal of scaling up to larger models. To see it in action, the best place to start is train_smol_gpt.py which will train GPT-2 from scratch. Let me know what you think!

Visit product View on Hacker NewsJul 17, 2024

Share card

Actual performance

2points

Did not reach leaderboard

Launch Intel predictions

Analyze your own launch →

Indie HackersFits the IH revenue-focused audience · Missing: supports, reddit linkedin, podcasting

85%85% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

Hacker NewsStrong engagement from HN community · Strong signals: io · Missing: https docs, excited, just released

68%68% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

AppSumoStrong fit for a featured deal · Missing: plus, platform, intuitive

54%54% predicted probability of success on AppSumo, based on ML models trained on real launch data.

Product HuntOn track for Day 1 leaderboard · Strong signals: model, models, code · Missing: mac, agents, macos

52%52% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

TrustMRRLess likely to generate early MRR · Missing: mobile apps, ios, personal

44%44% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

Acquire.comPre-revenue stage for this audience · Strong signals: training, active · Missing: arr, mrr, revenue

20%20% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Missing: web3, chat, crypto

0%0% predicted probability of success on BetaList, based on ML models trained on real launch data.

Incorrect prediction on native model

Similar products

*From Scratch* Implementations of Deep Learning Concepts (in Numpy)71%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

*From Scratch* Implementations of Deep Learning Concepts (in Numpy)

Hacker News2

Deep Learning Framework from scratch (60 step tutorial)67%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Deep Learning Framework from scratch (60 step tutorial)

Hacker News40

Mariana, the Cutest Deep Learning Framework68%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Mariana, the Cutest Deep Learning Framework

Hacker News9

Picard – Framework for exploring deep learning architectures64%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Picard – Framework for exploring deep learning architectures

Hacker News15

Bezos: build your own (deep) reinforcement learning framework62%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Bezos: build your own (deep) reinforcement learning framework

Hacker News2

Bender – a Deep Learning framework for iOS62%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Bender – a Deep Learning framework for iOS

Hacker News81

Simple RPC Framework in Golang from Scratch25%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Simple RPC Framework in Golang from Scratch

Hacker News1

GPT-2 from scratch in PyTorch – Video tutorial38%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

GPT-2 from scratch in PyTorch – Video tutorial

Hacker News1

Building a GPT from scratch: What I learned and why it mattered67%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Building a GPT from scratch: What I learned and why it mattered

Hacker News2

I built a deep learning engine from scratch in Python60%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

I built a deep learning engine from scratch in Python

Hacker News30

Browse all products Analyze your launch