Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB

Hacker News

I built an experiment that uses an overfitted transformer and arithmetic coding to compress individual files. Instead of training the model to generalize, I train a 900KB transformer to memorize a single file and predict the next byte. Those predictions are fed into an arithmetic coder to produce the compressed output. On a 100MB NYC taxi CSV, it compresses to about 7MB (~0.5 bits/byte). On a 100MB slice of enwik9, it compresses to about 21MB (~1.68 bits/byte). It's pretty slow right now (roughly 20–30 minutes of training and 45 minutes each for compression and decompression on my AMD 7800XT). Checkout the repo - https://github.com/samyak112/pym-particles

Visit product View on Hacker NewsJun 23, 2026

Share card

Actual performance

112points

71comments

Made the leaderboard

Launch Intel predictions

Analyze your own launch →

Hacker NewsStrong engagement from HN community · Strong signals: io · Missing: https docs, excited, just released

59%59% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

TrustMRRLess likely to generate early MRR · Missing: mobile apps, ios, personal

35%35% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

AppSumoMay struggle as an AppSumo deal · Missing: plus, platform, intuitive

32%32% predicted probability of success on AppSumo, based on ML models trained on real launch data.

Product HuntUnlikely to reach the leaderboard · Strong signals: model, single, coding · Missing: mac, agents, macos

30%30% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

Indie HackersIH features products with proven revenue · Missing: supports, reddit linkedin, podcasting

24%24% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

Acquire.comPre-revenue stage for this audience · Strong signals: training · Missing: arr, mrr, revenue

22%22% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Missing: web3, chat, crypto

5%5% predicted probability of success on BetaList, based on ML models trained on real launch data.

Correct prediction on native model

Similar products

CSV Driller48%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

CSV Driller

Hacker News5

Pastebin for CSV56%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Pastebin for CSV

Hacker News12

CSV Analyse30%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Zero-Hallucination CSV Analyser.

Indie Hackers1data

Sqlite3-dump - a fast SQLite to CSV and parquet70%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Sqlite3-dump - a fast SQLite to CSV and parquet

Hacker News19

RSV, a simple CSV unmarshaller in Go54%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

RSV, a simple CSV unmarshaller in Go

Hacker News2

JpgCompress66%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Compress JPEG image

Indie Hackerscommitment-side-project

Optimage 2 – compress images thoroughly48%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Optimage 2 – compress images thoroughly

Hacker News11

CSV for Modern C++65%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

CSV for Modern C++

Hacker News1

CSV file merging55%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

CSV file merging

Hacker News29

Compress Test37%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Compress Test

Hacker News3

Browse all products Analyze your launch