We scaled Git to support 1 TB repos

Hacker News

I’ve been in the MLOps space for ~10 years, and data is still the hardest unsolved open problem. Code is versioned using Git, data is stored somewhere else, and context often lives in a 3rd location like Slack or GDocs. This is why we built XetHub, a platform that enables teams to treat data like code, using Git. Unlike Git LFS, we don’t just store the files. We use content-defined chunking and Merkle Trees to dedupe against everything in history. This allows small changes in large files to be stored compactly. Read more here: https://xethub.com/assets/docs/how-xet-deduplication-works Today, XetHub works for 1 TB repositories, and we plan to scale to 100 TB in the next year. Our implementation is in Rust (client & cache + storage) and our web application is written in Go. XetHub includes a GitHub-like web interface that provides automatic CSV summaries and allows custom visualizations using Vega. Even at 1 TB, we know downloading an entire repository is painful, so we built git-xet mount - which, in seconds, provides a user-mode filesystem view over the repo. XetHub is available today (Linux & Mac today, Windows coming soon) and we would love your feedback! Read more here: - https://xetdata.com/blog/2022/10/15/why-xetdata - https://xetdata.com/blog/2022/12/13/introducing-xethub

Visit product View on Hacker NewsDec 13, 2022

Share card

Actual performance

279points

144comments

Made the leaderboard

Launch Intel predictions

Analyze your own launch →

Product HuntOn track for Day 1 leaderboard · Strong signals: mac, slack, user · Missing: agents, macos, agent

92%92% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

Hacker NewsStrong engagement from HN community · Strong signals: filesystem, ide, io · Missing: https docs, excited, just released

79%79% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

Indie HackersFits the IH revenue-focused audience · Missing: supports, reddit linkedin, podcasting

72%72% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

AppSumoStrong fit for a featured deal · Strong signals: platform, interface, soon · Missing: plus, intuitive, reviews

56%56% predicted probability of success on AppSumo, based on ML models trained on real launch data.

TrustMRRLess likely to generate early MRR · Missing: mobile apps, ios, personal

42%42% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

Acquire.comPre-revenue stage for this audience · Missing: arr, mrr, revenue

17%17% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Missing: web3, chat, crypto

0%0% predicted probability of success on BetaList, based on ML models trained on real launch data.

Correct prediction on native model

Similar products

Git-recon, reconcile your local Git repos55%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Git-recon, reconcile your local Git repos

Hacker News3

Gitgrep28%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Regex search all your git repos in milliseconds.

Indie Hackers2saas

Gita – Manage many Git repos with sanity53%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Gita – Manage many Git repos with sanity

Hacker News3

libvcs – pythonically control git, svn and mercurial repos58%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

libvcs – pythonically control git, svn and mercurial repos

Hacker News18

Hyades, a newsletter for your starred Git repos45%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Hyades, a newsletter for your starred Git repos

Hacker News1

Fuchur - Get the status of all your Git repos at once.57%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Fuchur - Get the status of all your Git repos at once.

Hacker News1

Gitorials – Tutorials as Git Repos64%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Gitorials – Tutorials as Git Repos

Hacker News147

EasyGit – Git repos on iCloud59%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

EasyGit – Git repos on iCloud

Hacker News9

Kite – Copy Files from Git Repos (No, That's It)52%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Kite – Copy Files from Git Repos (No, That's It)

Hacker News1

Visualize Git repos using multidimensional scaling62%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Visualize Git repos using multidimensional scaling

Hacker News2

Browse all products Analyze your launch