Minbpe-hs – byte-level byte pair encoding (BPE) in Haskell

Hacker News

minbpe-hs is a port of Andrej Karpathy's concise byte-level byte pair encoding (BPE) implementation, minbpe, to Haskell. During training, BPE compresses a sequence by repeatedly finding the most frequent pair of elements in the input and merging them into a new token whilst maintaining a record of the merges and the tokens. Encoding is performed by recursively merging pairs with the smallest merge index until there remain no pairs to be merged, and encoded representations can be decoded backed into the original through vocabulary look-ups. Given its recursive nature, BPE lends itself naturally to functional programming, and the objective of this repository is to offer an alternative functional formulation of it that is orthogonal to typical implementations in imperative languages such as Python or Rust to aid learners in gaining a firmer grasp on BPE by viewing it through a different lens. Similar to minbpe, BPE.Basic and BPE.Regex both supply the same fundamental functionalities of training, encoding, and decoding, but the former treats a text corpus as one continuous sequence whereas the latter, in addition to supporting special tokens, divides it into smaller segments using a regex pattern and considers merges only within these segments. BPE.Base contains core operations employed by the two tokenizers and also provides storing & loading utilities. There is no counterpart to minbpe's GPT4Tokenizer as tiktoken does not expose a Haskell frontend. Questions and feedback are welcome in the comments sections.

Visit product View on Hacker NewsMay 24, 2024

Share card

Actual performance

2points

Did not reach leaderboard

Launch Intel predictions

Analyze your own launch →

Indie HackersFits the IH revenue-focused audience · Missing: supports, reddit linkedin, podcasting

81%81% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

Hacker NewsStrong engagement from HN community · Strong signals: ide, io · Missing: https docs, excited, just released

56%56% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

Product HuntOn track for Day 1 leaderboard · Strong signals: new, presentations, using · Missing: mac, agents, macos

53%53% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

TrustMRRFits verified-revenue profile · Missing: mobile apps, ios, personal

53%53% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

AppSumoMay struggle as an AppSumo deal · Missing: plus, platform, intuitive

29%29% predicted probability of success on AppSumo, based on ML models trained on real launch data.

Acquire.comPre-revenue stage for this audience · Strong signals: training · Missing: arr, mrr, revenue

16%16% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Missing: web3, chat, crypto

5%5% predicted probability of success on BetaList, based on ML models trained on real launch data.

Incorrect prediction on native model

Similar products

Take Scheme to the next level with Scheme 2-D63%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Take Scheme to the next level with Scheme 2-D

Hacker News2

C-

C-Level E-Bikes43%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

C-Level E-Bikes

Hacker News1

C-

C-Level positions to fill before you incorporate62%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

C-Level positions to fill before you incorporate

Hacker News4

Level sets in any dimension38%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Level sets in any dimension

Hacker News1

Level Devil58%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Level Devil - Not A Troll Game

Indie Hackers

AppReviewer50%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Take your App to the Next Level!✔️

Indie Hackers3analytics

A simple level logger for golang38%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

A simple level logger for golang

Hacker News1

VidBump33%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Bump Your Videos to the Next Level with AI

Indie Hackers

TempStore.dev76%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Next-level Templating

Indie Hackers1ai

Waveform – Take Your Programmer Art to the Next Level57%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Waveform – Take Your Programmer Art to the Next Level

Hacker News11

Browse all products Analyze your launch