Wordchipper – Rust BPE tokenizer, 9x faster than tiktoken

Hacker News

Hey HN, We're ZSpaceLabs, Burn framework contributors working to make Rust a first-class AI/ML stack. We just released wordchipper, a Rust-native BPE tokenizer for the OpenAI GPT-2 tokenizer family (r50k, cl100k, o200k). On a 64-core machine with the o200k vocab (GPT-4o / GPT-5 tokenizer), we measured 2.4 GiB/s, about 9.2× faster than tiktoken-rs. Through the Python bindings it is typically 2–4× faster than tiktoken, depending on thread count. The main design goal was to make the internals easy to swap. The tokenizer is split into two parts: pre-tokenization (lexer) and BPE span encoding. Each part can be replaced independently, which makes it easy to experiment with different combinations of lexer backends and span encoding algorithms. Right now there are three lexer implementations. One uses fancy-regex and is fully compatible with tiktoken. Another uses regex-automata with a runtime DFA and is about 4–8× faster. The third uses logos with a compile-time DFA and is about 14–21× faster on cl100k and o200k. Write-up with more details: https://zspacelabs.ai/wordchipper/articles/substitutable/ GitHub: https://github.com/zspacelabs/wordchipper Happy to hear feedback, especially from people working on tokenization, large-scale inference pipelines, or Rust ML tooling.

Visit product View on Hacker NewsMar 24, 2026

Share card

Actual performance

2points

Did not reach leaderboard

Launch Intel predictions

Analyze your own launch →

Indie HackersFits the IH revenue-focused audience · Strong signals: compatible · Missing: supports, reddit linkedin, podcasting

75%75% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

Hacker NewsStrong engagement from HN community · Strong signals: just released, pipe, io · Missing: https docs, excited, exist

68%68% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

Product HuntOn track for Day 1 leaderboard · Strong signals: mac, openai, coding · Missing: agents, macos, agent

55%55% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

TrustMRRLess likely to generate early MRR · Missing: mobile apps, ios, personal

49%49% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

AppSumoMay struggle as an AppSumo deal · Missing: plus, platform, intuitive

42%42% predicted probability of success on AppSumo, based on ML models trained on real launch data.

Acquire.comPre-revenue stage for this audience · Missing: arr, mrr, revenue

16%16% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Missing: web3, chat, crypto

0%0% predicted probability of success on BetaList, based on ML models trained on real launch data.

Incorrect prediction on native model

Similar products

WisniaLang – Compile binaries 41x faster and 27x lighter than Rust58%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

WisniaLang – Compile binaries 41x faster and 27x lighter than Rust

Hacker News1

Splintr – Rust BPE tokenizer, 12x faster than tiktoken for batches50%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Splintr – Rust BPE tokenizer, 12x faster than tiktoken for batches

Hacker News1

A faster fuck (thefuck CLI rewritten in Rust)58%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

A faster fuck (thefuck CLI rewritten in Rust)

Hacker News2

StatsD in Rust51%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

StatsD in Rust

Hacker News1

TF-IDF in Rust51%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

TF-IDF in Rust

Hacker News1

Rust FFI in a Wireshark C dissector51%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Rust FFI in a Wireshark C dissector

Hacker News1

Rust-wasm-webpack50%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Rust-wasm-webpack

Hacker News6

Screencast: An Introduction to Rust57%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Screencast: An Introduction to Rust

Hacker News3

Handlebars for Rust51%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Handlebars for Rust

Hacker News2

Oxischeme – A Scheme implementation in Rust56%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Oxischeme – A Scheme implementation in Rust

Hacker News18

Browse all products Analyze your launch