TokenDagger – A tokenizer faster than OpenAI's Tiktoken

Hacker News

TokenDagger is a drop-in replacement for OpenAI’s Tiktoken (the tokenizer behind Llama 3, Mistral, GPT-3.*, etc.). It’s written in C++ 17 with thin Python bindings, keeps the exact same BPE vocab/special-token rules, and focuses on raw speed. I’m teaching myself LLM internals by re-implementing the stack from first principles. Profiling TikToken’s Python/Rust implementation showed a lot of time was spent doing regex matching. Most of my perf gains come from a) using a faster jit-compiled regex engine; and b) simplifying the algorithm to forego regex matching special tokens at all. Benchmarking code is included. Notable results show: - 4x faster code sample tokenization on a single thread. - 2-3x higher throughput when tested on a 1GB natural language text file.

Visit product View on Hacker NewsJun 30, 2025

Share card

Actual performance

281points

73comments

Made the leaderboard

Launch Intel predictions

Analyze your own launch →

Product HuntOn track for Day 1 leaderboard · Strong signals: openai, single, using · Missing: mac, agents, macos

83%83% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

Hacker NewsStrong engagement from HN community · Strong signals: llama, io · Missing: https docs, excited, just released

68%68% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

Indie HackersFits the IH revenue-focused audience · Missing: supports, reddit linkedin, podcasting

65%65% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

TrustMRRFits verified-revenue profile · Missing: mobile apps, ios, personal

54%54% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

AppSumoMay struggle as an AppSumo deal · Missing: plus, platform, intuitive

37%37% predicted probability of success on AppSumo, based on ML models trained on real launch data.

Acquire.comPre-revenue stage for this audience · Missing: arr, mrr, revenue

19%19% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Missing: web3, chat, crypto

8%8% predicted probability of success on BetaList, based on ML models trained on real launch data.

Correct prediction on native model

Similar products

Cataloging the NYRB and LRB with OpenAI Embeddings46%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Cataloging the NYRB and LRB with OpenAI Embeddings

Hacker News3

OpenAI O1 for Free55%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

OpenAI O1 for Free

Hacker News4

OpenAI Apps Handbook55%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

OpenAI Apps Handbook

Hacker News10

Finetune OpenAI Embeddings In-Browser61%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Finetune OpenAI Embeddings In-Browser

Hacker News14

Browser viz of OpenAI's "spaghetti" Navier–Stokes vortex (illustration)59%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Browser viz of OpenAI's "spaghetti" Navier–Stokes vortex (illustration)

Hacker News5

Faster exceptions without the ceremony66%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Faster exceptions without the ceremony

Hacker News3

Faster HyperLogLog Implementation57%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Faster HyperLogLog Implementation

Hacker News1

Plenc- Faster Serialization in Go48%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Plenc- Faster Serialization in Go

Hacker News2

Faster Integer Parsing61%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Faster Integer Parsing

Hacker News2

Faster Than CGAL Delaunay in Julia66%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Faster Than CGAL Delaunay in Julia

Hacker News7

Browse all products Analyze your launch