Tokiwi – An online tokenizer for any Hugging Face model

Hacker News

Hey HN, Like (perhaps?) many of you working with different open-weight LLMs, I often find myself wanting a quick way to check token counts for specific Hugging Face models without having to write a Python script each time. I often find myself googling something like "online Qwen tokenizer" and coming up empty-handed. So, I built Tokiwi ( https://tokiwi.dev ), the online tokenizer I've been looking for. What it does: - You paste your text. - You enter any model repository ID from Hugging Face (like `google/gemma-3-27b-it`, `deepseek-ai/DeepSeek-V3-0324`, etc.). - It fetches the tokenizer from HF and shows you the token count and the list of tokens. - It also supports gated models if you provide your HF token (handled securely via the backend, not stored). It's pretty straightforward right now. Built with Next.js on Vercel for the frontend and a FastAPI backend running on Modal ( https://modal.com/ ). Under the hood I'm using the `transformers` library. I'm thinking about adding support for proprietary models (Claude, Gemini) or even a simple API if there's interest. I'm also thinking about offering prompt compression via LLMLingua ( https://github.com/microsoft/LLMLingua ). Would love to get your feedback. Is this useful for anyone else? What's missing? Thanks for your time!

Visit product View on Hacker NewsMar 29, 2025

Share card

Actual performance

2points

Did not reach leaderboard

Launch Intel predictions

Analyze your own launch →

Indie HackersFits the IH revenue-focused audience · Strong signals: supports, gemini · Missing: reddit linkedin, podcasting, created

85%85% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

Product HuntOn track for Day 1 leaderboard · Strong signals: claude, model, google · Missing: mac, agents, macos

85%85% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

TrustMRRLess likely to generate early MRR · Strong signals: google, way · Missing: mobile apps, ios, personal

47%47% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

Hacker NewsMay not resonate with HN audience · Strong signals: ide, io · Missing: https docs, excited, just released

35%35% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

AppSumoMay struggle as an AppSumo deal · Missing: plus, platform, intuitive

30%30% predicted probability of success on AppSumo, based on ML models trained on real launch data.

Acquire.comPre-revenue stage for this audience · Missing: arr, mrr, revenue

16%16% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Missing: web3, chat, crypto

0%0% predicted probability of success on BetaList, based on ML models trained on real launch data.

Correct prediction on native model

Similar products

Tie Dye Face Masks and Clothing42%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Tie Dye Face Masks and Clothing

Hacker News1

GODFACE – see the face of god.42%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

GODFACE – see the face of god.

Hacker News1

Abrielle Ageless Face Cream37%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Abrielle Ageless Face Cream

Indie Hackerscommitment-full-time

Skuuddle – Arrange face-to-face interactions with online acquaintances47%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Skuuddle – Arrange face-to-face interactions with online acquaintances

Hacker News32

Online Face Generator41%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Online Face Generator

Hacker News3

Face GPT45%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

face ai

Indie Hackers1ai

Face Search51%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Find ANYONE with just a FACE

Indie Hackerscommitment-full-time

I turned my face rec system into a video codec61%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

I turned my face rec system into a video codec

Hacker News518

DCGAN Face Generator40%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

DCGAN Face Generator

Hacker News5

Face Generator40%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Face Generator

Hacker News256

Browse all products Analyze your launch