Local LLM AIME benchmarking tool

Hacker News

I made this simple tool to compare local LLMs. Any provider that supports OpenAI-like APIs can be used (LMStudio, Llama.cpp, Ollama) but you can also use Openrouter/OpenAI if you change the base URL accordingly. In my opinion it is not particularly useful for comparing different models from different companies since some models are optimized heavily on math or even trained on AIME problems. However it is really useful for testing different quantizations of the same model or the same quantization from different providers. Let me know what you think about it! Also check the README to see some examples of the results you will get from it.

Visit product View on Hacker NewsMay 31, 2025

Share card

Actual performance

1points

Did not reach leaderboard

Launch Intel predictions

Analyze your own launch →

Product HuntOn track for Day 1 leaderboard · Strong signals: model, models, openai · Missing: mac, agents, macos

79%79% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

Hacker NewsStrong engagement from HN community · Strong signals: llama, ide, io · Missing: https docs, excited, just released

58%58% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

Indie HackersFits the IH revenue-focused audience · Strong signals: supports · Missing: reddit linkedin, podcasting, created

53%53% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

TrustMRRFits verified-revenue profile · Missing: mobile apps, ios, personal

53%53% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

AppSumoMay struggle as an AppSumo deal · Missing: plus, platform, intuitive

31%31% predicted probability of success on AppSumo, based on ML models trained on real launch data.

Acquire.comPre-revenue stage for this audience · Missing: arr, mrr, revenue

13%13% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Missing: web3, chat, crypto

1%1% predicted probability of success on BetaList, based on ML models trained on real launch data.

Incorrect prediction on native model

Similar products

Redis-LLM – Redis module integrates LLM with Redis45%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Redis-LLM – Redis module integrates LLM with Redis

Hacker News2

LitLLM the Spiciest LLM Wrapper34%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

LitLLM the Spiciest LLM Wrapper

Hacker News1

LLM Reasonsers46%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

LLM Reasonsers

Hacker News2

Resilient LLM23%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Resilient LLM

Hacker News1

Hegelion – Force your LLM to argue with itself before answering58%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Hegelion – Force your LLM to argue with itself before answering

Hacker News1

I Stopped Hoping My LLM Would Cooperate49%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

I Stopped Hoping My LLM Would Cooperate

Hacker News3

Module for LLM Homeostasis (PoC)22%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Module for LLM Homeostasis (PoC)

Hacker News1

Doom Compiled into an LLM50%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Doom Compiled into an LLM

Hacker News2

The Smallest LLM46%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

The Smallest LLM

Hacker News3

LLM Hotkey39%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

TrustMRROther

Browse all products Analyze your launch