PyLLMs: – Connect and compare top AI models in Python

Hacker News

Hi HN, We needed a simple way to connect to the top AI models to experiment, prototype and evaluate them. Main features: - Connect to top LLMs in few lines of code (currenly OpenAI, Anthropic and AI21 are supported) - Response meta includes tokens processed, cost and latency standardized across the models - Multi-model support: Get completitions from different models at the same time - LLM benchmark: Eevaluate models on quality, speed and cost The benchmark uses predefine questions to test AI reasoning abilities across a range of "hard" queries. The outputs are then automatically evaulauted using a powerful model (gpt-4 recommended): https://github.com/kagisearch/pyllms/blob/990855968b4bc26ab6... This helped uncover a hidden gem among models: 'claude-instant-v1' which is 4x faster, 2x cheaper and similar quality to 'crowd favorite' gpt-3.5-turbo.

Visit product View on Hacker NewsApr 13, 2023

Share card

Actual performance

6points

1comments

Did not reach leaderboard

Launch Intel predictions

Analyze your own launch →

Product HuntOn track for Day 1 leaderboard · Strong signals: claude, model, models · Missing: mac, agents, macos

89%89% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

Indie HackersFits the IH revenue-focused audience · Missing: supports, reddit linkedin, podcasting

73%73% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

AppSumoStrong fit for a featured deal · Missing: plus, platform, intuitive

56%56% predicted probability of success on AppSumo, based on ML models trained on real launch data.

TrustMRRLess likely to generate early MRR · Strong signals: way · Missing: mobile apps, ios, personal

50%50% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

Hacker NewsMay not resonate with HN audience · Strong signals: lua, io · Missing: https docs, excited, just released

39%39% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

Acquire.comPre-revenue stage for this audience · Missing: arr, mrr, revenue

16%16% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Missing: web3, chat, crypto

2%2% predicted probability of success on BetaList, based on ML models trained on real launch data.

Correct prediction on native model

Similar products

contentable.ai35%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Compare AI models before you adopt AI

Indie Hackers

Test AI Models72%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Compare AI models side-by-side on same prompt

Indie Hackers1$9/moai

Replicover – Find the hottest AI models on Replicate31%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Replicover – Find the hottest AI models on Replicate

Hacker News1

WisGate62%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

AI Models PAI

Indie Hackerscommitment-full-time

Tinx.ai – Prebuilt AI Models for You33%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Tinx.ai – Prebuilt AI Models for You

Hacker News7

Polibench – compare political bias across AI models44%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Polibench – compare political bias across AI models

Hacker News3

intura49%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Compare, test, and optimize AI models

Product Hunt+169Analytics

CometAPI81%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

All AI Models in One API

Product Hunt+6

ModelAtlas – Find AI models that HuggingFace search can't48%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

ModelAtlas – Find AI models that HuggingFace search can't

Hacker News1

Aisir – AI models deliberate and critique each other like a council34%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Aisir – AI models deliberate and critique each other like a council

Hacker News3

Browse all products Analyze your launch