Product Analytics for LLMs

Hacker News

Over the past two years, I’ve developed a toolkit for helping dozens of clients improve their LLM-powered products, which I'm now open-sourcing. First up: a library to bring product analytics to conversational AI. One of the biggest challenges I see clients face is understanding how their assistants are performing in production. Evals are great for catching regressions, but they can’t surface the blind spots in your AI’s behavior. This gets even more challenging for conversational AI products that don’t have a single “correct” answer. Different users cohorts want different experiences. That makes measurement tricky. Coming from a product analytics background, my default instinct is always: “instrument the product!” However, tracking generic events like user_sent_message doesn’t tell you much. What you really want are insights like: - How frequently do users request to speak with a human when interacting with a customer support agent? - Which user journeys trigger self-reflection during a session with an AI therapist? - What percentage of the time does an AI tutor's explanation leave the student confused? This new library enables these types of insights through the following workflow: - Analyzes your conversation transcripts - Auto-generates a rich event schema - Tags each message with relevant events and event properties - Sends the events to your analytics tool (currently supports Amplitude and PostHog) Any thoughts or feedback would be greatly appreciated!

Visit product View on Hacker NewsApr 18, 2025

Share card

Actual performance

3points

Did not reach leaderboard

Launch Intel predictions

Analyze your own launch →

Product HuntOn track for Day 1 leaderboard · Strong signals: agent, user, new · Missing: mac, agents, macos

97%97% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

Indie HackersFits the IH revenue-focused audience · Strong signals: supports · Missing: reddit linkedin, podcasting, created

81%81% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

TrustMRRFits verified-revenue profile · Strong signals: users, way · Missing: mobile apps, ios, personal

53%53% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

Hacker NewsMay not resonate with HN audience · Strong signals: io · Missing: https docs, excited, just released

23%23% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

AppSumoMay struggle as an AppSumo deal · Strong signals: users · Missing: plus, platform, intuitive

23%23% predicted probability of success on AppSumo, based on ML models trained on real launch data.

Acquire.comPre-revenue stage for this audience · Missing: arr, mrr, revenue

14%14% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Missing: web3, chat, crypto

0%0% predicted probability of success on BetaList, based on ML models trained on real launch data.

Correct prediction on native model

Similar products

WallyMetrics79%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Walmart product analytics

Product Hunt+12

GPTCache – Redis for LLMs69%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

GPTCache – Redis for LLMs

Hacker News7

prompttest – pytest for LLMs34%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

prompttest – pytest for LLMs

Hacker News2

StatsKit – Product Analytics and A/B Testing43%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

StatsKit – Product Analytics and A/B Testing

Hacker News2

Adblock Analytics46%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Adblock Analytics

Hacker News42

Birthdays Analytics46%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Birthdays Analytics

Hacker News2

Election Analytics UIUC27%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Election Analytics UIUC

Hacker News2

StatMilk - NFL/NCAA Analytics46%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

StatMilk - NFL/NCAA Analytics

Hacker News1

Composable Analytics52%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Composable Analytics

Hacker News2

Nudge Analytics46%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Nudge Analytics

Hacker News1

Browse all products Analyze your launch