Continuous-eval – Granular evaluation of GenAI pipelines

Hacker News

Hi HN - we are the creators of “continuous-eval”, an open-source tool to test and evaluate generative AI apps. "Continuous-eval" came from our efforts to measure, validate and improve the reliability of a finance AI copilot we were developing for banks. End-to-end evaluation was not enough for us. We wanted to have granular evaluations that help pinpoint the bottlenecks and identify what / how to improve. We’ve since developed more metrics and made the framework more flexible so it can evaluate components like agent tool use, code change, retrieval steps, etc. Let us know what you think of our approach to GenAI App evaluation.

Visit product View on Hacker NewsFeb 25, 2024

Share card

Actual performance

10points

2comments

Made the leaderboard

Launch Intel predictions

Analyze your own launch →

Product HuntOn track for Day 1 leaderboard · Strong signals: agent, apps, code · Missing: mac, agents, macos

90%90% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

Indie HackersFits the IH revenue-focused audience · Missing: supports, reddit linkedin, podcasting

60%60% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

Hacker NewsStrong engagement from HN community · Strong signals: lua, ide, pipe · Missing: https docs, excited, just released

59%59% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

TrustMRRFits verified-revenue profile · Strong signals: apps · Missing: mobile apps, ios, personal

59%59% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

AppSumoMay struggle as an AppSumo deal · Missing: plus, platform, intuitive

19%19% predicted probability of success on AppSumo, based on ML models trained on real launch data.

Acquire.comPre-revenue stage for this audience · Missing: arr, mrr, revenue

14%14% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Missing: web3, chat, crypto

2%2% predicted probability of success on BetaList, based on ML models trained on real launch data.

Correct prediction on native model

Similar products

Buildbox Continuous Delivery Pipelines64%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Buildbox Continuous Delivery Pipelines

Hacker News1

AutoArena74%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Automated GenAI evaluation that works

Product Hunt+110Open Source

Continuous Fuzzing for Go75%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Continuous Fuzzing for Go

Hacker News1

Bencher – Continuous Benchmarking71%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Bencher – Continuous Benchmarking

Hacker News3

Rues an Expression Evaluation Sidecar56%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Rues an Expression Evaluation Sidecar

Hacker News1

Pipelines50%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Pipelines

Hacker News1

Pipelines50%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Pipelines

Hacker News5

GenAi Yule Lights50%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

GenAi Yule Lights

Hacker News1

Shadergarden: Create reloadable graphical pipelines with Lisp and GLSL60%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Shadergarden: Create reloadable graphical pipelines with Lisp and GLSL

Hacker News47

Create continuous development pipelines for static site generators62%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Create continuous development pipelines for static site generators

Hacker News4

Browse all products Analyze your launch