Ag

Agentic Evaluators for Agentic Workflows (Starting with RAG)

Hacker News

Agentic Evaluators for Agentic Workflows (Starting with RAG)

Hey all! Thought this group might find this interesting - new approach to evaluating RAG pipelines using 'agents as a judge'. We got excited by the findings in this paper ( https://arxiv.org/abs/2410.10934 ), about agents producing evaluations closer to human-evaluators, especially for multi-step workflows. Our first use case was RAG pipelines, specifically evaluating if your agent MISSED pulling any important chunks from the source document. While many RAG evaluators determine if your model USED its chunk in the output, there's no visibility on if your model grabbed all the right chunks in the first place. We thought we'd test the 'agent as judge', with a new metric called 'potential sources missed', to help evaluate if your agents are missing any important chunks from the source of truth. Curious what you all think!

Visit product View on Hacker NewsNov 20, 2024

Actual performance

4points

2comments

Did not reach leaderboard

Launch Intel predictions

Analyze your own launch →

Product HuntOn track for Day 1 leaderboard · Strong signals: agents, agent, model · Missing: mac, macos, cursor

98%98% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

TrustMRRLess likely to generate early MRR · Missing: mobile apps, ios, personal

36%36% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

Hacker NewsMay not resonate with HN audience · Strong signals: excited, lua, pipe · Missing: https docs, just released, exist

35%35% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

Indie HackersIH features products with proven revenue · Missing: supports, reddit linkedin, podcasting

30%30% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

AppSumoMay struggle as an AppSumo deal · Missing: plus, platform, intuitive

26%26% predicted probability of success on AppSumo, based on ML models trained on real launch data.

Acquire.comPre-revenue stage for this audience · Missing: arr, mrr, revenue

13%13% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Missing: web3, chat, crypto

1%1% predicted probability of success on BetaList, based on ML models trained on real launch data.

Correct prediction on native model

Similar products

PaperQA2, Agentic RAG for Science78%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

PaperQA2, Agentic RAG for Science

Vectara-Agentic58%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Vectara-Agentic

Vetara Agentic35%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Vetara Agentic

Overcut88%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Automate your SDLC with Agentic workflows

Product Hunt+138Developer Tools

Dealora – What we learned starting up43%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Dealora – What we learned starting up

RAG Labz22%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Agentic Agents that boost business

Indie Hackerscommitment-full-time

Vectorless RAG48%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Vectorless RAG

PageIndex – Vectorless RAG66%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

PageIndex – Vectorless RAG

Preprocess65%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Preprocess maximises RAG performances

Product Hunt+106API

Free Agentic RAG (no account needed)36%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Free Agentic RAG (no account needed)

Browse all products Analyze your launch