I Built a GitHub Action to Monitor LlamaIndex Performance

Hacker News

Hi all, I had been working with tools like LlamaIndex at work and realized that there were not many good options for monitoring RAG systems for performance. So I built a pretty simple Github Action + PyTest setup to measure the performance of LlamaIndex over time. The setup basically uses a library called Tonic Validate, which scores the quality of your RAG system’s answers (Disclaimer: My current company made Tonic Validate and I am an engineer on Tonic Validate). Using Tonic Validate, it scores the responses from LlamaIndex on a set of test data I created and then uploads it to Tonic Validate’s UI for visualization. If anyone is interested in it, you can find the full source code here [1]. I also wrote a guest blog post on LlamaIndex’s blog about it here [2] if anyone wants more details about how it works. 1. https://github.com/TonicAI/llama-validate-demo 2. https://blog.llamaindex.ai/tonic-validate-x-llamaindex-implementing-integration-tests-for-llamaindex-43db50b76ed9

Visit product View on Hacker NewsJan 26, 2024

Share card

Actual performance

2points

1comments

Did not reach leaderboard

Launch Intel predictions

Analyze your own launch →

Product HuntOn track for Day 1 leaderboard · Strong signals: visual, using, code · Missing: mac, agents, macos

70%70% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

AppSumoMay struggle as an AppSumo deal · Missing: plus, platform, intuitive

43%43% predicted probability of success on AppSumo, based on ML models trained on real launch data.

TrustMRRLess likely to generate early MRR · Strong signals: answers · Missing: mobile apps, ios, personal

38%38% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

Indie HackersIH features products with proven revenue · Strong signals: created · Missing: supports, reddit linkedin, podcasting

33%33% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

Hacker NewsMay not resonate with HN audience · Strong signals: llama, io · Missing: https docs, excited, just released

32%32% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

Acquire.comPre-revenue stage for this audience · Missing: arr, mrr, revenue

14%14% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Missing: web3, chat, crypto

0%0% predicted probability of success on BetaList, based on ML models trained on real launch data.

Correct prediction on native model

Similar products

AntiVirus Monitor – GitHub Action to combat false positives36%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

AntiVirus Monitor – GitHub Action to combat false positives

Hacker News12

Asciidoctor GitHub Action32%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Asciidoctor GitHub Action

Hacker News2

GitHub Action to find Log4j vulnerabilities38%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

GitHub Action to find Log4j vulnerabilities

Hacker News2

OpenHands' GitHub Action44%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

OpenHands' GitHub Action

Hacker News2

Healthyr - Rails app performance monitor38%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Healthyr - Rails app performance monitor

Hacker News3

There are 100s of similar Monitor Arms so I built this54%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

There are 100s of similar Monitor Arms so I built this

Hacker News41

There are 100s of similar Monitor Arms so I built this54%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

There are 100s of similar Monitor Arms so I built this

Hacker News1

There are 100s of similar Monitor Arms so I built this54%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

There are 100s of similar Monitor Arms so I built this

Hacker News32

Video Puppet GitHub Action28%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Video Puppet GitHub Action

Hacker News11

Raileo77%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Monitor your website downtime, ssl validity and performance

Indie Hackers5saas

Browse all products Analyze your launch