Open-source observability for LLM apps

Hacker News

Hi HN, Hugh and Vince here. LLMonitor helps you record, trace & search your LLM queries and chatbot conversations. You can also capture user feedback on your frontend and correlate it with backend LLM queries then use that to fine-tune your own models. The project started has an internal tool in our previous (failed) AI startup. We’re aware the LLM observability space is very crowded. Apart from being open-source, we differentiate with: - Model-agnostic and minimal lock-in (no MITM of requests). - High focus on DX and dashboard clarity. - Support for complex scenarios: e.g. a chatbot that invokes multiple agents & tools before giving an answer. - Targeting early-stage startups + indie developers. Two of our big focuses in the near future are: - Tests (evaluations) written in either code or English (with GPT-4) to know when queries go wrong. - OpenTelemetry support to expand traces to the rest of the stack. (Both of those have turned out to be much more complicated than anticipated.) One decision we made is to stick with Postgres for everything instead of going OLAP with something like Clickhouse. As we are writing & updating many rows concurrently for the traces, Postgres seems to be the best tool for that. It has been sufficient for our analytics and search needs at the moment. We hope that also makes it easier for people to self-host and contribute. The entire codebase is licensed under Apache 2.0 and we’re fully bootstrapped. Being open-source has already begun to foster a community, e.g. with people building integrations for tooling we wouldn't have been able to support otherwise. To be honest we're still quite embarrassed with the state of the product, but I guess we’d be doing something wrong if that wasn’t the case. We’d love to hear your feedback - thank you! Landing page: https://llmonitor.com Docs: https://llmonitor.com/docs Repo: https://github.com/llmonitor/llmonitor

Visit product View on Hacker NewsOct 28, 2023

Share card

Actual performance

10points

1comments

Made the leaderboard

Launch Intel predictions

Analyze your own launch →

Product HuntOn track for Day 1 leaderboard · Strong signals: agents, agent, model · Missing: mac, macos, cursor

96%96% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

Indie HackersFits the IH revenue-focused audience · Strong signals: started, ios · Missing: supports, reddit linkedin, podcasting

92%92% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

Hacker NewsStrong engagement from HN community · Strong signals: lua, clickhouse, io · Missing: https docs, excited, just released

68%68% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

TrustMRRLess likely to generate early MRR · Strong signals: ios, apps · Missing: mobile apps, personal, entrepreneurs

47%47% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

AppSumoMay struggle as an AppSumo deal · Strong signals: host · Missing: plus, platform, intuitive

29%29% predicted probability of success on AppSumo, based on ML models trained on real launch data.

Acquire.comPre-revenue stage for this audience · Strong signals: arr, bootstrapped · Missing: mrr, revenue, profit

14%14% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Strong signals: chat · Missing: web3, crypto, cryptocurrency

0%0% predicted probability of success on BetaList, based on ML models trained on real launch data.

Correct prediction on native model

Similar products

OpenLIT – Open-Source LLM Observability with OpenTelemetry69%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

OpenLIT – Open-Source LLM Observability with OpenTelemetry

Hacker News62

OpenLIT – Open-Source LLM Observability with OpenTelemetry

Hacker News1

Langfuse – Open-source observability and analytics for LLM apps76%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Langfuse – Open-source observability and analytics for LLM apps

Hacker News143

Open-Source LLM Observability and Export to Grafana, Datadog etc.77%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Open-Source LLM Observability and Export to Grafana, Datadog etc.

Hacker News3

OneUptime - An Open Source Observability Platform82%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

OneUptime - An Open Source Observability Platform

Hacker News2

OneUptime – open-source observability platform78%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

OneUptime – open-source observability platform

Hacker News3

Monosi60%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Open Source Data Observability

Indie Hackerscommitment-full-time

Autometrics – open-source observability stack84%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Autometrics – open-source observability stack

Hacker News2

LLM Observability Platform65%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

LLM Observability Platform

Hacker News1

UpTrain – Open-source ML observability and refinement tool75%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

UpTrain – Open-source ML observability and refinement tool

Hacker News88

Browse all products Analyze your launch