Mandoline – Custom LLM Evaluations for Real-World Use Cases

Hacker News

Share card

Mandoline – Custom LLM Evaluations for Real-World Use Cases

Hi HN! We're a small team of AI engineers who've spent the last few years building AI applications. Through this, we've experienced firsthand many of the challenges that come with evaluating and improving AI systems in real-world contexts. Standard LLM evaluations (and evaluation methods) often use simplified scenarios that don't reflect the complexity LLMs encounter in actual use. This leads to a disconnect between reported performance and real-world usefulness. We built Mandoline to bridge this gap, helping developers evaluate and improve LLM applications in ways that matter to end-users. Our approach allows you to design custom evaluation criteria that align with your specific product requirements. For a quick overview of how it works, check out our Python and / or Node SDK READMEs: - Python: https://github.com/mandoline-ai/mandoline-python/blob/main/R... - Node / Typescript: https://github.com/mandoline-ai/mandoline-node/blob/main/REA... Hopefully this design is flexible yet scalable, and helps you do things like: track LLM progress over time, make informed AI system design decisions, choose the best model for your use case, prompt engineer more systematically, and so on. Under the hood, Mandoline is a hybrid system using a combination of our own models and top general-purpose LLM APIs. We used Mandoline to evaluate and improve itself, which helped us make better decisions about system design. In the future, we’ll be adding visualization tools to more easily analyze trends, and expanding our in-house models capabilities to reduce reliance on (and hopefully outperform) external models. Check out our website ( https://mandoline.ai/ ) and documentation ( https://mandoline.ai/docs ) to learn more. We’d love to hear about your experiences with evaluating AI systems for production use. What have you found most challenging in evaluating AI systems? What behaviors are hard to quantify? How could Mandoline fit into your workflow? You can reach us here in the comments or send us an email (hi@mandoline.ai). We appreciate you taking the time to learn a bit about Mandoline.