An open-source AI Gateway with integrated guardrails

Hacker News

Hi HN, I've been developing Portkey Gateway, an open-source AI gateway that's now processing billions of tokens daily across 200+ LLMs. Today, we're launching a significant update: integrated Guardrails at the gateway level. Key technical features: 1. Guardrails as middleware: We've implemented a hooks architecture that allows guardrails to act as middleware in the request/response flow. This enables real-time LLM output evaluation and transformation. 2. Flexible orchestration: The gateway can now route requests based on guardrail verdicts. This allows for complex logic like fallbacks to different models or prompts based on output quality. 3. Plugin system: We've designed a modular plugin system that allows integration of various guardrail implementations (e.g., anthropic/constrained-llm, microsoft/guidance). 4. Stateless design: The guardrails implementation maintains the gateway's stateless nature, ensuring scalability and allowing for easy horizontal scaling. 5. Unified API: Despite the added complexity, we've maintained our unified API across different LLM providers, now extended to include guardrail configurations. Implementation details: * The guardrails are implemented as async functions in the request pipeline. * We use a combination of regex and LLM-based evaluation for output validation. * The system supports both pre-processing (input modification) and post-processing (output filtering/transformation) guardrails. Performance impact: * Latency increase is minimal (<50ms) for most deterministic guardrails. * We've implemented caching mechanisms to reduce repeated evaluations. * Since the gateway lives on the edge, it avoids longer roundtrips Challenges we're still tackling: * Balancing strict guardrails with maintaining model creativity * Standardizing evaluation metrics across different types of guardrails * Handling guardrail false positives/negatives effectively We believe this approach of integrating guardrails at the gateway level provides a powerful tool for managing LLM behavior in production environments. The code is open-source, and we welcome contributions and feedback. We're particularly interested in hearing about specific use cases or challenges you've faced in implementing reliable LLM systems. Detailed documentation: https://portkey.wiki/guardrails What are your thoughts on this approach? Are there specific guardrail implementations or orchestration patterns you'd like to see added?

Visit product View on Hacker NewsAug 14, 2024

Share card

Actual performance

21points

5comments

Made the leaderboard

Launch Intel predictions

Analyze your own launch →

Indie HackersFits the IH revenue-focused audience · Strong signals: supports · Missing: reddit linkedin, podcasting, created

96%96% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

Product HuntOn track for Day 1 leaderboard · Strong signals: model, models, code · Missing: mac, agents, macos

91%91% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

TrustMRRFits verified-revenue profile · Strong signals: way · Missing: mobile apps, ios, personal

52%52% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

Hacker NewsStrong engagement from HN community · Strong signals: lua, ide, pipe · Missing: https docs, excited, just released

51%51% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

AppSumoMay struggle as an AppSumo deal · Missing: plus, platform, intuitive

48%48% predicted probability of success on AppSumo, based on ML models trained on real launch data.

Acquire.comPre-revenue stage for this audience · Missing: arr, mrr, revenue

10%10% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Missing: web3, chat, crypto

0%0% predicted probability of success on BetaList, based on ML models trained on real launch data.

Correct prediction on native model

Similar products

GoModel – an open-source AI gateway in Go51%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

GoModel – an open-source AI gateway in Go

Hacker News217

Lightport – open-source AI gateway61%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Lightport – open-source AI gateway

Hacker News1

Kong's open-source AI Gateway to Democratize multi-LLM consumption52%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Kong's open-source AI Gateway to Democratize multi-LLM consumption

Hacker News3

APIPark73%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Your open source AI gateway & API developer portal

Product Hunt+715API

Bakery By Bagel68%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Monetizable open source AI

Product Hunt+190Open Source

Experiential Labs87%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Open source AI gateway turning traffic into a better model

Product Hunt+136Analytics

I built an open source AI gateway that proxy calls to multiple LLMs59%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

I built an open source AI gateway that proxy calls to multiple LLMs

Hacker News2

Extrapolate – Open-Source AI Aging App56%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Extrapolate – Open-Source AI Aging App

Hacker News5

Helicone.ai75%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

The open-source AI gateway for AI-native startups

Product Hunt+207Open Source

I made an open-source AI Headshot Generator57%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

I made an open-source AI Headshot Generator

Hacker News3

Browse all products Analyze your launch