Give your AI agent on-screen guides that show users where to click

Hacker News

Share card

Give your AI agent on-screen guides that show users where to click

Hey HN. I'm Christian, one of the founders of Frigade (YC W23). I've noticed that a lot of in-app AI agents struggle to actually understand the products they exist in. For instance, let's say a user asks an agent how to do something in a given SaaS product. In an ideal case, maybe that agent replies saying it has a tool to do the task and just automates that work entirely for the user. That's a great outcome. But often that's not the case. Maybe there is no tool call for that exact task, or maybe the user's question is best solved by a specific UI workflow or interface. In these cases, many agents tend to fall back on basic RAG on their help center, or sometimes even searching the internet for an understanding of their own product. This can be a very slow process and most of the time help center articles are outdated as products evolve faster than them today. Even worse, no one likes reading a long list of bullets and mapping that back to a UI. My tool (Assist API) solves this gap with a single tool call defined like this: const frigade_guide_tool = { description: 'Call this tool to answer product questions or guide the user through a task.', parameters: { query: { type: 'string', description: 'What the user is asking or wants to do', }, }, run: ({ query }) => frigade.assist({ query }), } Here's a demo I recorded on how to set it up with the Vercel AI SDK: https://www.youtube.com/watch?v=9WQ0UbLjC6I When called, the tool will do the following: 1) Gather context on what the user is seeing on screen, their permissions, feature flags, and more. Then one of the following: 2a) If solvable: Generate an on screen guide for how to fix a given problem 2b) If conceptual: Return text describing to the parent agent how to solve the problem 2c) Reject (i.e. unable to help) How does the tool know what to do? The tool learns a given application UI by using a browser-based agent. You provide a test account to your software (i.e. staging og preview), and a browser agent logs in and works its way through the entire product. It then builds its own map of how the application works which can the be queried about any product-related question or how to get from A to B in the UI. It also writes its own documentation from this map. The agent re-runs on a schedule or can be triggered through CI/CD. Docs and more details: https://frigade.com/assist-api