Retrieval Augmented Generation for LLMs from APIs with Function Calling

Hacker News

Share card

Retrieval Augmented Generation for LLMs from APIs with Function Calling

Dear HN, we'd love your feedback on a different approach to Retrieval Augmented Generation (RAG) for LLMs that uses function calling to retrieve the most relevant data from APIs. It aims to solve the problem of connecting LLMs with your data so that the LLM can pull in the context it needs to provide high quality answers to user questions. We are calling it "apiRAG" for now (yes, very creative ;-) You can run the examples in the GitHub repository or watch this video to see how it works: https://www.youtube.com/watch?v=Mx-slh6h42c We developed apiRAG because we found that existing RAG approaches (text search, vector based, FLARE, etc) don't work well with structured and semi-structured data. apiRAG can efficiently augment from structured and semi-structured data by translating user questions to relevant API requests and then presenting the result data to the user. It supports textual presentation (with Markdown for tables and such) and visual presentation by charting data when appropriate. Check out the IoT chatbot that can answer questions about collected sensor data or the credit card example that gives users a customized spending analysis (shown in the video). Unlike RAG approaches that use text or vector search, apiRAG uses the LLM to determine what information should be retrieved and augmented into the context via function calling. That means, it often does a better job at identifying relevant information by pushing down filters, like when you ask "Who appears in the third episode of season 2" in our "Rick and Morty" example. apiRAG also works great for customer sensitive data, like in our Nutshop and banking examples where customers can ask questions about their past purchases/transactions and the LLM can answers those effectively by pulling in the relevant data from the API while being "sandboxed" to only access a particular authenticated customer's data. Most importantly, apiRAG is really simple. Just a lightweight library configured with a single JSON file that specifies the API endpoints (you can reuse existing APIs). All the heavy lifting is done by the LLM. Let us know what you think of apiRAG and if you'd find it useful for building generative AI applications. How could we make it better?