I made a USMLE prep AI study buddy (VQA-RAG) for my girlfriend.

Hacker News

Share card

I made a USMLE prep AI study buddy (VQA-RAG) for my girlfriend.

Step 1 Buddy is a RAG (Retrieval-Augmented Generation) pipeline that processes USMLE Step 1 videos and allows medical students to ask questions about specific topics. It provides answers with references to the exact video timestamps used as sources. Try it out: https://step1buddy4panda.streamlit.app/ DISCLAIMER: You would need an OpenAI API key for answer generation currently. Depending on the feedback & in the future, I plan to replace it with a better suited/finetuned model. Background: my girlfriend is currently studying/ preparing for step 1 exam to be taken soon and I watch her study every day using various study materials and she always has many doubts. Now obviously I cant answer the doubts being a AI grad student but I wanted to help her. I gathered from her that getting the questions answered from the videos as reference is of great help since the videos generally have slides and she could refer back to them quickly which would help her study better. This project is a small effort towards helping her past this arduous journey in medicine. How it's different: Unlike general Q&A systems, Step 1 Buddy is specifically tailored for USMLE Step 1 content. It not only provides answers but also points users to the exact moments in the video lectures where the information is discussed, allowing for quick verification and deeper understanding. Technical details: 1. Video processing: - Audio extraction and transcription using OpenAI Whisper - JSON dataset creation with text, timestamp, and video path 2. RAG pipeline and answer generation: - BERT embeddings for transcribed sentences - Similarity search for context retrieval - Answer generation using GPT-4-mini, constrained to the given context 3. Video timestamp retrieval: - Fetching and displaying relevant video segments using OpenCV I'd love to hear your thoughts, suggestions, and any questions you might have about the project. Medical students, in particular, I'd be grateful for your feedback on how this tool could be improved to better serve your study needs.