Real-time voice chat with AI, no transcription

Hacker News

Share card

Real-time voice chat with AI, no transcription

Hi HN -- voice chat with AI is very popular these days, especially with YC startups ( https://twitter.com/k7agar/status/1769078697661804795 ). The current approaches all do a cascaded approach, with audio -> transcription -> language model -> text synthesis. This approach is easy to get started with, but requires lots of complexity and has a few glaring limitations. Most notably, transcription is slow, is lossy and any error propagates to the rest of the system, cannot capture emotional affect, is often not robust to code-switching/accents, and more. Instead, what if we fed audio directly to the LLM - LLM's are really smart, can they figure it out? This approach is faster (we skip transcription decoding) and less lossy/more robust because the big language model should be smarter than a smaller transcription decoder. I've trained a model in just that fashion. For more architectural information and some training details, see this first post: https://tincans.ai/slm . For details about this model and some ideas for how to prompt it, see this post: https://tincans.ai/slm3 . We trained this on a very limited budget but the model is able to do some things that even GPT-4, Gemini, and Claude cannot, eg reasoning about long-context audio directly, without transcription. We also believe that this is the first model in the world to conduct adversarial attacks and apply preference modeling in the speech domain. The demo is unoptimized (unquantized bf16 weights, default Huggingface inference, serverless speed bumps) but achieves 120ms time to first token with audio. You can basically think of it as Mistral 7B, so it'll be very fast and can also run basically anywhere. I am especially optimistic about embedded usage -- not needing the transcription step means that the resulting model is smaller and cheaper to use on the edge. Would love to hear your thoughts and how you would use it! Weights are Apache-2 and on Hugging Face: https://huggingface.co/collections/tincans-ai/gazelle-v02-65...