On-device, no-code LLMs with guardrails (for Apple Silicon)

Hacker News

Share card

On-device, no-code LLMs with guardrails (for Apple Silicon)

We've been working to make uncertainty quantification and interpretability first-class properties of LLMs. Reexpress one, a macOS app, is our first effort to make these properties widely available. Perhaps counter-intuitively, and contrary to common wisdom, LLMs can in fact be transformed to generate very reliable uncertainty estimates (i.e., "knowing what they do and don't know" by assigning a probability to the output). Getting there is a bit complicated, with vector matching/databases, prediction-time data dependencies, complicated inference, and multiple models flying all over the place. We've made it simple and efficient to use in practice with an on-device, no-code approach. Common document classification tasks can be handled with the on-device models (up to 3.2 billion parameters). Additionally, you can add these capabilities to another LLM (e.g., for QA or more complicated tasks) by connecting your existing model by simply uploading the output logits into the app. For example, if you're using an on-device Mistral AI model, or cloud-based genAI model, just upload the output logits into the app. Would be great to get feedback. Also, if you have another use case with a scale that doesn't fully fit into the on-device setting, happy to discuss and collaborate for your setting. And if anyone finds this interesting and wants to get involved more in building reliable AI, let us know! (Note that an Apple silicon Mac is required; ideally M1 Max or better with 64gb of RAM. You train the model yourself, which requires labeled data. The tutorial 1 video has a link to sentiment data in the JSON lines format; it's a good place to start: https://github.com/ReexpressAI/Example_Data/blob/main/tutori... )