Our command line tool to transpile AI Inference from Python to C++

Hacker News

Share card

Our command line tool to transpile AI Inference from Python to C++

Hey all! We're a small (semi-stealth) team that's been working on a tool to rewrite AI inference code from Python to C++ (similar to llama.cpp, whisper.cpp, and so on). Today, we're launching `muna transpile`. It takes a Python function and generates a self-contained, header-only C++ library and a corresponding CMakeLists.txt file. It pulls in required libraries automatically (e.g. llama.cpp, onnxruntime, mlx, and so on). You can then use it to build and ship an application or library. Try it yourself on Kokoro TTS: ``` $ pip install muna $ muna transpile https://github.com/muna-ai/muna-predictors/blob/main/text-to... --trust-remote-code --install-deps $ cd cpp && cmake -B build && cmake --build build $ ./kokoro_tts --text "Hello Hacker News!" --voice af ``` The command above will transpile our Kokoro sample from Python to C++, compile it, and run the example (which uses the generated header-only lib). Taking a step back: on-device AI is becoming a hot topic, but we think the dominant thought of cloud vs. on-device is misguided. If you look deeper, you'll realize that the key is portability. ggerganov proved this by building llama.cpp, which developers have deployed on everything from Blackwell GPUs to raspberry pi. The first step is always creating a bare-metal, hardware-optimized C/C++ implementation. `muna transpile` automates this for anything you can fit into a Python function. Note that this is free and freely-usable: your Python source code goes in, it's still your source code when it comes out (just converted to C++). We're working on building more stuff on top of this (e.g. choosing where inference runs in one line of code; and agent skills that run AI models locally), so we're using this as an opportunity to expand support for different kinds of AI models / Python functions. Try it out and lmk what you think.