Batteries Included AI Deployment

Hacker News

Share card

Batteries Included AI Deployment

Hi HN, I've spent the better part of the last year deploying AI inference systems for both personal and work projects. I've noticed a ton of "gotchas" and footguns throughout the process that make doing it right a pain, so I built "Batteries Included AI Deployment". Not a creative name but it does what it says on the tin. In short, it's a template for deploying AI inference APIs with FastAPI. In long, it uses Docker to encapsulate (almost) the entire development and deployment process. The repo includes: 1. A way to download and cache models straight from huggingface 2. A way to expose those cached models via a FastAPI server endpoint 3. A docker configuration that exposes a `debugpy` port so that you can debug your application within a container 4. A way to run tests 5. A way to debug tests (using `debugpy` as mentioned above) 6. A way to run pre-commits on staged files 7. A way to manually run pre-commits on all code in your repository 8. CI steps via GitHub Actions 9. Full Observability with a Grafana Dashboard 10. Metrics via Prometheus 11. Tracing via Tempo 12. Logs via Loki 13. GPU monitoring via DCGM 14. CD via GitHub actions and a `post-receive` hook on the server 15. Alerts that email you when something goes wrong in production I say "almost" because you still need a way to attach to the debugger port from outside the docker container and there's some one-time configurations that need to be set up manually, but not anything beyond that. I'd love to hear any feedback you might have :)