Leaping – Debug Python tests instantly with an LLM debugger

Hacker News

Share card

Leaping – Debug Python tests instantly with an LLM debugger

Hi HN! We’re Adrien and Kanav. We met at our previous job, where we spent about a third of our lives combating a constant firehose of bugs. In the hope of reducing this pain for others in the future, we’re working on automating debugging. We’re currently working on a platform that ingests logs and then automatically reproduces, root causes and ultimately fixes production bugs as they happen. You can see some of our work on this here - https://news.ycombinator.com/item?id=39528087 As we were building the root-cause phase of our automated debugger, we realized that we developed something that resembled an omniscient debugger. Like an omniscient debugger, it also keeps track of variable assignments over time but, you can interact with it at a higher level than a debugger using natural language. We ended up sticking it in a pytest plugin and have been using it ourselves for development over the past few weeks. Using this pytest plugin, you’re able to reason at a much higher level than conventional debuggers and can ask questions like: - Why did function x get hit? - Why was variable y set to this value? - What changes can I make to this code to make this test pass? Here’s a brief demo of this in action: https://www.loom.com/share/94ebe34097a343c39876d7109f2a1428 To achieve this, we first instrument the test using sys.settrace (or, on versions of python >3.12, the far better sys.monitoring!) to keep a history of all the functions that were called, along with the calling line numbers. We then re-run the test and use AST parsing to find all the variable assignments and keep track of those changes over time. We also use AST parsing to obtain the source code for these functions. We then neatly format and pass all this context to GPT. We’d love it if you checked the pytest plugin out and we welcome all feedback :) . If you want to chat bugs, our emails are also always open - kanav@leaping.io