Rust-split – save your tokens on large Rust source files

Hacker News

Share card

Rust-split – save your tokens on large Rust source files

A common failure mode for AI SW development is that agents tend to build large "God" files. This is bad on your token burn. A) because the larger your source file the more churn the LLM takes to find and emit edits, which are more likely to fail, and B) the real kicker - the agent's process is to move one item at a time, which is an O(n^2) algorithm, but especially bad because it inevitably breaks and the code mushes up and sadness ensues (burning time and tokens). So, I have had lots of success with rust-split and a few skill notes. rust-split uses the AST (syn) to find the correct boundaries. Two passes: explode cuts the file into top-level items with their comments and #[attrs] attached - lossless, cat the chunks together and you get the original back byte for byte, so you can check it before trusting it - then split writes the module layout under a LOC ceiling. What's left for the LLM is basically fixing imports, which doesn't burn anywhere near as many tokens. It's not perfect with the transition but fixing small import issues LLMs can do efficiently. rust-split is released as a GitHub attested release or a crates.io install: https://github.com/owebeeone/rust-split cargo install rust-split Be sure to add the skill file ( https://github.com/owebeeone/rust-split/blob/main/skills/rus... ) and set your project's line thresholds. I chose a soft limit of 1000 LOC to trigger the decision to split (I've found that at 2000 LOC agents lose it, so give yourself a safety margin) and the target max size of a split file to be 500 LOC, but preferably more smaller files than 2x500 LOC files. Enjoy