KV
KV-psi, using Linux PSI to to trim an LLM KV cache
Hacker NewsKV-psi, using Linux PSI to to trim an LLM KV cache
I thought it'd be interesting to use Linux PSI (Pressure Stall Information) for an LLM runtime to trim the KV cache. This is mainly useful imo for edge devices like the Jetson Orin super nano kit which have unified memory. I haven't benched much, but plan to do so more over time and see if I can make a real use of it as I run local LLMs. Let me know if it makes sense :P (I of course vibed this idea)
Share cardActual performance
8points
Made the leaderboard
Launch Intel predictions
Analyze your own launch →
Product HuntOn track for Day 1 leaderboard · Strong signals: using · Missing: mac, agents, macos72%72% predicted probability of success on Product Hunt, based on ML models trained on real launch data.
best fitHighest predicted score across all platforms for this description.
Hacker NewsStrong engagement from HN community · Strong signals: ide, io · Missing: https docs, excited, just released54%54% predicted probability of success on Hacker News, based on ML models trained on real launch data.
nativeThis product was originally launched on this platform.
TrustMRRLess likely to generate early MRR · Missing: mobile apps, ios, personal40%40% predicted probability of success on TrustMRR, based on ML models trained on real launch data.
AppSumoMay struggle as an AppSumo deal · Missing: plus, platform, intuitive39%39% predicted probability of success on AppSumo, based on ML models trained on real launch data.
Indie HackersIH features products with proven revenue · Missing: supports, reddit linkedin, podcasting32%32% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.
Acquire.comPre-revenue stage for this audience · Missing: arr, mrr, revenue15%15% predicted probability of success on Acquire.com, based on ML models trained on real launch data.
BetaListMay not resonate with beta-testers · Missing: web3, chat, crypto2%2% predicted probability of success on BetaList, based on ML models trained on real launch data.
Correct prediction on native model
Similar products
Cr
Credentials dumper for Linux using eBPF62%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.
Credentials dumper for Linux using eBPF
Hacker News235Gu
Guiding LLM outputs using Zod42%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.
Guiding LLM outputs using Zod
Hacker News3Va
Variant Symlink Filesystem for Linux85%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.
Variant Symlink Filesystem for Linux
Hacker News2Di
Distributed memcached in Linux kernel71%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.
Distributed memcached in Linux kernel
Hacker News7Co
Compressed directories on Linux60%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.
Compressed directories on Linux
Hacker News70Sq
Sqlite3-X64 – SQLite Precompiled Binaries for Linux X6467%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.
Sqlite3-X64 – SQLite Precompiled Binaries for Linux X64
Hacker News2Ex
ExpanDrive for Linux72%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.
ExpanDrive for Linux
Hacker News14An
An Aesthetically Pleasing Arch Linux Distro72%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.
An Aesthetically Pleasing Arch Linux Distro
Hacker News76Co
CoreOS, a Linux distro for containers67%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.
CoreOS, a Linux distro for containers
Hacker News41720
2048.ko: 2048 as a linux kernel module47%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.
2048.ko: 2048 as a linux kernel module
Hacker News3