Python Bindings for llama.cpp with some CLIs

Hacker News

These are my Python bindings for @ggerganov's llama.cpp. My Python bindings build on this work and provide an easy-to-use interface for Python developers to take advantage of LlamaCPP's powerful inference capabilities. The bindings currently use code from a pending PR of mine to make the original code into more of a library. Hopefully it will get merged into the main repository soon. I have also added a few CLI entry points that get installed along with the python package: * llamacpp-convert - convert pytorch models into GGML format. This is an alias for the existing Python script in llama.cpp and requires PyTorch * llamacpp-quantize - Perform INT4 quantization on the GGML mode. This is a wrapper for the "quantize" C++ program from the original repository and has no dependencies. * llamacpp-cli - This is a Python version of the "main.cpp" program from the original repository that utilizes the bindings. * llamacpp-chat - A wrapper over llamacpp-cli that includes a prompt that makes it behave like a chatbot. This is not very good as of right now. You should theoretically be able to do "pip install llamacpp" and get going on most linux/macOS platforms by just running `llamacpp-cli`. I do not have Windows builds on the CI yet and you may have to build it yourself. The package has no dependencies if you just want to run inference on the models.

Visit product View on Hacker NewsMar 19, 2023

Share card

Actual performance

2points

Did not reach leaderboard

Launch Intel predictions

Analyze your own launch →

Product HuntOn track for Day 1 leaderboard · Strong signals: mac, macos, model · Missing: agents, agent, cursor

82%82% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

Indie HackersFits the IH revenue-focused audience · Missing: supports, reddit linkedin, podcasting

79%79% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

Hacker NewsStrong engagement from HN community · Strong signals: exist, existing, llama · Missing: https docs, excited, just released

60%60% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

TrustMRRFits verified-revenue profile · Missing: mobile apps, ios, personal

50%50% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

AppSumoMay struggle as an AppSumo deal · Strong signals: platform, interface, soon · Missing: plus, intuitive, reviews

48%48% predicted probability of success on AppSumo, based on ML models trained on real launch data.

Acquire.comPre-revenue stage for this audience · Missing: arr, mrr, revenue

16%16% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Strong signals: chat · Missing: web3, crypto, cryptocurrency

0%0% predicted probability of success on BetaList, based on ML models trained on real launch data.

Incorrect prediction on native model

Similar products

Llama or Alpaca?78%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Llama or Alpaca?

Hacker News6

Llama 3.2 Interpretability with Sparse Autoencoders74%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Llama 3.2 Interpretability with Sparse Autoencoders

Hacker News579

CustomStringParser ( Python)54%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

CustomStringParser ( Python)

Hacker News1

Lusernews is an implementation of lamernews in python48%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Lusernews is an implementation of lamernews in python

Hacker News1

Do/While in Python55%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Do/While in Python

Hacker News4

Gatos, conCATenative functions for python54%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Gatos, conCATenative functions for python

Hacker News1

A wrapper around datetime, pytz and tzlocal for Python 3.2+38%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

A wrapper around datetime, pytz and tzlocal for Python 3.2+

Hacker News1

Parallelizing word2vec in Python55%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Parallelizing word2vec in Python

Hacker News3

piprot – How rotten are your [python] requirements?37%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

piprot – How rotten are your [python] requirements?

Hacker News1

Fancy iterators for Python 2.7 - 3.x54%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Fancy iterators for Python 2.7 - 3.x

Hacker News5

Browse all products Analyze your launch