llama.cpp
An open-source C/C++ runtime for running LLMs efficiently on CPUs and consumer GPUs.
llama.cpp runs language models with minimal dependencies, in portable C/C++, with aggressive quantisation. It made running capable models on a laptop CPU or a modest GPU genuinely practical, and its GGUF format is a de facto standard.
It is the engine quietly powering many higher-level tools, including much of what Ollama does under the hood.
Where it fits
A fit when models must run on constrained or CPU-only hardware, or when you want a tiny, dependency-light runtime.
Where it does not fit
A low-level tool: for high-concurrency GPU serving a dedicated engine is more efficient, and for an easy local setup a wrapper is friendlier.
Alternatives to llama.cpp
The closest options on the map, and where each fits instead.
-
Ollama Local model runtime
The fastest way to get a local model running for development, demos, or a small-scale internal tool.
Unlike llama.cpp Maturity: established
-
LocalAI Local model runtime
A fit when you want one self-hosted, OpenAI-compatible endpoint covering text, images, and audio.
Unlike llama.cpp Maturity: established
Pairs with
Commonly used alongside it.
