Skip to content
Atlas by Comet Lab
Model layer

Model serving, compared

Model serving turns model weights into an endpoint your application can call. It comes in three kinds: local runtimes for a laptop or a small internal tool, inference engines for self-hosted production traffic, and hosted platforms that run open models for you.

Learn the idea Open-source vs. closed AI models: what is actually open on Horizon

Open this layer in the explorer

At a glance

Every tool in this layer, with where it fits and the facts that usually decide it.

Tool Where it fits Licence Self-hostable Hosted API Maturity
Ollama Local model runtime The fastest way to get a local model running for development, demos, or a small-scale internal tool. Open source Yes No Established
Baseten Model deployment platform A fit when you need to deploy custom or fine-tuned models as reliable, autoscaling production endpoints. Proprietary No Yes Growing
Fireworks AI Hosted inference A fit when you want fast, cost-efficient hosted inference for open models without running GPUs. Proprietary No Yes Growing
Groq Hosted inference The pick when response latency is part of the experience (live voice, interactive agents) and a hosted service is acceptable. Proprietary No Yes Growing
llama.cpp Inference runtime A fit when models must run on constrained or CPU-only hardware, or when you want a tiny, dependency-light runtime. Open source Yes No Mature
LM Studio Local desktop runtime A fit when an individual wants to discover and run local models through a friendly desktop GUI. Proprietary Yes No Established
LocalAI Local model runtime A fit when you want one self-hosted, OpenAI-compatible endpoint covering text, images, and audio. Open source Yes No Established
Modal Serverless GPU A fit when you want to serve custom models or run GPU workloads on demand without managing infrastructure. Proprietary No Yes Established
Replicate Hosted inference A fit when you want to use or fine-tune open models, especially multimodal ones, through one hosted API. Proprietary No Yes Established
SGLang Inference engine A fit for high-throughput self-hosted serving, especially with structured output or heavy shared-prefix workloads. Open source Yes No Growing
Text Generation Inference Inference engine A fit for production self-hosted serving, especially for teams already deep in the Hugging Face ecosystem. Open source Yes No Established
Together AI Hosted inference A fit when you want hosted access to many open models, with fine-tuning, instead of running inference yourself. Proprietary No Yes Established
vLLM Inference engine The right engine when you self-host a model under real concurrency and need to use GPU capacity efficiently. Open source Yes No Established

How to choose

Start from your situation. Where a line says “our”, it comes from Comet Lab’s own builds.

  1. Running a model locally while you build

    Ollama is how we run models locally; LM Studio does the same through a desktop app. Neither is a production serving tier.

  2. Self-hosted production traffic

    vLLM is our default for production serving; SGLang suits structured output and shared-prefix workloads; TGI fits teams deep in Hugging Face.

  3. CPU-only or constrained hardware

    A tiny, dependency-light runtime for CPUs and consumer GPUs.

  4. One OpenAI-compatible endpoint for text, images and audio

    Self-hosted, over many model backends.

  5. Open models, without running GPUs

    Groq when the user feels the latency; Together AI and Fireworks for hosted inference on many open models; Replicate for multimodal models through one API.

  6. Deploying your own custom or fine-tuned models

    Autoscaling production endpoints, or GPU workloads on demand, without managing infrastructure.