> your AI agent picks dependencies from memory; give it dated facts — try starlog.dev ↗ vet your agent's deps ↗ vibe-coding is fine. vibe-importing isn’t. — try starlog.dev ↗ vibe-importing isn’t fine ↗ your agent has never seen your private packages — try starlog.dev ↗ facts for private packages ↗ a linter for the dependencies your AI agent picks — try starlog.dev ↗ a linter for agent deps ↗ whois is redacted, cdns mask the rest — get the real operator — try whoisgeni.us ↗ who really runs that domain ↗ domain attribution that shows its work — full evidence chain — try whoisgeni.us ↗ domain intel w/ evidence ↗

← Back to Articles

Ollama: The Docker-Inspired LLM Server That Made llama.cpp Accessible

[ View on GitHub ]

Ollama: The Docker-Inspired LLM Server That Made llama.cpp Accessible

Hook

Ollama has more GitHub stars than Docker Compose. In two years, it became the default way developers run local LLMs—by treating models like container images with layered blobs and content-addressable storage.

Context

Before Ollama, running local LLMs meant compiling llama.cpp from source, converting model weights with Python scripts, memorizing command-line flags for context windows and GPU layers, and manually downloading multi-gigabyte GGUF files from Hugging Face. Each model required reading documentation to figure out prompt templates. Want to switch from Mistral to Llama? Start over.

The fragmentation was worse for developers building applications. Every inference engine had different APIs—llama.cpp used stdio, vLLM required gRPC knowledge, and Hugging Face's text-generation-inference expected specific request formats. OpenAI's API had become the de facto standard, but replicating it locally meant writing adapter code for each backend. Ollama emerged in 2023 with a radical premise: treat LLMs like Docker treats applications. Models become versioned artifacts with declarative configuration files, the runtime handles hardware detection automatically, and a single REST API provides OpenAI compatibility. One command—'ollama run llama3.1'—downloads, configures, and starts inference. It worked so well that the project exploded from zero to 185,000 stars, becoming the bridge between llama.cpp's performance and developers who just want to ship features.

Technical Insight

Inference Engine

Storage Layer

Go Layer

Chat/Completion Request

Schedule

Load Model

Pull Missing Layers

Download Blobs

Read Weights & Config

Deduplicated Layers

Initialize

llama_load_model

Tensor Ops

Inference Call

Tokens

Streaming Response

HTTP Client

REST API Server

Port 11434

Model Registry

ollama.com

Request Scheduler

& Context Manager

Model Loader

& Memory Manager

Content-Addressable

Blob Store

Modelfile Parser

Layered Manifests

CGO Bindings

llama.cpp

C++ Runtime

GPU Backend

CUDA/Metal/ROCm

System architecture — auto-generated

Ollama's architecture is a three-layer cake with Go on top, llama.cpp at the bottom, and a clever storage system in between. The HTTP server exposes an OpenAI-compatible endpoint on port 11434, but the magic happens in how models are stored and loaded.

The Modelfile specification is genuinely innovative. It mirrors Dockerfile syntax but for neural networks:

FROM llama3.1:70b
PARAMETER temperature 0.8
PARAMETER num_ctx 4096
TEMPLATE """{{- if .System }}<|system|>{{ .System }}<|end|>{{- end }}
<|user|>{{ .Prompt }}<|end|>
<|assistant|>"""
SYSTEM "You are a helpful coding assistant."

This gets compiled into a layered blob structure stored in ~/.ollama/models/blobs. Each directive becomes a separate layer: one for base weights, one for configuration, one for the system prompt. The storage uses content-addressable hashing—if you create ten variants of Llama 3.1 with different system prompts, you store the 40GB weight file once. Running ollama create my-coder -f Modelfile produces an immutable digest like sha256:a3f5b8c2... that can be version-controlled.

The Go-to-C++ bridge is where performance meets pragmatism. Ollama uses CGO to call llama.cpp's inference engine directly:

// Simplified from internal/llm/llm.go
/*
#cgo LDFLAGS: -lllama
#include "llama.h"
*/
import "C"

type LLM struct {
    ctx *C.struct_llama_context
    model *C.struct_llama_model
}

func (l *LLM) Generate(prompt string) (string, error) {
    tokens := C.llama_tokenize(l.ctx, C.CString(prompt))
    for C.llama_decode(l.ctx, tokens) == 0 {
        token := C.llama_sample_token(l.ctx)
        // Stream token back to HTTP client
    }
}

This tight coupling delivers near-native performance—there's minimal overhead between the HTTP request and tensor operations. But it also means Ollama releases are locked to specific llama.cpp versions. When llama.cpp adds support for a new model architecture like Qwen2-VL, Ollama users wait weeks for integration.

The automatic backend selection is surprisingly sophisticated. At startup, Ollama probes for CUDA libraries, checks for Metal support on macOS, and falls back to CPU. It doesn't recompile—the installer ships with multiple llama.cpp binaries (cuda-11, cuda-12, rocm, metal, avx2, avx512) and selects at runtime. This is why the installer is 600MB but requires zero configuration.

The concurrency model reveals trade-offs clearly. Ollama uses a per-model mutex—incoming requests queue until the previous inference completes. You can run Mistral on GPU 0 and Gemma on GPU 1 simultaneously, but two users hitting the same Llama model serialize. There's no KV-cache sharing, no speculative decoding, no continuous batching like vLLM. For a local development server handling 1-2 requests per second, this simplicity is perfect. For production serving 100 concurrent users, it's a bottleneck.

The API compatibility layer is thin but carefully designed. A request to /api/chat transforms into llama.cpp calls:

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1",
  "messages": [{"role": "user", "content": "Explain pointers"}],
  "stream": true
}'

Returns server-sent events matching OpenAI's format. But look closely—the chunk boundaries differ. OpenAI sends tokens as they're sampled; Ollama batches every 5-10 tokens for efficiency. Most clients don't care, but if you're building a typewriter effect in a UI, you'll notice the choppier streaming.

The model registry at ollama.com is centralized, which is both the product's strength and risk. Running ollama pull deepseek-coder fetches from Ollama's CDN, not Hugging Face. Models are curated and tested, so you avoid broken quantizations or malformed configs. But you're also locked to Ollama's update cadence—when Meta releases Llama 3.2, you wait for Ollama to package it. Power users work around this by converting GGUF files manually and using ollama create with local Modelfiles, but that defeats the simplicity premise.

Gotcha

The single biggest limitation is memory management. Ollama loads entire models into VRAM or RAM with basic LRU eviction. If you have 24GB VRAM and load a 40GB quantized model, it falls back to system RAM at 10-20x slower inference speed. There's no automatic offloading of specific layers or dynamic batching. You're either fully on GPU or slogging through CPU, with no middle ground.

Multi-GPU support is stuck in 2023. You can assign different models to different GPUs using CUDA_VISIBLE_DEVICES, but you cannot shard a single 70B parameter model across two 24GB GPUs. For large model inference, you hit a hard ceiling at whatever fits in your biggest single GPU. Compare this to vLLM's tensor parallelism or TGI's pipeline parallelism, where splitting models across GPUs is built-in.

The Modelfile lack of dependency pinning creates silent reproducibility issues. FROM llama3.1 pulls whatever version is tagged latest at pull time. Six months later, rebuilding the same Modelfile might fetch updated weights if Meta pushed a revision. The fix is using digests (FROM llama3.1@sha256:a3f5b8c2...), but documentation barely mentions this, and most examples don't use it. In production, this is a ticking time bomb for model drift.

API incompleteness shows up in edge cases. The /api/embeddings endpoint exists but doesn't support batching—you must loop and make N requests for N inputs. Function calling requires manually parsing JSON from model outputs; there's no structured output mode like OpenAI's response_format. Vision model support is inconsistent—llava works via base64-encoded images in messages, but bakllava expects different formatting, and there's no documentation explaining why.

Verdict

Use if: You're building AI features locally or self-hosting inference where operational simplicity trumps optimization. Perfect for prototyping with open models, running personal AI assistants, or embedding LLM capabilities in desktop applications. If your workflow is 'just run this model and call it from Python,' Ollama eliminates every other step. Also ideal for teams who want reproducible model configurations without Dockerfiles—Modelfiles give you version-controlled AI infrastructure as code. Skip if: You're serving production traffic above 10-20 requests per second, need to run models larger than your single biggest GPU, or require fine-grained control over quantization formats beyond what llama.cpp offers. Skip if you're doing research on sampling strategies, need KV-cache optimization, or want to batch embeddings efficiently. For those scenarios, spend the extra days fighting vLLM or TensorRT-LLM configuration—the performance gains pay back quickly. Also skip if you're philosophically opposed to centralized model registries; Ollama's convenience is tied to their infrastructure.