Run Open Models Locally with Ollama: A Practical Guide

Why Run Models Locally?

Cloud models are convenient, but local inference offers three advantages that matter in practice: privacy, cost, and availability. Prompts and source code never leave your machine, there is no per-token bill, and the model keeps working when the network does not.

The trade-off is capability. A 7B or 8B model running on a laptop will not match a frontier model on complex reasoning. Local models are best for autocomplete, summarization, classification, extraction, simple code edits, and offline experimentation.

Install Ollama

macOS and Linux:

curl -fsSL https://ollama.com/install.sh | sh

Windows PowerShell:

irm https://ollama.com/install.ps1 | iex

On macOS you can also install the desktop app with brew install ollama. The app starts the background server automatically.

Pull and Run a Model

ollama run gemma4

The first run downloads the model weights. Subsequent runs start instantly. To list what you have installed:

ollama list

Popular starting points:

• gemma4 for a balanced general-purpose model
• qwen3 for strong multilingual and coding performance
• llama3.1 for a widely supported, well-documented baseline
• deepseek-r1 for reasoning-heavy tasks

Hardware Sizing

Memory is the constraint, not CPU speed alone. As a rough guide:

Quantized versions use less memory at a small quality cost. Ollama selects a sensible quantization by default.

The Local API

Ollama exposes an HTTP API on port 11434. Once the server is running, you can call it from any language:

curl http://localhost:11434/api/generate -d '{
  "model": "gemma4",
  "prompt": "Explain the difference between an agent and a workflow.",
  "stream": false
}'

This makes Ollama a drop-in backend for tools that support an OpenAI-compatible endpoint at http://localhost:11434/v1.

Connect It to Your Editor or Agent

Most AI coding tools let you add a custom OpenAI-compatible provider. Point the base URL at http://localhost:11434/v1, set any non-empty API key, and choose the model you pulled. This gives you private autocomplete and chat inside your existing editor without sending code to a vendor.

When Not to Use Local Models

Do not use a small local model for high-stakes refactors, security review, or tasks that require long-horizon planning. Use it for the work it is good at, and route hard problems to a frontier agent such as Claude Code, Codex, or Antigravity.

A Sensible Hybrid Setup

Run Ollama for routine, private tasks and keep one frontier agent for complex changes. You get most of the cost and privacy benefits without giving up quality when it matters.