Local AI is one of the biggest conversations on X right now: we counted about 9,300 posts on local models and Ollama in our sample. If you've never run a model on your own machine, Ollama is the easiest place to start. It's free to run locally, needs no API key, and your prompts stay on your computer.
Step 1: install
Download Ollama for macOS, Windows or Linux and open the app. Or open a terminal and run:
ollama
Then follow the setup prompts. You can sign in for cloud models or pick a local one. This guide is about local.
Windows: you need Windows 10 22H2 or newer. No admin rights are required; Ollama installs into your home folder. Linux: if the server isn't running, start it with ollama serve.
Step 2: run your first model
ollama run gemma4:e2b
Ollama downloads the model and opens a chat in your terminal. Type a question; type /bye to leave.
Will my computer handle it? Ollama says the download is about 7.2 GB, and it recommends 8 GB of available VRAM, or unified memory on a Mac. With less, Ollama can use system RAM, but responses may be slower. Larger context windows need more memory. Our VRAM guide explains the maths.
Tips for the chat:
- Wrap multi-line input in
"""…""". - Vision models can read images:
ollama run gemma4 "What's in this image? ./screenshot.png"
Step 3: the commands you'll actually use
| Command | Does |
|---|---|
ollama run <model> |
chat with a model (downloads it first if needed) |
ollama pull <model> |
download without chatting |
ollama ls |
list downloaded models |
ollama ps |
show models loaded in memory right now |
ollama stop <model> |
unload a running model |
ollama rm <model> |
delete a model and free disk space |
ollama serve |
start the server (mostly Linux) |
Models take up a lot of disk, from a few GB to hundreds. On Windows, set the OLLAMA_MODELS environment variable to keep them on another drive.
Step 4: give it a personality (Modelfile)
Create a file named Modelfile:
FROM gemma4
SYSTEM """You are a patient coding tutor. Explain things in plain English."""
Then build it with ollama create (see ollama create --help for naming options). It's a quick way to save a system prompt you use often.
Step 5: call it from your app
Ollama runs a local server at http://localhost:11434. It speaks its own API and OpenAI- and Anthropic-compatible ones:
curl http://localhost:11434/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "gemma4:e2b", "messages": [{"role": "user", "content": "Say hello in one sentence."}]}'
With the OpenAI SDK, change two lines:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1/", api_key="ollama") # key required but ignored
Ollama supports a subset of the OpenAI API, so test the features you rely on.
Step 6: connect a coding agent
From your project folder:
ollama launch claude # Claude Code
ollama launch codex # Codex CLI (install it first)
ollama launch opencode # OpenCode
Ollama offers to install Claude Code or OpenCode if they're missing. Pick a model when prompted, then try: "Explain how this repository is organized." A small starter model will struggle with real agent work. See our comparison of local coding models and the full OpenCode + local LLM setup.
Good habits
- Start small, then go bigger. Get a small model working before downloading a 60 GB one.
- Check
ollama psif your computer slows down. A loaded model holds memory until it's stopped or unloaded. - Local isn't automatically safe. Agents connected to local models can still run commands. Read running coding agents safely.
- Check the licence on each model page before you build a product on it.
