Ollama

How to Run a Model with Ollama

1. Install Ollama: curl -fsSL https://ollama.ai/install.sh | sh
2. Pull a model: ollama pull llama3
3. Run the model interactively: ollama run llama3
4. Call via REST API: POST http://localhost:11434/api/generate

Also see: Jan · LM Studio

Is Ollama Free?

Yes — Ollama is fully free and open-source. You run models on your own hardware with no API costs, no usage limits, and no data leaving your machine. The only cost is the compute you provide.

Which Ollama Model Should I Run for My Hardware?

Rule of thumb: your GPU VRAM (or unified RAM on Apple Silicon) must fit the quantized model size. 4–6 GB VRAM: llama3.2:3b, phi3:mini, gemma2:2b. 8 GB VRAM: llama3.1:8b, mistral:7b, qwen2:7b — the sweet spot for most developers. 16 GB VRAM: llama3.1:70b with Q4 quantization, deepseek-coder:33b. 24 GB+: llama3.1:70b at Q8 or any 13b model at full precision. Run 'ollama ps' to see what's currently loaded. On Apple Silicon, Ollama uses Metal — a 16 GB M-series chip handles 7b models comfortably.

Can I Fine-Tune a Model With Ollama?

Not directly — Ollama is an inference runtime, not a training framework. The typical workflow: fine-tune with Unsloth (uses ~40% less VRAM than standard LoRA), export the adapter as GGUF format via llama.cpp's convert script, then run 'ollama create my-model -f Modelfile' to import it into Ollama for local serving. Unsloth supports Llama 3, Mistral, Qwen2, Phi, and Gemma families.

Also see: Unsloth