The Local AI Self-Hosted Stack

A.R.C. Freshness Analysis

Tool Stack

1. Install Ollama and pull your first model

   Install Ollama from ollama.com — it runs as a local server that manages model downloads and inference. Run `ollama pull llama3.2` for a capable model that runs on most hardware. For 8GB+ VRAM, try `ollama pull mistral` or `ollama pull qwen2.5:14b`. Verify with `ollama run llama3.2` in your terminal before moving on.
2. Connect Open WebUI as your chat frontend

   Run Open WebUI via Docker: `docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway ghcr.io/open-webui/open-webui:main`. It auto-detects your local Ollama instance. Visit localhost:3000 and create a local account — you now have a ChatGPT-quality interface running entirely offline with conversation history, model switching, and file uploads.
3. Add Jan for desktop model exploration

   Install Jan from jan.ai for a native desktop app to browse, download, and run models without the command line. Use the model hub to try quantized Llama 3, Mistral, and Phi-3 variants before committing to a model for production. Jan runs its own local server on port 1337 and complements Ollama for model experimentation.
4. Configure Groq as your cloud inference fallback

   Sign up for a free Groq API key at console.groq.com — the free tier allows 14,400 tokens/min. In Open WebUI go to Settings → Connections and add Groq as an OpenAI-compatible endpoint using https://api.groq.com/openai/v1 and your key. You can now switch to llama-3.1-70b-versatile when local inference is too slow or you need higher output quality.
5. Create model presets and validate the full stack

   In Open WebUI, create two presets: one routing to local Ollama models (offline-safe, zero cost) and one to Groq (fast cloud). Test both with the same prompt and compare quality vs latency. Use local for drafts and iteration, Groq for final outputs where quality matters. You now have a fully private AI stack with a paid-quality escape hatch.