Running local AI in 2026 is practical enough that it belongs in a homelab checklist, but not magical enough to replace every API. Ollama makes model management approachable, llama.cpp gives power users fine-grained control, and Jetson-class or used-GPU hardware can handle useful workloads when expectations are realistic.
The best reason to run local models is not bragging rights. It is control. Local inference keeps sensitive prompts closer to your network, works when cloud APIs are down, enables cheap experimentation, and lets you wire AI into home automation, note search, code review helpers, or lab dashboards without sending every request outside.
Start with the workload
Before buying hardware, write down the jobs you want the model to do. Private note summarization, local embeddings, log classification, simple coding help, and home assistant commands are good local candidates. Deep architecture review, frontier coding agents, difficult math, and broad research synthesis are often better served by API models.
For each job, measure quality and latency. A model that produces 18 tokens per second may feel fine for short answers and painful for long reports. A small quantized model may classify logs well but hallucinate when asked to design a Kubernetes migration. Local AI is strongest when the task is narrow, the data is private, and the acceptable answer format is clear.
Hardware reality: memory, thermals, and patience
The limiting factor is usually memory bandwidth and VRAM, not just model size on disk. Quantization helps, but every compromise changes quality. Jetson-class devices are attractive for always-on edge workloads because they are small and efficient, yet they require patience around model selection, cooling, and container support. A used desktop GPU may be faster but louder, hotter, and more power hungry.
Track tokens per second, peak memory, temperature, and failed prompts in a simple spreadsheet. Do not judge a setup by one impressive demo. Let it run your actual tasks for a week. If it only saves time when you are excited to babysit it, it is not automation yet.
Ollama vs llama.cpp
Ollama is the friendly default for many homelabs: pull a model, run it behind an HTTP API, and connect tools quickly. It is excellent for experimentation and for services where ease of updates matters. llama.cpp is the workshop bench. It exposes more knobs around model formats, quantization, backends, and performance tuning, which matters when you are squeezing useful inference out of constrained hardware.
A practical stack can use Ollama for day-to-day model serving and llama.cpp for experiments or specialized deployments. Put both behind a small routing layer if you have multiple apps. That router can decide whether a request stays local, goes to a stronger API model, or asks the user before sending sensitive context outside.
When APIs still win
Hosted APIs still win for frontier reasoning, huge context windows, bursty traffic, and managed reliability. If your local box takes five minutes to answer a question that an API handles in ten seconds, privacy may not be worth the productivity loss unless the data is genuinely sensitive. Hybrid is not a compromise; it is the sane architecture.
Bring your model list, hardware specs, and benchmark results to the Tech Talk Live Local AI & Ollama or Home Lab stages. A screen share of your thermals, Docker logs, and prompt failures will get better advice than a generic “what GPU should I buy?” thread.