Running large models locally

Running a model locally means keeping its weights (the mass of numbers learned in training) as files on your machine and doing the maths on your own GPU or CPU. Questions and answers never leave the computer.

It helps to see the setup in layers: app, inference engine, GPU API and memory. When people talk about the "CUDA camp" and the "Metal camp", they mean the bottom two.

• CUDA is NVIDIA's platform for computing on its GPUs. Nearly all AI software, PyTorch first of all, supports it before anything else.
• Metal is Apple's GPU platform, MPS is the part that runs PyTorch on it, and MLX is Apple's own machine-learning framework for Apple Silicon. On a Mac, the MLX version of a model is usually the fastest.
• Vulkan is a vendor-neutral GPU interface; llama.cpp uses it on AMD and Intel GPUs.
• NPUs (Copilot+ PCs, Apple's Neural Engine) are built for AI too, but as of autumn 2026 they are not yet the main way to run large language models freely.

• LM Studio: search, download, load, chat and even an API server, all in one app. Apple Silicon Macs; Windows on x86 and ARM64 (Snapdragon); Linux on x86 and ARM64 (per the download page). This guide uses it.
• Ollama: the classic one-liner, ollama run model-name. Mac and Windows get a GUI app too; on Linux you install it from the command line.
• llama.cpp: the inference engine inside LM Studio and Ollama, for people who want fine control.
• vLLM: a fast serving engine for many simultaneous users, mainly Linux plus NVIDIA.

Model names and download pages are full of cryptic labels. Once you can read them, you can tell at a glance whether a model will run on your machine.

Memory needed is roughly parameters × bits ÷ 8. The chart shows the actual file sizes of Qwen3.8-27B, the model used here.

Speed is set almost entirely by how fast the weights can be read from memory (bandwidth), because every new token means reading all the weights again.

Release dates, sizes and licences were checked on each model's official Hugging Face page. "Commercial" is ticked for Apache 2.0, MIT and similar licences that allow it (check each licence for the details).

Text and chat (LLMs)

Text to image

Voice cloning (speech synthesis)

Only clone voices whose owners have agreed. Publishing audio made with someone else's voice as if it were theirs invites fraud and rights violations.

OCR (reading text in images and PDFs)

Translation

The LLMs above (Qwen3.8, Gemma 4) translate well too. Dedicated models win on being small and fast.

For one model that is easy to run locally and still highly accurate, our pick is Qwen3.8-27B, released in August 2026.

• Accurate: in Qwen's own comparison it beats the previous Qwen3.6-27B across the board; for writing, code, research and agent tasks it is at the top of its size class (SWE-bench Pro 61.7 against 53.5; GPQA Diamond 89.2).
• Just fits: 16.8 GB at 4-bit (Q4_K_M), so it sits on a 24 GB GPU (RTX 3090, 4090 and the like) or a Mac with 32 GB or more.
• Reads images: hand it photos, diagrams or PDF screenshots. Context is 262,144 tokens.
• Apache 2.0: free for commercial use.
• Runs anywhere: GGUF and MLX builds are published, so LM Studio, Ollama (ollama run qwen3.8), llama.cpp and vLLM all work.

The screenshots use LM Studio 0.4.25 for Linux. The Mac and Windows versions look the same apart from the window frame and menus.

• Open lmstudio.ai/download. The top of the page features a separate product, LM Studio Bionic (an agent app); what we want is Download LM Studio further down.
• Mac: open the dmg and drag LM Studio into Applications. Windows: run the installer. Linux: make the AppImage executable (chmod +x) and run it.

No sign-in is needed. That's the setup done.

• On a Mac, choose lmstudio-community/Qwen3.8-27B-MLX-4bit at step 3: the MLX build for Apple Silicon, which runs faster.
• If unsure about quantisation, take Q4_K_M. With memory to spare, Q6_K (22.4 GB) or Q8_0 (29.0 GB) are slightly more accurate.
• The mmproj file for images (about 0.9 GB) downloads too, for about 17.7 GB in all.
• The red Likely too large means "this machine probably lacks the memory". It shows because our test machine has 8 GB; you won't see it on a 24 GB GPU or a 32 GB Mac.

Choose the model from "Pick a model" under the chat (⌘+L on a Mac, Ctrl+L on Windows and Linux) and the load settings open.

• Context Length: 8,192 is fine to start. Raise it for long documents (memory use rises too).
• GPU Offload: on a machine with a GPU, slide it all the way right. All layers on the GPU is fastest; anything left over runs on the CPU and slows things down.

Once loaded, the model's name sits left of the input box. Type a question and press Enter. The Think button toggles the reasoning process: on for hard problems, off for everyday chat, which is faster.

LM Studio can also act as a server you call exactly like the OpenAI API, so your own programs, and tools that accept an OpenAI-compatible API, can use the model on your machine.

# Send a question from a terminal (write curl.exe in Windows PowerShell)
curl http://localhost:1234/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3.8-27b", "reasoning_effort": "none",
       "messages": [{"role": "user", "content": "Hello"}]}'

Through the API, Qwen3.8 also thinks out loud by default. Adding "reasoning_effort": "none" skips that and answers straight away (checked on our test machine).

Without the GUI, the lms command does the same (lms get to download, lms load to load, lms server start to serve). With Ollama, the one line ollama run qwen3.8 takes you from download to chat.