Local LLMs 2026: Run Qwen3.8, Gemma 4 and gpt-oss Locally

Start with Qwen3.8-27B at Q4_K_M for a capable workstation assistant, or Qwen3.5-4B for a smaller machine. Gemma 4 adds useful multimodal choices, while gpt-oss is a text-reasoning option with documented memory targets.

Choose the model and quantization together, then reserve memory for context and your applications. The recommendations below are starting configurations for testing, rather than guarantees that every prompt fits or every task succeeds.

Not sure what your GPU can handle? Use our free LLM VRAM Calculator to estimate a model's requirements, then check the loaded runtime.

Quick navigation: Why local · Hardware · Apple Silicon · Models · Llama vs Mistral vs Qwen · Llama 4 · Tools · Quantization · Sizing method · Speed expectations · Use cases · FAQ

Why Run LLMs Locally?

Local inference gives you control over model files, runtime and the destination of your prompts. Start by naming the work you want to keep local: document search, code assistance, extraction or offline chat.

Privacy: keep inference, retrieval and tools inside your chosen boundary. A local model attached to a web-search tool can still send data elsewhere. Check the complete application, including extensions, logs and fallback providers.

Cost: compare the whole system. Include equipment, electricity, storage, maintenance and time spent reviewing outputs. Calculate payback for your own workload and utilization.

Availability: test the workflow without internet access after downloading its files. LM Studio documents offline chat and document interaction; downloading models and using connected services remain separate operations. Offline support.

Customization: keep the checkpoint, prompt template and runtime under version control in your deployment records. Test a new configuration on the same tasks before replacing a working assistant.

Latency: measure the full interaction. Loading, prompt processing, retrieval and tool execution can dominate the wait even with local inference. Decide what response time your workflow requires.

A hybrid workflow can keep sensitive material local and route suitable tasks to a hosted model. Make routing visible to the person submitting the prompt, especially when an application offers automatic fallback.

Use the LLM API pricing calculator and cloud GPU pricing to build that comparison. Base the decision on completed tasks and utilization, rather than comparing a hardware purchase with one month's token bill.

Hardware Reality in 2026

The following tiers refer to GPU VRAM, unless labeled Apple unified memory. They assume one conversation and a short initial context. My suggested first test is 8,192 tokens; increase it after measuring memory.

Every recommendation links to the publisher's model card. The quantization column is the configuration I would try. It does not imply that all downloads with that model name use that representation.

Package sizes come from Ollama's 4B, 9B and 27B pages. These are downloads, not measured peak VRAM.

Google's figures include a loading allowance but exclude context and additional supporting software. Use the Gemma memory guide as the basis for a trial, not as the final allocation for a server.

For gpt-oss, keep the published quantized representation and compatible kernels. Loading a different representation can change the requirements substantially. Publisher deployment notes.

Do not add two cards' capacities and assume every runtime treats them as one device. Confirm the model's parallelism support, distribution settings and per-device allocations. A multi-GPU installation needs its own load test.

For a CPU-only machine, start with the smallest row and measure an actual task. Ollama can place models partly or entirely in system memory; use ollama ps to inspect placement. Placement documentation.

The goal is a configuration that remains responsive while your editor, browser and other applications are open. Leave the exact reserve to measurement rather than applying one fixed percentage across every operating system and runtime.

Running Local LLMs on Apple Silicon

MLX uses a shared memory model on Apple silicon, and llama.cpp provides a Metal backend. Both are relevant local inference paths; LM Studio supports GGUF through llama.cpp and MLX models on Macs. MLX, llama.cpp, LM Studio.

Which memory tier runs which model?

Treat installed unified memory as a shared system resource. The following are my planning choices, not vendor-certified Mac minimums. Start with short context, close unnecessary applications and check memory pressure during generation.

Ollama lists gpt-oss-120b as a 65 GB package. That makes it a candidate for a 128 GB Mac trial, with a meaningful reserve for the system and runtime; long-context capacity still needs measurement. Package listing.

Mac mini, MacBook Pro or Mac Studio?

Choose the form factor after the memory requirement. A portable machine may be the right purchase when you need offline work away from a desk; a desktop may suit a persistent local service. Test the intended workload before buying.

Avoid selecting by chip name alone. Ask for the complete machine specification, the exact quantized artifact and the context used in any demonstration. A result from a different memory configuration is not a prediction for your purchase.

Use our consumer-hardware comparison as a companion when designing your own comparison.

For setup details, continue with local LLMs on Mac. Record memory pressure and application responsiveness along with token generation speed.

MLX versus GGUF

GGUF and MLX packages are different artifacts. Match the download to the runtime rather than renaming a file or assuming one quantization tag means identical storage and execution everywhere.

My suggested starting point is the tool you can reproduce most easily. Use a GUI for exploring prompts, then keep a scripted launch configuration when you need repeatable comparisons or a service used by another application.

MLX-LM supports model conversion, quantization and fine-tuning. Its memory-wiring optimization for large models requires macOS 15 or later; that condition does not establish that every model will fit. MLX-LM README.

Model Picks 2026

General assistance and coding

My workstation shortlist begins with Qwen3.8-27B. Its card describes a dense vision-language model with 262,144 native context and extension to one million tokens. Treat the larger window as a separate deployment exercise. Card.

For smaller hardware, Qwen3.5-4B and 9B provide downloadable alternatives. Their names refer to language-model parameter counts; keep the vision components and actual packaged files in your memory accounting. 4B card, 9B card.

Qwen3-Coder-Next is a specialist candidate for a larger installation: 80B total, 3B active and 262,144 native context. Its active count does not make it a small download. Coding model card.

Text reasoning

Use gpt-oss-20b as a separate text-reasoning baseline. It has 21B total and 3.6B active parameters, while 120b has 117B total and 5.1B active. Both use Apache-2.0. 20b, 120b.

Keep the Harmony format in the integration. Evaluate a complete conversation with tool results and follow-up questions, since a successful single prompt does not test the whole application protocol.

Images, documents and smaller devices

Gemma 4 offers E2B, E4B, 12B, 26B A4B and 31B sizes. E2B and E4B refer to effective parameters; embeddings make their full storage larger. Family card.

The 12B Unified model accepts multimodal input without separate vision and audio encoders. Check your runtime's supported modalities rather than assuming that its text interface exposes every model capability. 12B card.

Large open weights

DeepSeek V4.1 Flash, GLM-5.3 and Kimi K3 have downloadable checkpoints. Their cards belong in server planning, rather than being substituted into a consumer-GPU command without sizing. DeepSeek, GLM, Kimi.

For their sizes and licensing differences, use the open-weight comparison. Verify that a newer hosted name also has the weights your deployment needs.

Llama vs Mistral vs Qwen: Which Family to Pick

Choose a checkpoint before adopting a family-wide conclusion. My first comparison would be deployment fit, followed by task quality. The table summarizes the specific releases relevant to that decision.

Mistral Small 4 has 119B total parameters and 6.5B active; Medium 3.5 has 128B dense parameters. “Small” and “Medium” do not identify laptop memory tiers. Small card, Medium card.

Keep license review attached to the exact artifact. Do not generalize from a vendor's compact model to its largest release, or from permissive inference code to the weights bundled with an application.

Can You Run Llama 4 Locally?

Llama 4 Scout and Maverick are downloadable models, with 109B and 400B total parameters respectively. Both activate 17B per token; Scout advertises 10M context and Maverick 1M. Meta's model card.

Whether either is practical depends on your checkpoint representation, runtime and available memory. Do not size Scout as a dense 17B model, and do not assume its advertised context fits wherever its quantized weights can load.

For a new consumer setup, I would first establish a working smaller assistant. Investigate Scout only when you have a specific requirement and a deployment plan that accounts for weights, caches and operating-system memory.

Compare a local trial with rental options through cloud GPU pricing and the model picker. Include setup time and idle capacity in the comparison.

The Colibri GLM deployment article is a companion for exploring offload approaches. Check its configuration carefully; a small GPU allocation is not the whole machine's memory requirement.

Tooling: How to Run Them

Ollama: a straightforward local starting point

Install Ollama from its official download instructions and start the application. On Linux, its quickstart says to run ollama serve if the server is not already running. Official quickstart.

For the smallest hardware tier in this guide, use the explicitly listed local model tag:

ollama run qwen3.5:4b

On the workstation tier, use:

ollama run qwen3.8:27b

Those tags are documented in the 4B and 27B library entries. After loading, inspect the model's placement:

ollama ps

Inside the interactive session, set an explicit context for your initial test:

/set parameter num_ctx 8192

Ollama documents num_ctx and OLLAMA_CONTEXT_LENGTH. It also supports parallel requests and multiple loaded models when memory allows; each additional workload needs capacity. FAQ.

For an application using Ollama's native API, this is a small local request after the model has been downloaded:

curl http://localhost:11434/api/chat \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3.5:4b","messages":[{"role":"user","content":"Explain this project in three sentences."}],"stream":false,"options":{"num_ctx":8192}}'

The chat endpoint accepts messages, runtime options and a streaming flag. This example requests one complete response. Chat API reference.

Check whether the selected tag runs locally before submitting private material. Ollama's library also contains cloud tags; sharing a command-line tool with local models does not make a cloud model local. Library example.

LM Studio: explore models with a GUI

LM Studio supports local GGUF and, on Apple silicon, MLX models. It provides chat, document interaction and local API serving. It also supports running without the GUI through llmster. Application documentation.

Download a compatible quantized model, then use the CLI to inspect what is available and select a load configuration:

lms ls
lms load --context-length 8192
lms server start

With no model key, lms load prompts for a downloaded model. For a named artifact, preview its allocation before loading:

lms load "MODEL_KEY_FROM_LMS_LS" --context-length 8192 --estimate-only

Replace the placeholder with the exact key. The estimator considers context, offload and vision settings; confirm its estimate with a real request. Load command, server startup.

Our LM Studio Bionic guide provides a companion route through the application. Keep the selected runtime and downloaded file recorded when comparing results.

llama.cpp: direct control over GGUF inference

The current upstream quickstart uses llama cli and llama serve. Its backends include Metal for Apple silicon and GPU/CPU inference paths. Use the installation method documented for your platform. Upstream README.

For a downloaded, compatible text GGUF, a deliberately limited local server example is:

llama serve \
  -m /absolute/path/to/model.gguf \
  -c 8192 \
  --host 127.0.0.1 \
  --port 8080

The file path is a placeholder. Vision configurations can require a matching projector; follow that model's instructions. The server documents model paths, context sizing and offline operation. Server reference.

Prefer explicit artifact paths during comparisons. Automatic downloads are convenient, but a reproducible deployment record should identify the file and revision actually used, along with every context and offload setting.

MLX-LM: a Mac-focused text workflow

Install MLX-LM in your chosen Python environment. This example uses a documented community conversion of gpt-oss-20b; it is distinct from OpenAI's original artifact. Conversion card.

pip install mlx-lm
mlx_lm.chat --model mlx-community/gpt-oss-20b-MXFP4-Q8

Check available options with mlx_lm.generate --help. MLX-LM supports generation, conversion and fine-tuning, but support for a new architecture must be checked separately. README.

Do not assume that a text example also exercises audio or image input. Keep your first test narrow, confirm the format and template, then add the modalities and tool integrations your application needs.

vLLM: evaluate a shared inference service

Use vLLM to evaluate a service for concurrent clients. It supports continuous batching and an OpenAI-compatible server. Choose the installation path for your accelerator. Project README.

For a compatible NVIDIA environment with uv available, the documented installation form is:

uv pip install vllm --torch-backend=auto
vllm serve openai/gpt-oss-20b --host 127.0.0.1 --max-model-len 8192

The short context is an example starting configuration. Follow the hardware prerequisites and model-specific requirements; a successful installation alone does not demonstrate compatible quantization kernels. Quickstart, gpt-oss deployment.

The serving CLI documents --host and --max-model-len; the latter limits the configured context for this example. Serving reference.

Keep serving settings with your evaluation results. Change concurrency and context separately so you can identify which setting caused an allocation failure or a latency increase.

Connecting an editor or application

Start by listing models from the running service and issuing a minimal request. Then test streaming, structured output and tool-call handling individually. Compatible endpoint shapes do not prove identical behavior across all features.

See AI Coding Assistants 2026 for integration options. Confirm the destination endpoint and model before opening a private repository to the assistant.

Quantization Briefly Explained

Quantization represents weights with reduced precision to reduce storage and memory demands. The format is part of the model configuration: two files with similar bit counts can have different tensor layouts, kernels and quality trade-offs.

llama.cpp documents GGUF conversion and quantization workflows, including Q4_K_M and other formats. OpenAI distributes gpt-oss with MXFP4 expert weights. Quantization guide, gpt-oss notes.

My default is to compare a smaller model at a moderate quantization with a larger, more compressed model. Keep prompts and scoring identical. Choose the version that completes the work within your memory and latency budget.

Avoid a universal “negligible quality loss” label. Test the details that matter for your use case: exact identifiers, quotations from documents, valid structured output and the ability to recover from a failed tool call.

Preserve the conversion source and revision when using a community artifact. If a result changes, you should be able to distinguish a model change from a quantization, template or runtime change.

Quick Sizing Method

Start with the downloaded artifact, then add the allocations required by the runtime and workload. Google's memory guide explicitly separates static loading estimates from context and supporting software. Memory guide.

1. Record the actual files. Include weight shards, embeddings, projectors and auxiliary modules used by the selected runtime.
2. Set the workload. Specify context, maximum output, image inputs and concurrent requests before comparing memory figures.
3. Estimate, then load. Use a runtime estimator where available, and compare its result with a representative request.
4. Increase one setting at a time. Keep a record of the first failure or slowdown, then select a configuration with operating headroom.

For a mathematical lower bound, multiply parameter count by bits per weight and divide by eight. This excludes metadata, mixed-precision tensors, caches and software. Use it to check plausibility before loading.

Worked example: the 24 GB workstation

Ollama lists Qwen3.8-27B Q4_K_M at 18 GB. Subtracting that rounded package figure from a nominal 24 GB budget leaves roughly 6 GB before accounting for runtime behavior and other allocations. Package.

That subtraction is an illustration, not a peak-memory measurement. Start at the chosen short context, inspect the allocation and test the actual prompt. If it does not fit comfortably, shorten the workload or choose a smaller model.

Worked example: the 128 GB Mac

The 65 GB gpt-oss-120b Ollama package leaves substantial nominal space in a 128 GB system. Treat this as a reason to test, not a guarantee of full-context operation. Package.

Observe system memory pressure with your normal applications running. Repeat the trial with realistic prompts and tools; a blank chat window cannot establish the capacity of a document-heavy working session.

For MoE models, count all stored experts. Active parameter counts help describe computation but do not replace the full weight budget. The VRAM calculator is a planning aid; the loaded service is the final check.

Realistic Speed and Quality Expectations

A tokens-per-second figure needs a model revision, quantization, runtime, hardware configuration, prompt length and concurrency. Without those details, it cannot support a comparison between your laptop and someone else's server.

For a current hosted comparison, use GPT-6 Astra or Sol, Claude Opus 5.5, and Gemini 3.8 Flash as named baselines. Their vendors document those offerings. OpenAI, Anthropic, Google.

Do not label a local model “GPT-6 quality” based on a different benchmark or an API result. Matched local-quantization results against those current baselines are not published in the model cards used here.

Compare representative work: a repository fix with tests, a cited answer from supplied documents, and an extraction with a known expected result. Set the scoring criteria before looking at model names.

Record incomplete answers and failed runs as failures. A model that requires repeated corrections may cost more time even when its successful generations are quick. Give each candidate the same opportunity to use tools and context.

For hosted workflow choices, consult the coding comparison and Claude developer guide.

Use Cases Where Local Fits

Private code assistance: begin with explanation, search and test generation. Require a reviewed patch and passing checks before accepting edits. Include the repository's conventions in your acceptance criteria.

Document question answering: assemble a representative collection, ask questions with known sources and inspect citations. Include questions the collection cannot answer so you can test whether the system invents support.

Offline drafting: download everything needed, disconnect and test the whole workflow. Check that retrieval, templates and any supporting services are available locally before relying on the setup away from a network.

Extraction: define the schema and validate outputs against it. Include ambiguous examples, missing fields and unusual formatting in the trial. Score the final structured record rather than the surrounding explanation.

Tool-using assistants: start with narrow permissions and reversible actions. Validate arguments and inspect results. Add more tools only after a repeatable test shows that the assistant can choose and use the existing ones correctly.

For each use case, keep a small evaluation set with expected outcomes. Run it whenever you change the model, quantization, prompt template or runtime. This gives you evidence for keeping a change or returning to a working configuration.

FAQ

What local LLM should I run on an 8 GB GPU?

My starting recommendation is Qwen3.5-4B Q4_K_M. Its listed Ollama package is 3.4 GB, which leaves room to test context and runtime allocations. Start short and inspect actual placement rather than forcing a larger model into the tier.

Is a 32 GB Mac enough for Qwen3.8?

For Qwen3.8-27B at Q4_K_M, it is a reasonable trial configuration based on the 18 GB package. It does not imply that the much larger Qwen3.8 releases fit, or that the full extended context will fit on that machine.

Should I use Ollama, LM Studio or MLX?

Choose Ollama for the command examples here, LM Studio when you want a GUI and load estimator, or MLX-LM for a scripted Mac text workflow. Compare identical artifacts where possible and verify architecture support before switching.

Can a local model replace GPT-6 or Claude Opus 5.5?

It may satisfy a particular workflow, but there is no supported blanket equivalence in this guide. Compare the current hosted model with your exact local quantization on the same tasks, including retries and review effort.

Does running a model locally make the whole application private?

Check every connection: inference, retrieval, tools, telemetry and fallback services. Test offline when that is a requirement. A locally loaded model is one component of the application, so review the full route taken by a prompt.

Sources

• Qwen3.5-4B card, Ollama package
• Qwen3.5-9B card, Ollama package
• Qwen3.8-27B card, Ollama package
• Qwen3-Coder-Next card
• Gemma 4 12B card
• Gemma 4 31B and family card, memory requirements, Ollama library
• gpt-oss-20b card, gpt-oss-120b card, publisher README, Ollama packages
• DeepSeek V4.1 Flash card
• GLM-5.3 card
• Kimi K3 card
• Mistral Small 4 card, Medium 3.5 card
• Llama 4 card and license
• Ollama quickstart, FAQ, chat API
• LM Studio application docs, load command, server startup
• llama.cpp README, server reference, quantization guide
• MLX framework, MLX-LM README, community conversion card
• vLLM README, quickstart, serving CLI
• OpenAI current model guidance, Anthropic Opus, Gemini 3.8 Flash

Last reviewed and updated: September 2026.