OpenCode on a Remote Local-Model Host
Run the OpenCode coding agent on one machine while a second machine serves the model, using Ollama or llama.cpp, with Windows and Linux mixed freely.
For people who want (a) a Mixture-of-Experts model larger than their VRAM, or (b) a dense model at 128k+ context.
Author: Jason Wolff · Written 2026-10-02 · Based on a working three-machine lab: a Windows Ollama host, a Linux llama.cpp host and a Linux harness (section 14).
1. What you are building
Two machines, one job each. Machine A (host) owns the GPU and RAM and serves an OpenAI-compatible HTTP endpoint. Machine B (harness) runs the OpenCode terminal agent and edits files, runs tests and so on locally. Only text crosses the network.
Ollama
Least setup: one installer, a tray app on Windows, automatic GPU/CPU placement, and models pulled by name. The context window is baked into a named model with a Modelfile (section 6). Less control over exactly where each tensor goes.
llama.cpp (llama-server)
Full control: per-tensor placement (the key to big MoE models), KV-cache quantization, exact context. More flags to learn. A router mode with a preset file serves many models.
Recommendation & Notes
Recommendation. Dense model that fits in VRAM: either engine works. MoE bigger than VRAM: either engine works on current versions. Measured on a 16 GB card with a 25 GB MoE, Ollama 0.34.4's automatic placement matched a hand-tuned llama.cpp --n-cpu-moe setup within a few percent (section 5). Choose llama.cpp when you want the placement explicit and fixed. Choose Ollama for less setup.
2. The one rule: match the numbers
Most "it works but gets dumb after a while" problems are a context mismatch. There are three numbers, and they must agree.
| Number | Where it lives | What it controls |
|---|---|---|
| **Served context** | Host: Ollama `num_ctx` / llama.cpp `--ctx-size` | How many tokens the server really allocates and attends to. Beyond this, it truncates or errors. |
| **Model maximum** | Inside the GGUF (`context_length`) | The ceiling you can serve without RoPE scaling. Do not exceed it casually. |
| **Harness limit** | Harness: `limit.context` and `limit.output` in `opencode.jsonc` | When OpenCode decides to compact (summarize) history. It trusts this number and never asks the server. |
Why this bites each engine differently
Warning
Ollama: the OpenAI-compatible /v1 endpoint that OpenCode uses cannot set num_ctx per request. Without a server-side setting, Ollama falls back to its default (in 0.34.4: "4k/32k/256k based on VRAM", per ollama serve --help), which may be far below what OpenCode assumes, and the server then truncates. OpenCode's own system prompt plus tool schemas fill the first several thousand tokens: 5,933 to 6,231 tokens were measured with OpenCode 1.18.32 on 2026-10-02, and more with extra tools or MCP servers. Truncation therefore destroys the agent immediately. The fix is to bake num_ctx into a named model with a Modelfile (section 6). llama.cpp: --ctx-size is the total KV budget, shared across parallel slots. With --parallel 4 each request gets about a quarter of it (unless unified KV is enabled). For a single agent, use --parallel 1 so one request gets the entire window.
Habit that prevents drift: put the size in the model name
Name the served model for its window, for example qwen3.8-27b-64k. The same string goes into opencode.jsonc as the model id. If someone later changes the host to 32k, the name is now visibly wrong, instead of a hidden mismatch. For llama.cpp router mode the section name in the preset file is the name clients send.
Section 14 shows every model on the working hosts, the context each one is served at, and the matching OpenCode entry. Use it as a template.
3. Sizing the memory
Total memory = weights + KV cache + compute buffers. Weights are the GGUF file size, roughly. The KV cache is what grows with context:
KV bytes = 2 x layers x kv_heads x head_dim x context_tokens x bytes_per_elementRead layers, kv_heads (attention.head_count_kv) and context_length from the model metadata. On Ollama, ollama show --verbose <model> or the model_info field of /api/show has them. On llama.cpp, the server log prints them at load.
Worked example (illustrative numbers, not a specific model)
A dense model with 64 layers, 8 KV heads, head_dim 128, at 131072 tokens:
| KV cache type | Bytes/element | KV size at 128k |
|---|---|---|
| f16 (default) | 2 | about 32 GiB |
| q8_0 | about 1 | about 17 GiB |
| q4_0 | about 0.5 | about 9 GiB |
Recommendation & Notes
Caveats. Hybrid architectures (some layers use linear attention or state-space blocks instead of full attention) need far less KV than this formula says. Their effective layers is only the full-attention layers. Always confirm with a real load and watch actual VRAM. The formula is for planning, not for promises.
Recommendation & Notes
Measured: hybrid 27B, q8_0 KV (RTX 5060 Ti 16 GB, Ollama 0.34.4, 2026-09-28 and 2026-10-02). The GGUF reports 64 layers, 4 KV heads, head_dim 256, and full_attention_interval = 4, so only 16 layers hold KV. The formula with 16 layers predicts 2.1 GiB at 64k and 8.5 GiB at 256k. The load logs showed 2176 MiB (2040 GPU + 136 CPU) at 65536 and 8704 MiB (4352 GPU + 4352 CPU) at 262144, an exact match. Using all 64 layers would have predicted four times as much.
4. Dense model, 128k or larger context
Dense models read all weights every token, so the weights should live entirely on GPU, or generation becomes slow. The only variable you can trade is the KV cache.
- **Choose the quant so weights leave room.** Weights + KV + about 1 GB overhead must fit in total VRAM. If the weights alone take 95% of VRAM, a long context cannot fit.
- **Quantize the KV cache** (q8_0 first, q4_0 only if needed). K is more quality-sensitive than V. q8_0 is close to lossless. q4_0 can hurt recall at long context, so test it.
- **Flash attention must be on** for a quantized V cache, and it also cuts compute memory.
- **Multiple GPUs:** `--split-mode layer` divides layers across cards and works over plain PCIe. Each token still passes through the cards in turn, so expect roughly single-card speed or a bit less, not a speed-up. The gain is capacity.
- **Prefer to find the limit by test.** Load at your target context, then run a long prompt (section 9). If you hit out-of-memory, reduce context before quantizing weights harder.
Recommendation & Notes
Measured example: a dense model that almost fits Hardware: RTX 5060 Ti 16 GB (16311 MiB, 1118 MiB used by the desktop at idle), Ryzen 5 7600X (6 cores), 2×16 GB DDR5 running at 4800, Windows 11, Ollama 0.34.4. Model: 27B hybrid dense, IQ3_S, 12 GB GGUF, with a vision projector. KV q8_0, flash attention on.
| Served context | Layers on GPU (auto fit) | Prompt tok/s | Generation tok/s | Date |
|---|---|---|---|---|
| 65536 | 61/65, `ollama ps`: 8%/92% CPU/GPU, VRAM 15609/16311 MiB | about 600–770 | about 9–10 at 40–50k tokens in context; about 16 at 4–6k | 2026-09-29 to 2026-10-02 |
| 262144 | 33/65 | about 300–450 | 3.6 falling to 1.3 as context grew | 2026-09-28 |
Warning
A fast model with a lot of context is not always a usable one. Many models are trained at 32k or 64k and stretched past that with RoPE scaling. Quality at the far end of the window can be poor even when it loads. Treat "trained maximum" as the honest ceiling, and read the model card.
5. MoE model larger than VRAM
A Mixture-of-Experts model has many expert feed-forward blocks but activates only a few per token. That asymmetry is the trick:
- **Attention, shared layers and the KV cache** are small, touched every token, and bandwidth-hungry. *Keep these on the GPU.*
- **Expert tensors** are most of the file size but only a few are read per token. *These can live in system RAM* and be computed by the CPU with a modest speed cost.
llama.cpp: how
The command used for the measured example below, on the Windows host. The 16 GB card holds the attention of all 54 layers, and the experts of the first 26 layers run from system RAM:
Windows
bash
# --n-gpu-layers all puts every layer on the GPU first,
# then --n-cpu-moe 26 sends the experts of the first 26 layers back to the CPU.
# --threads = physical cores (6 on a Ryzen 5 7600X), not hyperthreads.
# --fit off: fail loudly instead of re-arranging placement on its own.
.\llama-server.exe `
--model <path to the nemotron-3.5-lightning 30B Q4_K_M .gguf> `
--alias nemotron-3.5-lightning-30b-64k `
--ctx-size 65536 --parallel 1 --jinja --flash-attn on `
--cache-type-k q8_0 --cache-type-v q8_0 `
--n-gpu-layers all --n-cpu-moe 26 --fit off --threads 6 `
--host 0.0.0.0 --port 8080On Linux, use the same flags with \ line continuations. Find the right --n-cpu-moe with the tuning loop below. 28 left 1.8 GB of VRAM free on this card and 26 left 0.9 GB.
Warning
Using the llama-server.exe that ships inside Ollama. It is in %LOCALAPPDATA%\Programs\Ollama\lib\ollama, but run on its own it prints no usable GPU found and silently runs on the CPU. The CUDA backend is in a subfolder that Ollama adds when it launches the server. Set this first (PowerShell; cuda_v13 was the one that worked on an RTX 5060 Ti):
powershell
$d = "$env:LOCALAPPDATA\Programs\Ollama\lib\ollama"
$env:PATH = "$d\cuda_v13;$d;" + $env:PATH
$env:GGML_BACKEND_PATH = "$d\cuda_v13\ggml-cuda.dll"
& "$d\llama-server.exe" --list-devices # must list CUDA0A standalone llama.cpp CUDA release does not need this.
--cpu-moe is shorthand for "all experts on CPU".
Equivalent regex form, which gives finer control: -ot "blk\.(\d+)\.ffn_.*_exps\.=CPU" keeps matching expert tensors in system RAM. You can write different patterns per GPU on multi-card machines.
Tuning loop
- Start with `--n-cpu-moe` set to **all layers** (every expert on CPU). It will certainly load. Note VRAM used.
- Lower the number step by step. Each step moves one layer's experts onto the GPU, which is faster. Stop when VRAM is about 1–1.5 GB from full. Leave headroom, since a long prompt allocates compute buffers late.
- Set `--threads` to your physical core count. More is not better here. The CPU side is limited by RAM bandwidth, so dual-channel or faster memory matters more than core count.
- Keep the model file on fast storage and let it be memory-mapped. System RAM must hold the expert weights that stay on CPU, plus the OS. Under-provisioning RAM causes paging, and speed collapses.
Warning
Offload trades heat and speed for capacity. The CPU runs at sustained full load for every generated token. If your CPU cannot take sustained load (small cooler, compact case), stay fully on GPU and pick a smaller model or quant.
Recommendation & Notes
Recent llama.cpp builds have an automatic fit mode (--fit), on by default, which adjusts unset options such as --n-gpu-layers to leave a free-VRAM margin (--fit-target, default 1024 MiB per device). It is a good first try. It is also how a model silently ends up partly on CPU: in the measured example in section 4, the load log showed a free-memory target of 1936 MiB, and fit put 61 of 65 layers on the GPU to meet it. Nothing reported this except the load log and ollama ps. If you want deterministic behavior, as in the preset file in section 7, use --fit off and set placement yourself, so a model that no longer fits fails loudly instead of quietly spilling to RAM.
Ollama and MoE
Earlier Ollama releases were reported to split by layer only (PARAMETER num_gpu N), which cannot do the "attention on GPU, experts on CPU" split. Ollama 0.34.4 on Windows runs a bundled llama-server with fit on, and its fit does make that split. The load log shows filling dense-only layers, then converting dense-only layers to full layers ... with overflow to ... system memory, ending in a line like 54 layers (30 overflowing): every layer's attention on the GPU, with the experts of 30 layers in system RAM. ollama ps shows the result in the PROCESSOR column, for example 54%/46% CPU/GPU. On Windows its log also shows disabling mmap ... reason=windows_cuda, so the CPU-side weights are loaded into RAM rather than memory-mapped.
Recommendation & Notes
Measured example: a 25 GB MoE on a 16 GB card, Ollama vs llama.cpp Hardware: RTX 5060 Ti 16 GB, Ryzen 5 7600X (6 cores), 2×16 GB DDR5 running at 4800, Windows 11. Model: nemotron-3.5-lightning:30b (nemotron_h_moe, 33B total, 128 experts, 54 layers, Q4_K_M, 23.7 GB GGUF). Measured 2026-10-02.
| Ollama 0.34.4, automatic fit | llama-server (bundled with Ollama), explicit | |
|---|---|---|
| Placement | auto: 54/54 layers on GPU, experts of 30 layers on CPU | `--n-gpu-layers all --n-cpu-moe 26 --fit off --threads 6` |
| Context, KV | 262144 (host default at the time), q8_0 | 65536, q8_0 |
| VRAM used (of 16311 MiB) | 15240 MiB peak | 15374 MiB after load, 15921 MiB peak |
| Generation | 43–57 tok/s | 49–53 tok/s |
| Prompt, 16k tokens | 1255 tok/s | 1293 tok/s |
| OpenCode task (`opencode run`: write and run a script, 3 model calls) | 47 s, first prompt 5,933 tokens at 454 tok/s | 52 s, first prompt 6,231 tokens at 161 tok/s (first request after load) |
| CPU load avg/max, GPU temp max | 42/73%, 53 °C | 64/94%, 65 °C |
| Free RAM, lowest | 4.1 GB | 0.4 GB |
6. Host: Ollama on Windows
Runs on: Machine A (Windows)
6.1 Make it listen on the LAN
By default Ollama binds to localhost only. Set the environment variables, then fully quit Ollama (tray icon, Quit) and start it again, because it only reads them at start. User scope (plain setx) is enough for the tray app and is what the measured setup uses. Use setx /M from an admin shell for machine scope if Ollama runs as another user or a service.
powershell
# PowerShell. setx writes persistent user variables.
setx OLLAMA_HOST "0.0.0.0:11434"
setx OLLAMA_FLASH_ATTENTION "1"
setx OLLAMA_KV_CACHE_TYPE "q8_0" # requires flash attention
setx OLLAMA_CONTEXT_LENGTH "65536" # default for models without num_ctx (optional)
setx OLLAMA_KEEP_ALIVE "-1" # recommended: keep the model loaded; the default is 5mWarning
The Ollama app's own Settings override OLLAMA_CONTEXT_LENGTH. The Windows app has a context-length setting, stored as context_length in %LOCALAPPDATA%\Ollama\db.sqlite, and it passes that value to the server. On 2026-10-02 the user variable read 65536 while the app setting read 131072, and after a restart the server used 131072. Set the context length in the app's Settings, or make sure both agree. The app also has an "expose to network" setting (expose), which does the same job as OLLAMA_HOST=0.0.0.0.
Confirm what the server actually read: the first line of server.log after a restart is a server config line listing every OLLAMA_* value in effect.
6.2 Open the firewall for the harness only
New-NetFirewallRule -DisplayName "Ollama from harness" -Direction Inbound `
-Protocol TCP -LocalPort 11434 -RemoteAddress <HARNESS_IP> `
-Profile Private -Action AllowSet -Profile to whatever Get-NetConnectionProfile reports for the LAN adapter. A domain-joined PC is usually Domain, and a rule for Private alone will not apply there. Avoid Public.
Caution
Ollama has no authentication. Anyone who can reach port 11434 can run models and pull or delete them. Restrict by source IP as above, keep it off public networks, and never port-forward it.
6.3 Bake the context into a named model (the critical step)
Create a Modelfile. Use a base you have already pulled, or a GGUF from Hugging Face.
bash
# Modelfile (save as C:\ollama\qwen3.8-27b-64k.Modelfile)
FROM hf.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF:IQ3_S
PARAMETER num_ctx 65536
# Optional, if you want sampling defaults baked in too:
# PARAMETER temperature 0.6bash
ollama create qwen3.8-27b-64k -f C:\ollama\qwen3.8-27b-64k.Modelfile
ollama show qwen3.8-27b-64k # confirm: Parameters -> num_ctx 65536Warning
Template pitfall. A GGUF imported from Hugging Face may carry no chat template, or you may have overridden it. OpenCode needs working tool calls. After creating the model, check ollama show for the tools capability, and run a real tool-calling test. If the model replies with raw JSON text instead of invoking tools, the template is the problem, not OpenCode. Observed on Ollama 0.34.4 (2026-10-02): a Hugging Face GGUF model created with TEMPLATE {{ .Prompt }} still returned structured tool_calls (finish_reason: tool_calls) through /v1/chat/completions. Do not rely on that on other versions. Run the test.
Setting OLLAMA_CONTEXT_LENGTH as a global default also works (0.34.4 checked), but the per-model Modelfile is more explicit, and the model name documents the window.
Pull the model first if you use FROM hf.co/.... Disk space needed is the size of the GGUF, and Windows keeps it under %USERPROFILE%\.ollama\models.
7. Host: llama.cpp
Binaries: build from source, or download a release for your platform (CUDA build for NVIDIA, Vulkan for broad GPU support). Check that llama-server --version runs and that the log shows your GPU(s) on load.
Recommendation & Notes
What was checked. Every flag in this section exists in llama-server --help of the binary bundled with Ollama 0.34.4 for Windows (reports 0.4.1-dev, commit 161755f29), checked 2026-10-02. The Linux router setup runs on a separate GPU-only host. The standalone Windows commands in 7.1 were not run.
7.1 Single model, direct command
Windows
powershell
.\llama-server.exe --model D:\models\Qwen3.8-27B-GSQ-RCO-IQ3_S.gguf --alias qwen3.8-27b-64k `
--ctx-size 65536 --parallel 1 --jinja --flash-attn on `
--n-gpu-layers all --host 0.0.0.0 --port 8080Linux
bash
./llama-server --model /opt/ai/models/granite-4.0-h-tiny-Q4_K_M.gguf --alias granite-4.0-h-tiny \
--ctx-size 65536 --parallel 1 --jinja --flash-attn on \
--n-gpu-layers all --host 0.0.0.0 --port 8080What each important flag does and why:
| Flag | Purpose |
|---|---|
| `--ctx-size` | Total context to allocate. This is the number the harness must match. |
| `--parallel 1` | One slot, so the whole window serves one agent request. The default can split the context. |
| `--jinja` | Use the model's own chat template, required for correct tool-call formatting. Default on in recent builds. Passing it is harmless. |
| `--flash-attn on` | Lower memory, and required for quantized V cache. |
| `--n-gpu-layers all` | Everything on GPU. For MoE, combine with `--n-cpu-moe` (section 5). |
| `--alias` | The model name clients send. Match it to `opencode.jsonc`. |
| `--split-mode layer / none`, `--main-gpu` | Multi-GPU placement. Use `none` plus `--main-gpu` to pin to one card when the model fits, which is faster than splitting. |
| `--fit off` | Optional. Fail on overflow instead of auto-moving work to RAM, so mistakes are loud. |
7.2 Many models: router mode with a preset file
This is the real preset file from the Linux llama.cpp host (2× GTX 1650 4 GB, llama.cpp 0.4.0-dev build 10950, read 2026-10-02). The comments are trimmed slightly. One llama-server process holds one model at a time and swaps when a client asks for a different name. Each model's settings live in the INI file, so adding or changing a model is an edit and a restart, not a new service.
bash
# /etc/ai-lab/llama-server.env (read by the systemd unit in 7.3)
LLAMA_ARGS=--models-preset /etc/ai-lab/models.ini --models-max 1 --host 0.0.0.0 --port 8080
# Make device N match nvidia-smi GPU N, so main-gpu below means the card you expect.
CUDA_DEVICE_ORDER=PCI_BUS_ID
CUDA_CACHE_PATH=/var/cache/llama-server
LLAMA_CACHE=/var/cache/llama-serverini
; /etc/ai-lab/models.ini -- llama-server router presets.
; One section per model; the section name is the model name clients send.
; Keys are llama-server's long flags without the dashes.
; Edit, then: sudo systemctl restart llama-server
;
; GPU-only. Every model here was fit-tested at exactly this ctx-size:
; all layers on the GPUs, no weights in system RAM. --models-max 1 means each
; model has both cards to itself. n-gpu-layers = all with fit = off means a
; model that no longer fits fails to load instead of quietly moving to RAM.
;
; OpenCode's own system prompt + tool schema is ~6k tokens (measured with
; OpenCode 1.18.32, more with MCP tools), so an 8k window is gone almost at once. Client configs must match these sizes.
;
; Names have no colons: the router reads ":suffix" as a Hugging Face quant tag
; and rewrites it, so "qwen2.5:3b-instruct" would 404.
version = 1
[*]
flash-attn = on
jinja = true
threads = 4
parallel = 1
n-gpu-layers = all
fit = off
; default placement: one card (faster than splitting when the model fits);
; the last card, since GPU 0 also drives the console
split-mode = none
main-gpu = 1
; ---- Default. Loaded at boot. The only MoE that fits the cards; 68 tok/s.
[granite-4.0-h-tiny]
model = /opt/ai/models/granite-4.0-h-tiny-Q4_K_M.gguf
ctx-size = 65536
split-mode = layer
load-on-startup = true
; ---- One card
; thinking model: needs a generous max_tokens. 64k: 2.8GB.
[lfm2.5-2.6b]
model = /opt/ai/models/LiquidAI_LFM2.5-2.6B-Q4_K_M.gguf
ctx-size = 65536
; 131072: fit-tested f16/q8_0 PASS (~2.9/3.3GB per card split)
[nemotron-3-nano-4b]
model = /opt/ai/models/NVIDIA-Nemotron3-Nano-4B-Q4_K_M.gguf
ctx-size = 131072
split-mode = layer
; 32k is the model's own trained maximum.
[qwen2.5-3b-instruct]
model = /opt/ai/models/qwen2.5-3b-instruct-Q4_K_M.gguf
ctx-size = 32768
; Writes tool calls as JSON text rather than real tool_calls,
; so agents cannot use it for tools.
[qwen2.5-coder-1.5b]
model = /opt/ai/models/qwen2.5-coder-1.5b-instruct-Q4_K_M.gguf
ctx-size = 32768
; Image description. The GGUF has no chat template; without Ollama's
; Question/Answer format every image gets an empty reply. Clients should send
; stop: ["<|endoftext|>", "Question:"]. 2048 is the model's trained context.
[moondream]
model = /opt/ai/models/moondream2-text.gguf
mmproj = /opt/ai/models/moondream2-mmproj.gguf
chat-template-file = /opt/ai/templates/moondream-ollama.jinja
ctx-size = 2048
; ---- Both cards (did not fit on one card at this context)
[granite-4.2-3b]
model = /opt/ai/models/granite-4.2-3b-Q4_K_M.gguf
ctx-size = 65536
cache-type-k = q8_0
cache-type-v = q8_0
split-mode = layer
; thinking model
[minicpm5-2b]
model = /opt/ai/models/MiniCPM5-2B-Q8_0.gguf
ctx-size = 32768
split-mode = layer
; thinking model
[spark-x2.5-4b]
model = /opt/ai/models/Spark-X2.5-4B-Q4_K_M.gguf
ctx-size = 32768
split-mode = layer
; ---- Both cards, 8-9B (~19 tok/s: each token passes through both in turn)
; thinking model. 32k fits; 64k runs out of memory.
[qwen3.5-9b]
model = /opt/ai/models/Qwen3.5-9B-Q4_K_M.gguf
ctx-size = 32768
split-mode = layer
; 16k uses 3.6GB of each card's 3.7GB; 32k runs out of memory.
[ministral-3-8b]
model = /opt/ai/models/Ministral-3-8B-Instruct-2512-Q4_K_M.gguf
ctx-size = 16384
split-mode = layer
; thinking model. 8k only: 16k and 32k run out of memory. Too small for
; OpenCode (its prompt alone is ~6k, plus history); fine for plain API calls.
[granite-4.2-8b]
model = /opt/ai/models/granite-4.2-8b-Q4_K_M.gguf
ctx-size = 8192
split-mode = layer
; 128k via q4_0 KV (f16/q8_0 KV ran out of memory on 2x4GB).
; 2026-09-24: loads and responds, ~3.6/4GB per card. Not fit-tested; watch quality at long context.
[qwen2.5-coder-7b]
model = /opt/ai/models/qwen2.5-coder-7b-instruct-Q4_K_M.gguf
ctx-size = 131072
split-mode = layer
cache-type-k = q4_0
cache-type-v = q4_0The live file's comments say "~12k" for OpenCode's prompt. That figure was corrected here to the measured value. A section only needs the keys that differ from [*]. The original qwen2.5-coder-7b section repeats the defaults, which does no harm.
Recommendation & Notes
Lessons baked into that file:
- **Do not use colons in section names.** The router reads `:suffix` as a Hugging Face quant tag and rewrites it, so a name like `qwen2.5:3b` returns 404. Use `qwen2.5-3b`.
- **Record the fit-tested context in a comment** beside each model, and the client configs must match it.
- **Try one card first, then split.** `split-mode = none` with `main-gpu` is faster. Use `split-mode = layer` only for models or contexts that do not fit on one card.
- **Quantize KV before giving up on context.** `granite-4.2-3b` reaches 64k with q8_0 KV. `qwen2.5-coder-7b` reaches 128k only with q4_0 KV, so its long-context quality is the thing to test.
- **Set context large enough for the agent.** OpenCode's system prompt and tool schema take about 6k tokens (measured, OpenCode 1.18.32) and more with MCP servers, before you type anything. An 8k window fails almost immediately. Treat 32k as a practical floor and 64k+ as comfortable.
- **Some models cannot do tools.** Models that write tool calls as plain JSON text rather than real `tool_calls` are fine for plain API use but useless for an agent. Record that in the comment too.
7.3 Run it as a service
Linux (systemd)
bash
# /etc/systemd/system/llama-server.service (the real unit from the Linux host)
[Unit]
Description=llama.cpp server on the NVIDIA GPUs, LAN-facing
After=network-online.target nvidia-persistenced.service
Wants=network-online.target nvidia-persistenced.service
[Service]
Type=exec
# Unprivileged system user: the process that parses network input owns nothing
# and can write nothing.
User=llama
Group=llama
# /dev/nvidia* are 0666 on Ubuntu, so no video/render group is needed.
# CUDA's JIT cache (CUDA_CACHE_PATH in the env file); the user has no home.
CacheDirectory=llama-server
EnvironmentFile=/etc/ai-lab/llama-server.env
ExecStart=/opt/ai/llama.cpp/build/bin/llama-server $LLAMA_ARGS
Restart=on-failure
RestartSec=5
# A cold page cache plus two cards to fill makes the first load slow.
TimeoutStartSec=300
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
RestrictSUIDSGID=yes
LockPersonality=yes
ReadOnlyPaths=/opt/ai
# No PrivateDevices (CUDA needs /dev/nvidia*) and no MemoryDenyWriteExecute
# (the CUDA runtime maps W+X pages).
[Install]
WantedBy=multi-user.targetbash
sudo systemctl daemon-reload && sudo systemctl enable --now llama-server
journalctl -u llama-server -f # watch the load, look for n_ctx and GPU buffer sizesWatch the first load of each model: the log prints n_ctx, n_ctx_seq and the GPU buffer sizes. With fit = off, a model that does not fit fails here, not later under load.
Windows Use Task Scheduler (run at startup, highest privileges, restart on failure) or a service wrapper such as NSSM pointing at llama-server.exe with the same arguments. Add a firewall rule like the one in section 6.2 for port 8080.
8. Harness: OpenCode on Linux
Machine B: Linux Install OpenCode following its official instructions, then edit ~/.config/opencode/opencode.jsonc for one user, or /etc/opencode/opencode.jsonc for every user on the machine. /etc/opencode is OpenCode's built-in system-wide config folder (C:\ProgramData\opencode on Windows), and the working harness uses it with no environment variable. Each backend is an OpenAI-compatible provider, and each model inside it carries the limit block that must match the host.
json
{
"$schema": "https://opencode.ai/config.json",
"model": "local/qwen3.8-27b-64k", // provider-id/model-id: default model
"provider": {
// ---- Option 1: Ollama on the Windows host -----------------------
"local": {
"npm": "@ai-sdk/openai-compatible",
"name": "Workstation Ollama",
"options": { "baseURL": "http://<HOST_IP>:11434/v1" },
"models": {
"qwen3.8-27b-64k": { // EXACT name from `ollama create`
"name": "Qwen3.8 27B IQ3_S 64k",
"reasoning": true, // true for thinking models
"attachment": false, // true only if the model takes images
"limit": { "context": 65536, // == Modelfile num_ctx
"output": 8192 } // max tokens per reply
}
}
},
// ---- Option 2: llama-server router on the Linux GPU host --------
"llamacpp-108": {
"npm": "@ai-sdk/openai-compatible",
"name": "llama.cpp router",
"options": { "baseURL": "http://<HOST_IP>:8080/v1" },
"models": {
"nemotron-3-nano-4b": { // EXACT preset section name / --alias
"name": "Nemotron 3 Nano 4B (128k)",
"reasoning": false,
"attachment": false,
"limit": { "context": 131072, // == ctx-size in models.ini
"output": 8192 }
}
}
}
}
}The "npm" line names the AI SDK package for the provider. It is optional for an OpenAI-compatible endpoint: OpenCode 1.18.32 with a provider that had only name, options.baseURL and models reached Ollama's /v1/chat/completions and answered (tested 2026-10-02). One of the working harness configs has no npm field either. Keeping it does no harm.
How the match works, field by field:
| OpenCode field | Must equal |
|---|---|
| model key (`"qwen3.8-27b-64k"`) | Ollama model name, or llama.cpp preset section / `--alias`, character for character. Mismatch gives 404. |
| `limit.context` | Ollama `num_ctx`, or llama.cpp `ctx-size` (with `parallel 1`). |
| `limit.output` | Your choice, but smaller than the context minus the agent's prompt overhead (about 6k measured, more with MCP tools). Thinking models need a generous value, since reasoning tokens count toward it. |
| `baseURL` | `http://<HOST_IP>:<port>/v1` including the `/v1`. Use an IP or a stable DNS name or DHCP reservation, because a changing address breaks it silently. |
Compaction settings tied to context
"compaction": {
"auto": true, // summarize history automatically near the limit
"prune": true, // drop stale tool output first
"reserved": 12000, // tokens kept free for the next reply and tool results
"preserve_recent_tokens": 12000 // newest history kept verbatim through a compaction
}Key names checked against OpenCode 1.18.32 (2026-10-02). That version also accepts keep.tokens and buffer as aliases for preserve_recent_tokens and reserved. These are what make a small window survivable. With limit.context correct, OpenCode compacts before the server would truncate. With it wrong, compaction fires too late, and the failure looks like a model quality problem. Scale reserved up with limit.output, and keep it at least as large as one big tool result you expect.
9. Verify the match
Do all of these. Each catches a different failure. Replace <HOST_IP>. On Windows PowerShell, call curl.exe, because plain curl is an alias for a different command. Windows PowerShell 5.1 also strips the inner quotes from -d '{...}' JSON, so use PowerShell 7 or Invoke-RestMethod there.
9.1 Server-side: what is configured
bash
# Ollama: shows num_ctx baked into the model, and the model's trained maximum
curl http://<HOST_IP>:11434/api/show -d '{"model":"qwen3.8-27b-64k"}'
# look for: PARAMETER num_ctx 65536 and *.context_length (trained max)
# Ollama: what is loaded RIGHT NOW, the CONTEXT it was loaded with, and the CPU/GPU split
ollama ps
# llama.cpp: running config; router mode lists models, per-model n_ctx shows once loaded
curl http://<HOST_IP>:8080/props
curl http://<HOST_IP>:8080/v1/modelsCompare each against limit.context in opencode.jsonc. Also run ollama ps after a real OpenCode session: its CONTEXT column shows what the server actually allocated, which is the truth, regardless of the Modelfile.
9.2 End to end: prove the full window is used
A needle-in-a-haystack test. Put a unique fact at the start, pad toward your limit, ask for the fact back:
python
python3 - <<'EOF'
import json, urllib.request
HOST="http://<HOST_IP>:11434/v1/chat/completions"; MODEL="qwen3.8-27b-64k"
FILL = int(65536*0.8) # ~80% of limit.context, rough tokens
filler = ("The quick brown fox jumps over the lazy dog. " * (FILL//10))
msg = "SECRET CODE: PINEAPPLE-4471.\n" + filler + "\nWhat was the secret code at the start?"
req = urllib.request.Request(HOST, json.dumps({"model":MODEL,"messages":[{"role":"user","content":msg}],"max_tokens":3000}).encode(),
{"Content-Type":"application/json"})
r = json.load(urllib.request.urlopen(req, timeout=900))
print(r["usage"]); print(r["choices"][0]["message"].get("content"))
EOFRun from the Linux harness against the Windows Ollama host on 2026-10-02 (Python 3.10, model qwen3.8-27b-64k): prompt_tokens 52,495, answer PINEAPPLE-4471, 95 s in total over the LAN. max_tokens is generous so a thinking model's reasoning cannot use up the reply budget and leave an empty answer. The same test from PowerShell, adapted from the run on 2026-10-02 (section 4 has the result):
Windows
powershell
$fill = 'The quick brown fox jumps over the lazy dog. ' * [int](65536*0.8/10)
$msg = "SECRET CODE: PINEAPPLE-4471.`n" + $fill + "`nWhat was the secret code at the start? Reply with the code only."
$body = @{ model='qwen3.8-27b-64k'; max_tokens=3000; messages=@(@{role='user';content=$msg}) } | ConvertTo-Json -Depth 5
$r = Invoke-RestMethod -Uri http://<HOST_IP>:11434/v1/chat/completions -Method Post `
-ContentType 'application/json' -Body $body -TimeoutSec 900
$r.usage; $r.choices[0].message.contentRead usage.prompt_tokens. If it is far below what you sent, or the answer is wrong, the server truncated. That means num_ctx is not what you think. If it fails with an out-of-memory error at a context you believed fit, reduce context or quantize KV (sections 3–4).
9.3 Watch the logs
- Ollama on Windows: `%LOCALAPPDATA%\Ollama\server.log`, rotated to `server-1.log`, `server-2.log` and so on. On 0.34.x the log is `llama-server` output: check `n_ctx` and `offloaded N/M layers to GPU` at load, and `truncated = 0` on each `slot release` line. Older versions logged `truncating input` instead.
- llama.cpp: the load log prints `n_ctx` and `n_ctx_seq`/slot size, which is the per-request window, and the KV buffer size per device.
9.4 Check speed on the first long reply
For MoE offload, note prompt-processing and generation tokens/second. If generation is a fraction of what you expected, check that experts are on the CPU, attention is on the GPU, and RAM is not paging.
10. Optional agent tuning
These come from the working setup. They are not needed to connect, but they improve unattended agent runs on small local models.
10.1 A second, smaller model for housekeeping
small_model is used for cheap tasks (titles, summaries). Point it at a small model that fits entirely in GPU memory, possibly a different machine or provider. It keeps the big model free for real work.
"small_model": "llamacpp-108/nemotron-3-nano-4b"10.2 A local executor subagent
A subagent on a small, fast model for edit-and-test loops, with the big model as planner:
"agent": {
"local-exec": {
"description": "Local executor. Runs edits + tests offline.",
"mode": "subagent",
"model": "llamacpp-108/nemotron-3-nano-4b",
"prompt": "You are a local executor. Read, edit small scopes, then run python3 -m py_compile and pytest until green. No network. Paste tool output, do not claim would-pass. Keep working until the task is complete or you are genuinely blocked."
},
"build": { "steps": 2000500 }, // effectively unlimited agent steps
"plan": { "steps": 500 },
"general": { "steps": 500 },
"explore": { "steps": 200 }
},
"subagent_depth": 1"No claim of would-pass" matters: small models tend to say tests pass without running them. Forcing pasted tool output makes that visible.
10.3 Autonomy switches (understand before copying)
"permission": { "doom_loop": "allow" },
"experimental": { "continue_loop_on_deny": true }Warning
doom_loop: "allow" turns off the confirmation OpenCode normally asks for when the agent repeats the same failing action, and unlimited steps removes the other brake. On a local model this costs only electricity, but combined they let a stuck agent spin for hours. Use them for supervised overnight runs on throwaway checkouts, not on anything where tool calls can cause harm.
10.4 A behavior-shaping system prompt
Local models drift and over-explore more than hosted ones. A short, rule-based system prompt helps. Adapt it. Remove anything that describes tools you do not have:
"system": "You are a concise, direct assistant. Follow these rules strictly:\n\n1. ANSWER THE QUESTION ASKED. Do not go on tangents or explore unrelated topics.\n2. Answer general knowledge and factual questions from your own training knowledge. Do not use web search for facts you should already know.\n3. When you genuinely need external information, use the built-in `websearch` tool, NOT curl or bash network commands. For a specific URL use the `webfetch` tool.\n4. If you do not know something, say so clearly. Never invent numbers, facts, or sources. If a fact depends on a variable (species, model, version), ask or state the dependency.\n5. Do not wander into files, logs, or history unrelated to the task. Reading and analyzing whatever the task requires is expected.\n6. Keep prose short. This applies to your replies, not to the work: take as many tool calls and steps as the task needs.\n7. Do not claim services or units exist without verifying them first.\n8. When diagnosing, identify the root cause, not symptoms.\n9. If you make a mistake, acknowledge it directly and correct course.\n10. Network requests: do not hammer a failing URL; one attempt, note the failure, move on. For edits, tests and builds, a failure is information: read the error, fix the cause, try again.\n11. PERSIST. Work autonomously until the task is fully done and verified. Do not stop to ask permission or hand back partial work unless genuinely blocked on something only the user can provide.\n\nBe helpful, accurate, and concise."Rule 5 is worth a note. An earlier, stricter version ("do not analyze files unless asked") made agents refuse to read code they needed. The reworded version fixed that.
11. Windows and Linux gotchas
| Area | Detail |
|---|---|
| Env vars on Windows | Ollama reads them at process start. Quit it from the tray and relaunch after `setx`. A new PowerShell window does not restart the background app. |
| Firewall profile | The rule must apply to the network profile the machine is on. A LAN set to "Public" or "Domain" ignores a "Private"-only rule, and connections time out with no error. Check `Get-NetConnectionProfile`. |
| Test reachability first | From B: `curl http://<HOST_IP>:11434/api/tags` (Ollama) or `/v1/models` (llama.cpp). If this fails, fix networking before touching OpenCode. A *timeout* usually means firewall or wrong IP. A *refused* means the service is bound to localhost only. |
| Stable address | Give the host a DHCP reservation or static IP, or use a DNS name. A changed address looks like "model unavailable". |
| Paths in Modelfiles | Windows paths in `FROM` use backslashes and need no quoting unless they contain spaces. The Linux harness never sees them, and only the model *name* crosses the wire. |
| Line endings | Preset files, env files and shell scripts edited on Windows can pick up CRLF. On Linux, that breaks `EnvironmentFile` and shebang lines. Use LF, or run `dos2unix`. |
| Harness inside WSL2 | If B is WSL2 and A is another PC, outbound LAN calls work. If A is the Windows host running WSL2, use mirrored networking or the host's LAN IP rather than `localhost`. |
| GPU sharing on the host | If the same GPU drives the host's display, desktop apps eat VRAM (on Windows, browsers and the compositor can take 1–2 GB). Budget for it or run the display from another GPU or the iGPU. With `fit off`, a model that fit yesterday can fail today. |
| Sleep and power | Disable sleep on the host and set the power plan to high performance, or the first request after idle times out. |
| Clock | Keep both machines on NTP. Skew is rarely fatal here but muddles log comparison. |
12. Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Fine at first, then forgets instructions or the task | Server context smaller than `limit.context`. Silent truncation. | Section 9.1, 9.2. Set `num_ctx` / `ctx-size` to match, or lower `limit.context`. |
| Agent useless from the first message | Context under about 16k, and OpenCode's own prompt fills it. | Use 32k or more. |
| 404 / "model not found" | Name mismatch, or a colon in a llama.cpp router section name. | Match names exactly. Remove colons. |
| Connection timeout | Firewall, wrong profile, wrong IP. | Section 11. |
| Connection refused | Service bound to `127.0.0.1`. | Set `OLLAMA_HOST=0.0.0.0:11434` or `--host 0.0.0.0`, and restart. |
| Tool calls show up as text/JSON in the reply | Missing or wrong chat template, or a model with no real tool training. | llama.cpp: add `--jinja`. Ollama: fix `TEMPLATE`. Otherwise pick a model with tool support. |
| Empty or cut-off replies from a thinking model | `limit.output` too small, with reasoning tokens consuming it. | Raise it (8192 or more) and make sure `"reasoning": true`. |
| Out-of-memory at load | Weights + KV + buffers exceed VRAM. | Quantize KV, reduce context, move MoE experts to CPU, or use a smaller quant. |
| OOM only on long prompts | Compute buffers allocated late. | Leave 1–1.5 GB headroom. Lower `--ubatch-size`. |
| MoE is very slow | Layer-split offload (Ollama), RAM paging, or too many CPU threads. | Use llama.cpp with `--n-cpu-moe`. Ensure RAM fits experts. Threads = physical cores. |
| First request after idle takes minutes | Model unloaded, reloading from disk. | `OLLAMA_KEEP_ALIVE=-1`, or keep the router model resident. Use fast storage. |
| Session titles or summaries fail or hang, while the main model works | `small_model` (or a subagent's `model`) points at a provider whose endpoint is down or gone. OpenCode uses it in the background. | Point `small_model` at a model on a host you know is up, and test that host from the harness with `curl .../v1/models`. |
| Router swaps models unexpectedly | Two clients ask for different names while `--models-max 1`. | Give each workload its own host, or accept the reload cost and avoid concurrent use. |
13. Checklist
- [ ] Host reachable from harness:
curl http://<HOST_IP>:<port>/v1/modelsreturns JSON. - [ ] Firewall limited to the harness IP, and not exposed to the internet.
- [ ] Context chosen from memory math (section 3) and a real load test.
- [ ] Ollama:
PARAMETER num_ctxin a Modelfile and a named model. llama.cpp:--ctx-sizewith--parallel 1. - [ ] Model name contains its window (
-64k) and matches the OpenCode model key exactly. - [ ]
limit.contextequals the served context.limit.outputset. - [ ] KV cache quantized if needed, and flash attention on.
- [ ] MoE: attention and KV on GPU, experts on CPU,
--n-cpu-moetuned, RAM sufficient. - [ ] Chat template and tool calling verified with a real tool call.
- [ ] Needle test passes at about 80% of the window.
- [ ] Compaction values set. Autonomy switches chosen deliberately.
14. The working setup, model by model
This is the lab the guide was written from, recorded 2026-10-02. Every model has its context set on the host, and the OpenCode entry uses the same number. Copy the pattern, not necessarily the models.
| Machine | Hardware | Role |
|---|---|---|
| Windows Ollama host | RTX 5060 Ti 16 GB, Ryzen 5 7600X (6 cores), 2×16 GB DDR5, Windows 11, Ollama 0.34.4 | Serves the big models (27–33B). Also runs OpenCode locally against `localhost`. |
| Linux llama.cpp host | 2× GTX 1650 4 GB, llama.cpp router mode | Serves small models (1.5–9B), **GPU-only on purpose** because its CPU overheats under sustained load. No CPU or RAM offload. |
| Linux harness | OpenCode | Runs the agent and calls both hosts over the LAN. |
14.1 How the context is chosen on the Windows Ollama host
Two layers, so that nothing falls back to an unknown default:
- **A host-wide default.** The context-length setting in the Ollama app (section 6.1), currently 131072. Every model without its own `num_ctx` is served at that size. The clients ask for 65536 or 32768, so this is more than they use. Ollama never serves more than the model's trained maximum. Measured: `qwen2.5:3b-instruct` (trained 32768) loaded at `CONTEXT 32768` in `ollama ps`, even with the 64k default.
- **A pinned, named copy for each model that matters,** with `num_ctx` in its Modelfile and the size in its name. This copy keeps its window even if the host default changes.
bash
# qwen3.8-27b-64k: the main agent model used from the Linux harness
FROM hf.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF:IQ3_S
TEMPLATE {{ .Prompt }}
PARAMETER num_ctx 65536bash
# nemotron-3.5-lightning:30b-128k: MoE, 25 GB, larger than the 16 GB card
FROM nemotron-3.5-lightning:30b
TEMPLATE {{ .Prompt }}
PARAMETER num_ctx 131072
PARAMETER draft_num_predict 2
PARAMETER temperature 1
PARAMETER top_p 0.95bash
# qwen35-64k-xml: Qwen3.5 9B with a custom ChatML template (no tool block)
FROM qwen3.5:9b
TEMPLATE """...ChatML template, see ollama show --template qwen35-64k-xml..."""
PARAMETER num_ctx 65536
PARAMETER stop <|im_start|>
PARAMETER stop <|im_end|>
PARAMETER stop <|endoftext|>
PARAMETER temperature 0.3Create each one with ollama create <name> -f <Modelfile> after pulling the base. Then confirm with ollama show <name> (Parameters, num_ctx) and, after the first request, ollama ps (CONTEXT column).
Warning
The host default and the pinned models are separate. The default only applies to models with no num_ctx of their own. A pinned model such as nemotron-3.5-lightning:30b-128k (num_ctx 131072) keeps its window whatever the default is. On this host the default was 262144 for a while (from the app setting or an older value of the variable, section 6.1), while every client asked for 65536 or less. Nothing was truncated, but the larger default cost speed (section 15). It is now 131072. Check the first server config line of server.log and ollama ps after any change.
14.2 Windows Ollama host: every model and its numbers
"Served" is what the host allocates: num_ctx if pinned, otherwise the host default, capped at the trained maximum. The rows below show 65536 for unpinned models. Since 2026-10-02 the host default is 131072, so unpinned models are now served at 131072 (or their trained maximum, if smaller). The two OpenCode columns are the two client configs that use this host: the Linux harness (/etc/opencode/opencode.jsonc, which sets context and output: 8192 only), and OpenCode running on the Windows host itself against localhost. "—" means the model is not in that config. MoE means the file is mostly expert weights (section 5).
| Ollama model name (exact) | Type, file size | Trained max | Served | Linux harness `limit.context` | Host-local OpenCode `context / input / output` | Tools |
|---|---|---|---|---|---|---|
| `qwen3.8-27b-64k` | dense hybrid 27B IQ3_S, 12 GB, vision | 262144 | **65536 pinned** | see note | — | yes (tested) |
| `nemotron-3.5-lightning:30b-128k` | MoE 33B Q4_K_M, 25 GB | 1048576 | **131072 pinned** | 131072 | — | yes |
| `qwen35-64k-xml:latest` | Qwen3.5 9B Q4_K_M, 6.6 GB, custom template | 262144 | **65536 pinned** | 65536 | 65536 / 57344 / 8192 | yes |
| `nemotron-3.5-lightning:30b` | MoE 33B Q4_K_M, 25 GB | 1048576 | 65536 | — | 65536 / 57344 / 8192 (default model) | yes |
| `nemotron-cascade-2:latest` | MoE 32B Q4_K_M, 24 GB | 262144 | 65536 | 65536 | 65536 / 57344 / 8192 | yes |
| `hf.co/unsloth/NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:UD-Q8_K_XL` | MoE 32B Q8, 36 GB, vision | 1048576 | 65536 | 32768 | 65536 / 57344 / 8192 | yes |
| `LoPld/qwen3-coder-30b-a3b-q4_K_S:latest` | MoE 30B Q4_K_S, 17 GB | 262144 | 65536 | 32768 | 65536 / 57344 / 8192 | yes |
| `hf.co/unsloth/GLM-4.7-Flash-GGUF:UD-Q3_K_XL` | MoE 30B Q3, 13 GB | 202752 | 65536 | 32768 | 65536 / 57344 / 8192 | yes |
| `muse-glimmer:30b` | 28B Q4_K_M, 18 GB, vision | 131072 | 65536 | 65536 | 65536 / 57344 / 8192 | yes |
| `qwen3.8:27b` | 27B Q4_K_M, 17 GB, vision | 262144 | 65536 | 32768 | 65536 / 57344 / 8192 | yes |
| `hf.co/unsloth/Qwen3.8-27B-GGUF:UD-Q3_K_XL` | 27B, 14 GB, vision | 262144 | 65536 | 32768 | 65536 / 57344 / 8192 | yes |
| `hf.co/bloomer010/Ling-3.0-tiny-GGUF:UD-Q8_K_XL` | MoE 7.9B Q8, 11 GB | 131072 | 65536 | 65536 | 65536 / 57344 / 8192 | yes |
| `qwen3.5:9b` | 9.7B Q4_K_M, 6.6 GB, vision | 262144 | 65536 | 65536 | 65536 / 57344 / 8192 | yes |
| `mistral-nemo:latest` | 12B Q4_0, 7.1 GB | 1024000 | 65536 | 65536 | 65536 / 57344 / 8192 | yes |
| `gemma4:e4b` | 8B Q4_K_M, 9.6 GB, vision, audio | 131072 | 65536 | 65536 | 65536 / 57344 / 8192, `tool_call: false` | see 14.3 |
| `ministral-3:8b` | 8.9B Q4_K_M, 6.0 GB, vision | 262144 | 65536 | 65536 | 65536 / 57344 / 4096, `tool_call: false` | see 14.3 |
| `hf.co/ibm-granite/granite-4.2-8b-GGUF:Q4_K_M` | 8.8B Q4_K_M, 5.3 GB | 131072 | 65536 | 65536 | 65536 / 57344 / 8192 | yes |
| `hf.co/ibm-granite/granite-4.0-h-tiny-GGUF:Q4_K_M` | MoE hybrid 7B, 4.2 GB | 1048576 | 65536 | 65536 | 65536 / 57344 / 8192 | yes |
| `hf.co/nvidia/NVIDIA-Nemotron-3-Nano-4B-GGUF:Q4_K_M` | 4B Q4_K_M, 2.8 GB | 1048576 | 65536 | 65536 | 65536 / 57344 / 8192 | yes |
| `hf.co/bartowski/MiniCPM5-2B-GGUF:Q8_0` | 2.5B Q8_0, 2.7 GB | 131072 | 65536 | 65536 | 65536 / 57344 / 8192 | yes |
| `hf.co/Myric/Spark-X2.5-4B-GGUF:Q4_K_M` and `hf.co/XHToken/Spark-X2.5-4B-GGUF:Q4_K_M` | 4.1B Q4_K_M, 2.5 GB | 1048576 | 65536 | 65536 | 65536 / 57344 / 8192 | yes |
| `hf.co/ibm-granite/granite-4.2-3b-GGUF:Q4_K_M` | 3.7B Q4_K_M, 2.2 GB | 131072 | 65536 | 65536 | 65536 / 57344 / 8192 | yes |
| `hf.co/bartowski/LiquidAI_LFM2.5-2.6B-GGUF:Q4_K_M` | 2.7B Q4_K_M, 1.7 GB | 128000 | 65536 | 65536 | 65536 / 57344 / 8192 | yes |
| `hf.co/unsloth/Qwen2.5-Coder-14B-Instruct-GGUF:Q4_K_M` | 14.8B Q4_K_M, 9.0 GB | 32768 | 32768 (capped) | 32768 | 32768 / 28672 / 4096 | yes |
| `qwen2.5:3b-instruct` | 3.1B Q4_K_M, 1.9 GB | 32768 | 32768 (capped, measured) | 32768 | 32768 / 28672 / 4096 (`small_model`) | yes |
| `qwen2.5-coder:1.5b` | 1.5B Q4_K_M, 1.0 GB | 32768 | 32768 (capped) | 32768 | 32768 / 28672 / 4096 | yes |
| `moondream:latest` | 1B vision, 1.7 GB | 2048 | 2048 (capped) | — | 2048 / 1024 / 1024, `attachment: true` | no |
Real use, from OpenCode's own history on the Windows host (largest prompt per reply, input plus cache read). nemotron-3.5-lightning:30b, the default model, gave 819 replies between 2026-08-24 and 2026-09-23, and its largest prompt was 62,477 tokens against a served 65,536. qwen35-64k-xml peaked at 49,635 and muse-glimmer:30b at 49,202. Real agent sessions do fill a 64k window, so the served context and the client limit have to agree. To pull the same numbers from your own install, query message in ~/.local/share/opencode/opencode.db for tokens.input + tokens.cache.read.
Long-context load tests are in sections 4 and 5. Speed, placement and a tool-call check for every row are in section 15. The MoE rows larger than 16 GB run partly from system RAM. The 36 GB Nemotron-3-Nano-Omni was not benchmarked, to avoid the risk of running the 32 GB host out of RAM.
14.3 The OpenCode side of the same models
The pattern used for every entry. context equals the served context. output is what one reply may use. input is context − output, so a full prompt plus a full reply never exceeds the window. OpenCode 1.18.32 uses limit.input, when set, to decide when to compact (checked in its source).
"ollama": {
"npm": "@ai-sdk/openai-compatible",
"name": "Ollama (local)",
"options": {
"baseURL": "http://localhost:11434/v1", // the harness uses http://<HOST_IP>:11434/v1
"timeout": 600000, // long prompts on big models take minutes
"chunkTimeout": 120000
},
"models": {
"nemotron-3.5-lightning:30b": {
"name": "Nemotron 3.5 Lightning 30B MoE",
"tool_call": true,
"limit": { "context": 65536, "input": 57344, "output": 8192 }
},
"qwen2.5:3b-instruct": {
"name": "qwen2.5 3b-instruct (1.9GB)",
"tool_call": true,
"limit": { "context": 32768, "input": 28672, "output": 4096 } // trained max is 32k
},
"moondream:latest": {
"name": "moondream latest (1.7GB, vision)",
"tool_call": false,
"attachment": true,
"limit": { "context": 2048, "input": 1024, "output": 1024 }
}
}
}Set "tool_call": false for any model that fails the tool test in section 6.3. OpenCode then stops offering it tools instead of failing on every turn.
14.4 Linux llama.cpp host: every model and its numbers
From the real models.ini in section 7.2, and the harness provider for this host (baseURL http://<HOST_IP>:8080/v1, output: 8192). "both" means split-mode = layer across the two 4 GB cards. The speeds are generation tokens/s as recorded in the model labels of the client config. Measured values from 2026-10-02 are in section 15 and agree within a few tok/s. Every harness entry matches ctx-size exactly. "—" means the model is not in the harness config.
| Preset name (exact) | GGUF | Cards | `ctx-size` | KV type | Gen tok/s | Harness `limit.context` | Tools |
|---|---|---|---|---|---|---|---|
| `granite-4.0-h-tiny` | `granite-4.0-h-tiny-Q4_K_M.gguf` (MoE) | both, loaded at boot | 65536 | f16 | ~68 | 65536 | yes |
| `lfm2.5-2.6b` | `LiquidAI_LFM2.5-2.6B-Q4_K_M.gguf` | one | 65536 | f16 | ~59 | 65536 | yes (thinks) |
| `nemotron-3-nano-4b` | `NVIDIA-Nemotron3-Nano-4B-Q4_K_M.gguf` | both | 131072 | f16 | ~36 | 131072 | yes (thinks) |
| `qwen2.5-3b-instruct` | `qwen2.5-3b-instruct-Q4_K_M.gguf` | one | 32768 (trained max) | f16 | ~48 | 32768 | yes |
| `qwen2.5-coder-1.5b` | `qwen2.5-coder-1.5b-instruct-Q4_K_M.gguf` | one | 32768 | f16 | ~83 | 32768 | no (JSON as text) |
| `moondream` | `moondream2-text.gguf` + mmproj, custom template | one | 2048 (trained max) | f16 | — | — | no (vision) |
| `granite-4.2-3b` | `granite-4.2-3b-Q4_K_M.gguf` | both | 65536 | q8_0 | ~42 | 65536 | yes |
| `minicpm5-2b` | `MiniCPM5-2B-Q8_0.gguf` | both | 32768 | f16 | ~40 | 32768 | yes (thinks) |
| `spark-x2.5-4b` | `Spark-X2.5-4B-Q4_K_M.gguf` | both | 32768 | f16 | ~33 | 32768 | yes (thinks) |
| `qwen3.5-9b` | `Qwen3.5-9B-Q4_K_M.gguf` | both | 32768 (64k OOM) | f16 | ~19 | 32768 | yes (thinks) |
| `ministral-3-8b` | `Ministral-3-8B-Instruct-2512-Q4_K_M.gguf` | both | 16384 (32k OOM) | f16 | ~19 | — | yes |
| `granite-4.2-8b` | `granite-4.2-8b-Q4_K_M.gguf` | both | 8192 (16k OOM) | f16 | — | — | too small for OpenCode |
| `qwen2.5-coder-7b` | `qwen2.5-coder-7b-instruct-Q4_K_M.gguf` | both | 131072 | q4_0 | — | 131072 | not recorded |
14.5 A second client on the same hosts: dsh
The Linux harness also runs dsh (DeepSeek AI Studio Harness, 0.1.0-rc.7) as a web UI against the same two hosts. It is a different client with the same rule: its per-model contextWindow must not exceed what the host serves. Its providers live in a patch file under ~/.dsh/. dsh --profile web --dump-config prints the composed result, so you can check it. The relevant part, read 2026-10-02:
providers:
local-ollama: # the Windows Ollama host
api: openai-completions
baseURL: http://<HOST_IP>:11434/v1
defaultContextWindow: 65536 # = OLLAMA_CONTEXT_LENGTH on the host
defaultMaxTokens: 8192
timeoutMs: 900000
streamIdleTimeoutMs: 600000
models:
- id: nemotron-3.5-lightning:30b # served at 65536 (host default)
contextWindow: 65536
maxTokens: 8192
- id: qwen2.5:3b-instruct # trained max 32k, so served 32768
contextWindow: 32768
maxTokens: 4096
gpu-108-llamacpp: # the Linux llama.cpp host
api: openai-completions
baseURL: http://<HOST_IP>:8080/v1
defaultContextWindow: 32768
models:
- id: granite-4.0-h-tiny # = ctx-size 65536 in models.ini
contextWindow: 65536
maxTokens: 16384Every dsh entry checked is equal to or below the served context. Two entries are below it (nemotron-3-nano-4b at 32768 against 131072 served, and granite-4.2-3b at 32768 against 65536), which is safe. qwen2.5-coder-7b is not listed in dsh. To use the 128k Nemotron from dsh, add id: nemotron-3.5-lightning:30b-128k with contextWindow: 131072. That pinned name exists on the host, but a 128k load of that model on the 16 GB card has not been measured.
14.6 Adding a model the same way
- **Decide the use first.** Agent work needs 32k or more, and 64k is comfortable (section 7.2). Quick chat or titles can live with less. Vision-only models only need their trained maximum.
- **Find the largest context that fits.** Do the memory math (section 3), load the model, and read the load log or `ollama ps`. On a GPU-only host, a load that spills is a failure. Lower the context.
- **Pin it on the host.** Ollama: a Modelfile with `num_ctx` and a name with the size in it (14.1). llama.cpp: a preset section with `ctx-size` and a comment recording the fit test.
- **Add the OpenCode entry** with the exact name, `context` = served, `output` chosen, `input = context − output`, and the right `tool_call`.
- **Verify** with section 9: `ollama ps` or the load log for the real context, then the needle test at about 80%.
15. Benchmarks: every model on both hosts
Run from the Linux harness over the LAN on 2026-10-02, 21:16–21:26, both hosts at once. Hardware as in section 14: Windows Ollama host (RTX 5060 Ti 16 GB, Ollama 0.34.4) and Linux llama.cpp host (2× GTX 1650 4 GB, llama.cpp 0.4.0-dev). Each model got three requests:
- A short question with up to 256 tokens of answer, which gives load time and generation speed.
- An 8,000-token prompt, which gives prompt-processing speed.
- One tool-call request, with a weather tool and "use the tool", which shows whether a structured `tool_calls` entry comes back.
End to end with OpenCode (opencode run on the harness with its real system config: write fib.py, run it, report the value): Windows host nemotron-3.5-lightning:30b-128k in 37 s, Linux host granite-4.0-h-tiny in 34 s. Both made real write and bash tool calls and reported 832040.
| Model (Windows Ollama host) | `num_ctx` set by the benchmark | Load s | Gen tok/s | Prompt tok/s (8k) | Placement | Tool call |
|---|---|---|---|---|---|---|
| `qwen3.8-27b-64k` | 65536 | already loaded | 16.4 | 783.3 | 61/65 layers on GPU (section 4) | yes |
| `nemotron-3.5-lightning:30b` | 65536 | 17.1 | 50.6 | 1212.4 | 13.1 of 25.8 GB on GPU | yes |
| `nemotron-3.5-lightning:30b-128k` | 131072 | 18.1 | 48.4 | 1181.9 | 12.8 of 26.0 GB on GPU | yes |
| `nemotron-cascade-2:latest` | 65536 | 15.9 | 49.8 | 1228.4 | 14.4 of 25.8 GB on GPU | yes |
| `LoPld/qwen3-coder-30b-a3b-q4_K_S:latest` | 32768 | 11.2 | 67.1 | 1898.9 | 14.8 of 19.4 GB on GPU | yes |
| `hf.co/unsloth/GLM-4.7-Flash-GGUF:UD-Q3_K_XL` | 32768 | 9.3 | 101.1 | 1714.4 | all on GPU (14.7 GB) | yes |
| `muse-glimmer:30b` | 65536 | 13.6 | 8.6 | 639.2 | 13.1 of 17.6 GB on GPU | yes |
| `qwen3.8:27b` | 32768 | 13.2 | 9.6 | 428.2 | 12.0 of 18.6 GB on GPU | yes |
| `hf.co/unsloth/Qwen3.8-27B-GGUF:UD-Q3_K_XL` | 32768 | 9.8 | 18.8 | 840.7 | 14.0 of 15.2 GB on GPU | yes |
| `hf.co/unsloth/Qwen2.5-Coder-14B-Instruct-GGUF:Q4_K_M` | 32768 | 5.1 | 42.2 | 1554.1 | all on GPU (12.2 GB) | **no** |
| `mistral-nemo:latest` | 65536 | 4.3 | 54.8 | 2320.6 | all on GPU (12.8 GB) | yes |
| `hf.co/bloomer010/Ling-3.0-tiny-GGUF:UD-Q8_K_XL` | 65536 | 7.0 | 29.5 | 5896.8 | all on GPU (11.3 GB) | yes |
| `qwen3.5:9b` | 65536 | 5.3 | 66.1 | 2749.9 | all on GPU (7.2 GB) | yes |
| `qwen35-64k-xml:latest` | 65536 | 3.2 | 69.1 | 2749.1 | all on GPU (7.2 GB) | yes |
| `gemma4:e4b` | 65536 | 8.0 | 92.0 | 4626.6 | all on GPU (3.5 GB) | yes |
| `ministral-3:8b` | 65536 | 5.0 | 69.4 | 2985.2 | all on GPU (10.2 GB) | yes |
| `hf.co/ibm-granite/granite-4.2-8b-GGUF:Q4_K_M` | 65536 | 3.5 | 68.2 | 2554.9 | all on GPU (11.2 GB) | yes |
| `hf.co/ibm-granite/granite-4.0-h-tiny-GGUF:Q4_K_M` | 65536 | 3.0 | 197.8 | 6830.9 | all on GPU (5.2 GB) | yes |
| `hf.co/nvidia/NVIDIA-Nemotron-3-Nano-4B-GGUF:Q4_K_M` | 65536 | 2.7 | 120.6 | 4581.4 | all on GPU (3.9 GB) | yes |
| `hf.co/Myric/Spark-X2.5-4B-GGUF:Q4_K_M` | 65536 | 2.5 | 104.1 | 4665.9 | all on GPU (3.3 GB) | yes |
| `hf.co/ibm-granite/granite-4.2-3b-GGUF:Q4_K_M` | 65536 | 2.2 | 133.5 | 4879.7 | all on GPU (5.5 GB) | yes |
| `hf.co/bartowski/MiniCPM5-2B-GGUF:Q8_0` | 65536 | 2.5 | 130.2 | 8253.0 | all on GPU (4.4 GB) | yes |
| `hf.co/bartowski/LiquidAI_LFM2.5-2.6B-GGUF:Q4_K_M` | 65536 | 2.2 | 184.0 | 8612.2 | all on GPU (2.8 GB) | yes |
| `qwen2.5:3b-instruct` | 32768 | 2.2 | 151.3 | 6544.0 | all on GPU (2.8 GB) | yes |
| `qwen2.5-coder:1.5b` | 32768 | 1.9 | 239.8 | 11525.4 | all on GPU (1.7 GB) | **no** |
| Preset (Linux llama.cpp host) | ctx-size | Load s (approx.) | Gen tok/s | Prompt tok/s (8k) | Tool call |
|---|---|---|---|---|---|
| `granite-4.0-h-tiny` | 65536 | 2.0 | 69.7 | 315.9 | yes |
| `lfm2.5-2.6b` | 65536 | 1.4 | 60.3 | 246.0 | yes |
| `nemotron-3-nano-4b` | 131072 | 7.2 | 36.0 | 254.9 | yes |
| `qwen2.5-3b-instruct` | 32768 | 5.1 | 50.7 | 158.2 | yes |
| `granite-4.2-3b` | 65536 | 5.7 | 45.4 | 193.0 | yes |
| `minicpm5-2b` | 32768 | 6.4 | 42.4 | 323.1 | yes |
| `spark-x2.5-4b` | 32768 | 6.3 | 35.4 | 225.9 | yes |
| `qwen3.5-9b` | 32768 | 2.5 | 19.5 | 140.4 | yes |
| `ministral-3-8b` | 16384 | 11.8 | 20.5 | 115.1 | yes |
| `qwen2.5-coder-1.5b` | 32768 | 3.8 | 88.5 | 281.9 | **no** |
| `qwen2.5-coder-7b` | 131072 | 11.1 | 24.2 | 137.3 | **no** |
Recommendation & Notes
How to read these.
- **Placement matters more than size.** MoE models larger than the card (Nemotron, Qwen3-Coder-30B-A3B) still generate at 48–67 tok/s with experts in RAM. Dense 27–28B models that spill a few layers drop to 9–19 tok/s.
- **The small 4 GB cards are usable, not fast.** They give 20–88 tok/s generation, but prompt processing is 115–323 tok/s, so a fresh 30k-token agent context takes about 1.5 to 4.5 minutes before the first word. For the same models, the 16 GB card processes prompts 20 to 40 times faster: `qwen2.5-3b` 6544 vs 158, `granite-4.0-h-tiny` 6831 vs 316, and `qwen3.5-9b` 2750 vs 140 tok/s.
- **A single tool test is a quick screen, not proof.** The coder models (`Qwen2.5-Coder` 1.5B, 7B and 14B) returned no structured tool call, matching the preset comments. `ministral-3:8b` and `gemma4:e4b` did return one here, although one client config marks them `tool_call: false`. Re-test in your client before changing it.
- **The host default changes these numbers.** OpenCode and dsh reach unpinned models through `/v1`, so they get the host default, not the client's limit. The same four models were measured that way at two host defaults (generation tok/s, same 256-token request):
The server sizes the KV cache for the full default window and moves weights off the GPU to make room. Hybrid models such as Nemotron have a tiny KV cache and do not care. Dense models lose the most. Set the default no larger than your clients need, and pin a separately named model for the cases that need more (14.1). Measured 2026-10-02.Model Host default 262144 Host default 131072 `num_ctx` 32768 (table above) `qwen3.8:27b` 5.8 (22/66 layers on GPU) 7.3 (33/66) 9.6 `hf.co/unsloth/Qwen3.8-27B-GGUF:UD-Q3_K_XL` 4.3 (33/66) 6.8 (46/66) 18.8 `hf.co/unsloth/GLM-4.7-Flash-GGUF:UD-Q3_K_XL` 56.6 (served 202752, its maximum) 72.1 101.1 `LoPld/qwen3-coder-30b-a3b-q4_K_S:latest` 28.2 (44/49) 44.6 (49/49) 67.1 `nemotron-3.5-lightning:30b` and `:30b-128k` about 50 51.1 and 50.6 — - Load times are cold loads from disk, or from the page cache where a model ran recently. The 8k needle answer is not scored here: thinking models spend the 64-token budget on reasoning. Sections 4 and 9.2 cover truncation properly.
The benchmark script (Python 3, standard library only)
Edit the two addresses and the model lists, then run `python3 bench.py ollama > ollama.jsonl` and `python3 bench.py llama > llama.jsonl`. Each line is one model as JSON.bash
#!/usr/bin/env python3
"""Benchmark every chat model on both hosts from the harness.
Usage: python3 bench.py ollama|llama > results.jsonl
Per model: short generation, ~8k-token prompt, one tool-call check, placement."""
import json, sys, time, urllib.request
OLLAMA = "http://<HOST_IP>:11434"
LLAMA = "http://<LLAMA_HOST_IP>:8080"
# (model, context the client config uses)
OLLAMA_MODELS = [
("qwen3.8-27b-64k", 65536), ("nemotron-3.5-lightning:30b", 65536),
("nemotron-3.5-lightning:30b-128k", 131072), ("nemotron-cascade-2:latest", 65536),
("LoPld/qwen3-coder-30b-a3b-q4_K_S:latest", 32768), ("hf.co/unsloth/GLM-4.7-Flash-GGUF:UD-Q3_K_XL", 32768),
("muse-glimmer:30b", 65536), ("qwen3.8:27b", 32768), ("hf.co/unsloth/Qwen3.8-27B-GGUF:UD-Q3_K_XL", 32768),
("hf.co/unsloth/Qwen2.5-Coder-14B-Instruct-GGUF:Q4_K_M", 32768), ("mistral-nemo:latest", 65536),
("hf.co/bloomer010/Ling-3.0-tiny-GGUF:UD-Q8_K_XL", 65536), ("qwen3.5:9b", 65536), ("qwen35-64k-xml:latest", 65536),
("gemma4:e4b", 65536), ("ministral-3:8b", 65536), ("hf.co/ibm-granite/granite-4.2-8b-GGUF:Q4_K_M", 65536),
("hf.co/ibm-granite/granite-4.0-h-tiny-GGUF:Q4_K_M", 65536), ("hf.co/nvidia/NVIDIA-Nemotron-3-Nano-4B-GGUF:Q4_K_M", 65536),
("hf.co/Myric/Spark-X2.5-4B-GGUF:Q4_K_M", 65536), ("hf.co/ibm-granite/granite-4.2-3b-GGUF:Q4_K_M", 65536),
("hf.co/bartowski/MiniCPM5-2B-GGUF:Q8_0", 65536), ("hf.co/bartowski/LiquidAI_LFM2.5-2.6B-GGUF:Q4_K_M", 65536),
("qwen2.5:3b-instruct", 32768), ("qwen2.5-coder:1.5b", 32768),
]
LLAMA_MODELS = ["granite-4.0-h-tiny", "lfm2.5-2.6b", "nemotron-3-nano-4b", "qwen2.5-3b-instruct", "granite-4.2-3b",
"minicpm5-2b", "spark-x2.5-4b", "qwen3.5-9b", "ministral-3-8b", "qwen2.5-coder-1.5b", "qwen2.5-coder-7b"]
SHORT = "Explain TCP slow start in about 150 words."
LONG = ("CODE: KIWI-5521.\n" + "The quick brown fox jumps over the lazy dog. " * 800 +
"\nWhat was the code at the start? Reply with the code only.") # ~8k tokens
TOOLS = [{"type": "function", "function": {"name": "get_weather", "description": "Get current weather for a city",
"parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}]
TOOL_Q = "What is the weather in Paris right now? Use the tool."
def post(url, body, timeout=1800):
t = time.time()
req = urllib.request.Request(url, json.dumps(body).encode(), {"Content-Type": "application/json"})
r = json.load(urllib.request.urlopen(req, timeout=timeout))
return r, time.time() - t
def get(url):
return json.load(urllib.request.urlopen(url, timeout=30))
def rate(n, ns):
return round(n / (ns / 1e9), 1) if n and ns else None
def bench_ollama(model, ctx):
o = {"num_ctx": ctx}
def chat(msg, n, tools=None, keep="5m"):
b = {"model": model, "stream": False, "keep_alive": keep, "options": dict(o, num_predict=n),
"messages": [{"role": "user", "content": msg}]}
if tools: b["tools"] = tools
return post(f"{OLLAMA}/api/chat", b)
a, wall = chat(SHORT, 256)
ps = [m for m in get(f"{OLLAMA}/api/ps")["models"] if m["name"] == model or m["model"] == model]
p = ps[0] if ps else {}
b, _ = chat(LONG, 64)
c, _ = chat(TOOL_Q, 1024, TOOLS, keep=0)
tc = (c.get("message") or {}).get("tool_calls")
return {"host": "ollama", "model": model, "ctx": p.get("context_length", ctx),
"load_s": round(a.get("load_duration", 0) / 1e9, 1), "gen_tps": rate(a.get("eval_count"), a.get("eval_duration")),
"prompt_tps_8k": rate(b.get("prompt_eval_count"), b.get("prompt_eval_duration")), "prompt_n": b.get("prompt_eval_count"),
"needle_ok": "KIWI-5521" in (b.get("message") or {}).get("content", ""),
"vram_gb": round(p.get("size_vram", 0) / 1e9, 1), "total_gb": round(p.get("size", 0) / 1e9, 1),
"tool_call": bool(tc), "tool_args": json.dumps(tc[0]["function"]["arguments"]) if tc else None}
def bench_llama(model):
def chat(msg, n, tools=None):
b = {"model": model, "max_tokens": n, "messages": [{"role": "user", "content": msg}]}
if tools: b["tools"] = tools
return post(f"{LLAMA}/v1/chat/completions", b)
a, wall = chat(SHORT, 256)
ta = a.get("timings", {})
b, _ = chat(LONG, 64)
tb = b.get("timings", {})
c, _ = chat(TOOL_Q, 1024, TOOLS)
tc = c["choices"][0]["message"].get("tool_calls")
load = wall - (ta.get("prompt_ms", 0) + ta.get("predicted_ms", 0)) / 1000
return {"host": "llama", "model": model, "load_s_approx": round(load, 1),
"gen_tps": round(ta.get("predicted_per_second", 0), 1), "prompt_tps_8k": round(tb.get("prompt_per_second", 0), 1),
"prompt_n": tb.get("prompt_n"), "needle_ok": "KIWI-5521" in (b["choices"][0]["message"].get("content") or ""),
"tool_call": bool(tc), "tool_args": tc[0]["function"]["arguments"] if tc else None}
if __name__ == "__main__":
which = sys.argv[1]
items = OLLAMA_MODELS if which == "ollama" else [(m, None) for m in LLAMA_MODELS]
for m, ctx in items:
try:
r = bench_ollama(m, ctx) if which == "ollama" else bench_llama(m)
except Exception as e:
r = {"host": which, "model": m, "error": str(e)[:200]}
r["when"] = time.strftime("%Y-%m-%d %H:%M")
print(json.dumps(r), flush=True)Written by Jason Wolff, 2026-10-02. Flag names and defaults for Ollama and llama.cpp change quickly. When a flag in this guide is rejected, check the release notes or --help for your version. The worked KV table in section 3 is illustrative. The measured boxes give their hardware and date. Verified on 2026-10-02: Ollama 0.34.4 on Windows 11 (RTX 5060 Ti 16 GB), its bundled llama-server flags, and OpenCode 1.18.32 config keys. MoE offload measured on the same Windows host, through Ollama and through its bundled llama-server (section 5). Not verified: a standalone llama.cpp release on Windows.