An Agent on Your Own Machine

Gemma 4 and Hermes Desktop, with nothing leaving the room — served by Ollama, Docker Model Runner or vLLM, measured on a 24 GB Mac; and LM Studio’s Bionic, which is agent and runtime in one app

Author

Behzad Samadi

Published

September 13, 2026

A paper-craft study at night: a laptop glowing amber on a wooden desk, a lamp, a mug, a plant, a window on a starry sky, a closed door with light around its frame, and the five-dot AI glyph on the wall. Title: An Agent on Your Own Machine.

There is one reason to run an AI agent on your own computer that outweighs every other: what you tell it stays there. Not “is encrypted in transit,” not “is not used for training on the paid tier” — stays there, on the disk you can unplug. For a tool that reads bank statements, that is the whole argument. It is also true for offline trains, for metered budgets, and for the simple pleasure of a thing that keeps working when a company changes its terms.

This guide installs that agent — Google’s Gemma 4 as the model, Nous Research’s Hermes as the agent, Hermes Desktop as the window — and then measures what you get, on a 24 GB Mac, against the same task done by a hosted model. The numbers are the part most guides leave out. They are not flattering to the local model, and they are the reason to read to the end before you decide which size to download.

NoteTL;DR
  1. The first thing that breaks is the context window. Ollama defaults to 4,096 tokens; Hermes refuses to start an agent session under 64,000. Neither quick start mentions the other. Set OLLAMA_CONTEXT_LENGTH=65536 before you do anything else.
  2. Pull gemma4:12b, not gemma4. The default tag is the 9.6 GB E4B; the 12B is 7.6 GB and the model Google calls the unified one. Check df first — a 7.6 GB pull on a disk with 15 GiB free left 721 MiB.
  3. What a 24 GB Mac delivers: 12 tokens a second, 8.2 GB resident, and a tool-using task that a hosted model finished correctly in 2 minutes taking 20 minutes with three self-inflicted errors — because the model thinks for minutes between steps, and because an agent’s prompt is re-read every turn. Measure prefill and generation, not one number; turn reasoning down; keep a person in the loop; prefer a mixture-of-experts model; or buy memory.
  4. Three servers, one decision each. Ollama on a laptop. Docker Model Runner if Docker Desktop is already there. vLLM if the machine is a Linux box with an NVIDIA card. And one way to skip the seam: LM Studio’s Bionic is an agent with the runtime inside it — local models, its cloud, or another of your machines, chosen per session.
  5. Desktop is not a different agent. Same config, keys, skills and sessions as the terminal; hermes desktop opens it.

Three parts, and where the seams are

A local agent is three pieces that talk over one protocol:

Part What it is Here
The model Weights on disk; the thing that thinks Gemma 4, 12B, 4-bit
The server Loads the weights onto the GPU and answers an OpenAI-shaped /v1/chat/completions Ollama — or Docker Model Runner, or vLLM
The agent and its window The tool loop, memory, skills; and a UI Hermes, in the terminal or Hermes Desktop

The seam between the server and the agent is a URL and a model name. That is why the three servers are interchangeable, and why the agent does not care which one is behind the port. The seam between the model and the server is where the quiet failure lives — the context window — and it gets its own section.

Gemma 4: which size is honest for your machine

Gemma 4 is Google’s open-weights family: text, image and (on the smaller models) audio and video in, up to 256K tokens of context on the medium sizes, function calling built in. The licence permits commercial use with conditions; read it before you ship on it. Five sizes, and Google’s own table of what the weights take at 4-bit — the precision every local runner uses:

Size Weights at 4-bit On Ollama Fits, with room for a 64K context
E2B 2.9 GB gemma4:e2b, 7.2 GB on disk 8 GB machine
E4B 4.5 GB gemma4:e4b = gemma4:latest, 9.6 GB 8–16 GB
12B 6.7 GB gemma4:12b, 7.6 GB 16–24 GB
26B A4B (mixture of experts — 4B active per token, so it generates like a small model) 14.4 GB gemma4:26b, 19 GB 32 GB
31B 17.5 GB gemma4:31b, 20 GB 48 GB

Two things the table hides. The weights are not the whole cost: the key/value cache for a 64K context sits on top, and the operating system and your other applications want memory too — with the 12B loaded on a 24 GB Mac, a browser and an editor open, free memory read 0.1 GB. And the default tag is a trap: ollama run gemma4 gets you the E4B, which is larger on disk than the 12B and a smaller model.

Step one: the server

Ollama (the walkthrough)

Download the app from ollama.com/download — macOS Sonoma or newer, Apple silicon for the GPU — and let it link the ollama command into your path when it asks. Or, without the app, brew install ollama. Models live in ~/.ollama, and the uninstall is documented file by file, which is more than most installers offer.

Then, and this is the order that matters:

launchctl setenv OLLAMA_CONTEXT_LENGTH 65536     # then quit and reopen the Ollama app
ollama pull gemma4:12b                           # 7.6 GB — check df first
ollama run gemma4:12b "In one sentence, what is a bank statement?"

If you run the server by hand rather than through the app, the variable goes on the command: OLLAMA_CONTEXT_LENGTH=65536 ollama serve.

The rule that is documented on both sides and mentioned by neither quick start

Ollama’s FAQ: “By default, Ollama uses a context window size of 4096 tokens.” Hermes’s providers page: “Hermes Agent requires at least 64,000 tokens of context for agent use with tools. Smaller windows are rejected at startup.” Hermes goes on to call this “the #1 source of confusion when integrating Ollama with tools like Hermes.”

You cannot fix it from the agent’s side. The OpenAI-compatible API has no field for it; it is a server setting, and on the Ollama app it is the launchctl line above. The check that it took is ollama ps after the first request — the CONTEXT column should read your number. On this machine it read 65536, and the model reported 8.2 GB · 100% GPU.

WarningWhat “64K” costs you

A longer context is memory: the cache grows with every token in the window. On 24 GB, the 12B with a 64K window fit with nothing to spare. If you are on 16 GB, drop to the E4B, or accept that Hermes needs a restricted toolset — its docs suggest -t file,web for small models — rather than a full window.

Step two: the agent

Two installers, same result. On macOS and Windows the recommended path is the Hermes Desktop installer, which installs the command line and the app together. Or the command line alone:

curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash

It brings its own Python, Node, ripgrep and ffmpeg, puts the code in ~/.hermes/hermes-agent and a hermes command in ~/.local/bin, and finishes by asking which model provider you want. Skip that; you are about to point it at your own.

Give the local agent its own profile, so its settings never collide with a hosted one you might also use:

hermes profile create local --no-skills     # creates the `local` command
local config set model.default gemma4:12b
local config set model.provider custom
local config set model.base_url http://localhost:11434/v1
local config set model.context_length 64000
local chat -q "What can you see on this machine?"

Or interactively: local model, choose Custom endpoint, paste the URL, leave the key blank, type the model name. Ollama’s own shortcut, ollama launch hermes --model gemma4:12b, does the same thing for Hermes’s default profile; it is the quickest path if you only ever want one.

Step three: the window

local desktop

Hermes Desktop is not a second product. Its own page says it plainly: “same config, same API keys, same sessions, same skills, same memory” — a native app over the agent the terminal drives. On a command-line-only install, hermes desktop builds the packaged app the first time (on this machine it landed inside the checkout at apps/desktop/release/mac-arm64/Hermes.app, 311 MB, Electron) and opens it after that. If you installed with the Desktop installer it is already there. Any skill installed in the profile is a slash command in the app, and a session started in the terminal can be resumed in the window.

That is the install. The rest of this guide is what it is like.

What you get, measured

The numbers below are one machine, one afternoon: an M4 with 24 GB, a browser and an editor open. They are the order of magnitude, not the value.

Raw speed. Through Ollama’s API, gemma4:12b processed prompts at about 60 tokens a second and generated at 12 tokens a second, thinking on or off. Twelve a second reads comfortably; it is roughly the pace of a fast typist. Cold — first request after boot — a one-sentence answer took 61 seconds, most of it loading the weights and the model’s reasoning.

Thinking. Gemma 4 reasons before it answers, by default. For a three-sentence question, that was 809 characters of thought before the answer: 22.5 seconds with thinking, 7.2 seconds without. In a chat, the difference is a pause. In an agent loop, it multiplies by every tool call.

The task. The honest test was a real job: the OWL Planner’s interview — run a budget tool on an invented household, read the two merchants it could not classify, write the rules, merge them, run again, report. The same skill, the same invented data, once with a hosted model and once with the local one:

Hosted model (free tier) Gemma 4 12B, local
Time 2 min 11 s 20 min 0 s
Tool calls 10 22
Mistakes on the way none a merchant-name typo, a path typo, an empty pattern
Result one correct rule; the ambiguous merchant left unsure one mistyped rule that matches nothing, two junk patterns, the ambiguous merchant left unsure
Its own report correct claimed nothing remained; both merchants still did

Read the last two rows twice. The local model’s reasoning trace shows it noticing its typo — “fixed newton to newtown” — and then writing the typo anyway. When the tool refused an empty string as a pattern, the model filled the section with a placeholder word instead of deleting the section, and a pattern of transfer matches every description with that word in it. That is the exact failure the tool’s own prompt warns against: a wrong rule is worse than no rule.

Two things went right, and they are the design lesson. The tool’s gate refused the empty pattern — an importer that reconciles, a merge that validates, catches what a model will not. And the ambiguous merchant stayed unsure, because the instruction to say so was in the prompt the tool wrote, not left to the model’s judgement.

TipWhat this means for choosing
  • For chat, drafting, and questions about your own documents, a local 12B on 24 GB is fine, and private. Twelve tokens a second is not the bottleneck; you are.
  • For an agent loop with consequences — rules that persist, files that change — either turn reasoning down (hermes chat --reasoning low; Hermes offers levels from none up), keep a person answering each step (the OWL skill’s real mode asks through a form rather than deciding), or run a larger model on a larger machine. The 26B mixture-of-experts wants 32 GB; the 31B wants 48.
  • Do not read the twenty minutes as “local is slow.” It is “thinking is expensive and the loop paid it twenty-two times.” That is a setting.

Two numbers, not one: prefill and generation

The measurements above report generation speed, and generation is the number every local-AI page reports. For an agent it is the less important of two. The other is prefill — how fast the model reads the prompt — and an agent’s prompt is not a question. It is the system prompt, the tool schemas, the instruction files and the conversation so far, re-sent on every turn of the loop. Rob Braxman, who runs his agents entirely locally and describes the setup in a twenty-minute video from 11 September 2026, puts a typical agent prompt at about 16,000 tokens, and his arithmetic is the one to carry around: at 100 tokens a second of prefill, that is three minutes before the model has said a word, on every uncached turn.

Read the interview run above again with that in mind. Twenty-two tool calls means twenty-two prompts, each carrying everything before it. The “thinking” was part of the twenty minutes; the reading was the other part, and this guide did not measure it — the prompt-speed figure earlier, about 60 tokens a second, came from a 30-token prompt, far too short to say what a 16,000-token one would cost. What rescues the loop is the KV cache: the server keeps the computed state of a prompt prefix, so when the next turn begins with the same system prompt and the same history, only the new tokens are read. Braxman’s observation is that after the first turn, his prefill drops “to one or two seconds.” That is why an agent loop on a local model is not twelve times the cost of one question — as long as the cache is on and the prefix really is the same.

His machine is worth naming because it is not a Mac: an AMD Strix Halo with 128 GB of unified memory, 96 GB of it available to the GPU, about $4,000. On it he runs gpt-oss-120b — a 117-billion-parameter mixture-of-experts model at 69 GB — and, after tuning, measures 688 tokens a second of prefill and 53 of generation, up from roughly 100 and 20. His four levers, in his order:

  1. llama.cpp directly, not through Ollama — “a doubling of the speed” for him. Ollama is built on llama.cpp; the server underneath exposes flags the wrapper does not.
  2. Vulkan instead of ROCm on AMD — with the caveat that which of the two is faster “is never constant.”
  3. A GPU clock boost on that hardware.
  4. Mixture-of-experts models. Only a fraction of the weights work on each token, so a large MoE generates like a small model: he calls the difference “often five times.” His coding model, a 35B MoE, prefills at 1,071 tokens a second on the same box. In the Gemma 4 table above, the 26B A4B is that kind of model; on a machine with the 32 GB it wants, it is the size to prefer.

The server command he shows — checked against llama.cpp’s server documentation — is a good reading of what matters for an agent:

llama-server -m gpt-oss-120b-MXFP4.gguf \
  -ngl 999 -c 131072 --jinja -fa on \
  -ub 512 -b 2048 \
  --reasoning off --reasoning-format deepseek \
  --cache-prompt --host 0.0.0.0 --port 8081

-c 131072 is the context rule from this guide, set to 128K rather than 64K. -ngl 999 puts every layer on the GPU. -fa on is flash attention. --reasoning off is the thinking switch this guide measured at 22.5 versus 7.2 seconds. --cache-prompt is the KV cache — the README lists it, and --jinja, as on by default now, so writing them out is documentation rather than configuration. And -b 2048 -ub 512 are the defaults too. The endpoint is OpenAI-compatible; Hermes reaches it as a custom endpoint like any other, with its own warning that a full default toolset can overflow a 32K window on llama.cpp-family servers.

Two more of his cautions transfer whole. Watch the context: past a certain size, compaction on a local model can corrupt what the agent knows, and the remedy is to trim or start a new session, and whatever your harness calls its context report is the thing to keep an eye on. And the economics that make any of this worth tuning are his, not the guide’s: he reports 1.2 billion tokens on one week-long coding job through a hosted open model, at a cost he puts at about $70 that week, and calls the same volume on a frontier API “over ten grand.” None of his figures were reproduced here; they are the numbers of one careful practitioner on one machine, offered as such.

Good enough for the job

Greg Isenberg’s I’m Obsessed With Local AI. Here’s Why, thirty-nine minutes from 8 September 2026, is a map for founders rather than a setup guide — sponsored by Google, built around Gemma, and without a measurement in it — but it asks the question this guide’s numbers were circling. Not is the local model smarter than the hosted one, which it is not, but is it good enough for the job, and does running it here make the product better. Three things from it belong beside the measurements above.

  • Local and hosted as a pipeline, not a choice. The table above puts them in two columns. His arrangement puts them in sequence: the local model reads the private files and does the first pass; a sanitised version of the problem goes to a hosted model when the reasoning is hard; a person approves before anything important leaves. The bank statements never leave the machine; a question about a merchant category could go anywhere.
  • Ten runs, then one comparison. One folder, one model, one output file. Run it ten times and note where the model gets confused; fix the prompt, add an example, add a checklist. Then run the same ten inputs through a frontier model and compare — did the local model catch the same items, pull the right quotes, keep the format? That is an evaluation cheap enough to actually do, and it tells you where local is already enough. The twenty-minute run above was one such pair, on one input: the hosted model’s answers are in OWL Planner’s first pull request, the local model’s in the research note.
  • Read the model card slowly. Six things to look for before anything else: what the model is for, how big it is, its license, what hardware people run it on, which inputs it takes (text, images, audio, tool use), and whether quantised files exist. The rest of the card can wait.

Two places the video and this guide part ways. He starts with the E4B — ollama run gemma4:e4b — and the size table above says why the 12B is the better pull on 16 GB or more: smaller on disk, and the stronger model. And the context window, the first thing that broke here, is not mentioned once.

The other two servers

Both were checked against their documentation; neither was run for this guide — Docker because the disk had 721 MiB left after the pull, vLLM because it wants hardware this laptop does not have. The research notes say so.

Docker Model Runner — if Docker Desktop is already on the machine

Docker Desktop 4.40 or newer on macOS ships a model runner that serves OpenAI- and Ollama-compatible APIs, with llama.cpp as the engine on a Mac and vLLM available on Linux with NVIDIA GPUs. Models are OCI artifacts on Docker Hub — ai/gemma4 had 49 tags on the day of writing, 7.3 GB for the default, including MLX builds for Apple silicon — or straight from Hugging Face.

docker desktop enable model-runner --tcp=12434     # note the =; the space form is silently mis-parsed
docker model pull ai/gemma4
docker model run ai/gemma4

The endpoint for Hermes is http://localhost:12434/engines/v1; from inside a container, http://model-runner.docker.internal. Context size is set per model with docker model configure, and the 64K rule applies unchanged.

vLLM — for a GPU server, not a laptop

vLLM is what you run on a Linux machine with an NVIDIA card when several people or processes share the model. It lists Gemma 4 among its supported models — including the 12B’s unified architecture — takes weights from Hugging Face, and serves one model per process at http://localhost:8000/v1.

uv pip install vllm --torch-backend=auto
vllm serve google/gemma-4-12B-it --max-model-len 65536 \
  --enable-auto-tool-choice --tool-call-parser <parser>

The two tool flags are not optional: without them the model’s tool calls come back as plain text and Hermes cannot act on them. Which parser matches Gemma 4’s format was not verified for this guide — vLLM’s list has a functiongemma entry; Hermes’s page names the parsers it has used. Test with one tool call before trusting the setup.

The fourth way: an agent with the runtime inside it

Everything above keeps the model server and the agent as two programs joined at a URL. LM Studio — the desktop app that has run open models on laptops since before agents were the point — now ships a second app that folds the two together. Bionic is, in its own words, “the AI agent designed for open models,” and it is a separate application from LM Studio, not a mode of it. Neither is installed on the machine this guide was measured on, so this section is what the documentation says, and nothing here was timed.

What it changes is where the model runs, chosen per session from one picker:

  • Local — a model downloaded inside Bionic (Settings → Local Models → Explore, with device-fit information beside each file), on the LM Studio runtime — MLX on Apple silicon, llama.cpp elsewhere. No account, no credits, nothing leaves the machine. Gemma 4 12B is in LM Studio’s catalogue with “minimum system memory 7 GB” and tool use listed.
  • Remote — a local model on another of your devices, through LM Link; the other machine does the inference, the conversation stays where you are.
  • Cloud — “frontier open models” (the site names GLM 5.2, Kimi K3, DeepSeek V4 Pro) in LM Studio Secure Cloud, “under Zero Data Retention,” paid in credits from a signed-in account. The docs are careful about the line: local and remote models “do not consume credits and do not require billing set up.”

The agent itself is organised as projects — a folder, and for code a toggle, Allow coding, that lets it search the repository, edit files, use Git and run shell commands in the folder you chose — and sessions inside them, which run in the background when you switch away, open side by side, and fork. Documents it creates are saved as it goes. Voice input is transcribed locally (Voxtral, from Mistral). And skills are the standard Agent Skills format — a SKILL.md with optional support files — which the docs say includes “compatible skills you already use in other apps such as Codex and Claude Code.” The collegica skill this site publishes is that format; nothing in its text is specific to one agent.

Two things to weigh against the convenience. First, the same context-window rule applies to any runtime an agent sits on; Bionic’s docs do not state a minimum, and the honest position is to check what a session is actually given before trusting a long task. Second, LM Studio’s own requirements are narrower than Ollama’s: Apple silicon only (Intel Macs “currently not supported”), macOS 13.4 or newer, 16 GB “recommended” with 8 GB workable “for smaller models and modest context sizes.”

And if you already run Hermes, LM Studio is also just a server. It serves OpenAI-style endpoints on localhost:1234, can start headless at login (lms server start, or the setting in the app), and loads models on demand when a request names one. Hermes lists it as a first-class provider — hermes model → LM Studio, or provider: lmstudio in config.yaml, with an optional LM_API_KEY — so the Ollama walkthrough above transfers with a port number changed. The 64K rule still applies; set the context in LM Studio’s model settings, not in Hermes.

Space, and getting it back

The install is not small, and the numbers are worth having before you start: the model 7.6 GB, Hermes with its own Node and Python about 4 GB, the Desktop app 311 MB, Ollama a few hundred MB. On this machine, 15 GiB free became 721 MiB, and the cleanup that followed — package caches, a Docker build cache — recovered 3 GB without touching anything that mattered.

Ollama’s macOS page lists every file its uninstall must remove; ollama rm gemma4:12b alone gives back the 7.6 GB. Hermes has hermes uninstall, and a profile is hermes profile delete local.

What to do this week

  • Set the context variable before you pull the model. It is the one step with no error message until later.
  • Pull the 12B, run ollama ps, read the two numbers — memory and context — before you install anything else.
  • Give the local agent a profile of its own, and disable the outward toolsets in it. A local model with web enabled is only private until it searches.
  • Try one real task with reasoning on, then off, and time both. Decide from your numbers, not these.
  • Run one task ten times, then once through a hosted model. Compare the outputs, not the benchmarks. Where they agree, local is enough.
  • Keep the gates in the tools. The model that mistyped a rule was stopped once — by a merge that validates. Put the validation where the model cannot talk its way past it.

Sources

Research notes — what was read in full, what was measured, and what was documented but not run — are in the accompanying folder.