LoFiCS — Local First, Cloud Second

Local-First Cloud-Second proxy using Ternary Bonsai 27B as the local model and Claude as the cloud model.

Serves the local 27B (via llama.cpp) as Hermes's default model, and transparently escalates hard queries to cloud Claude Opus (via your claude CLI subscription seat). Hermes sees one normal OpenAI endpoint; the escalation decision happens invisibly inside the router.

Hermes ──/v1/chat/completions──▶ router (:8000) ──▶ llama-server 27B (:8080)   default
 provider: custom                     │
 base_url: :8000/v1                   └── on escalation ──▶ claude -p --model opus

Components

• ~/projects/ternary/llama-prism-b8846-d104cf1/ — prismML llama.cpp fork (ROCm/gfx1201) + Ternary-Bonsai-27B-Q2_0.gguf
• router.py — the escalating proxy (Python stdlib only, no deps)
• run.sh — starts llama-server + router, health-checks both

Requirements

What	Why	How to get it
GPU with ≥8 GB VRAM (or CPU)	run the 27B locally	ROCm (AMD), CUDA (NVIDIA), Vulkan, Metal (Mac), or CPU all work. The Q2_0 27B needs ~7.4 GB VRAM fully offloaded (measured on an RX 9070 XT / 16 GB). Less VRAM → lower BONSAI_NGL to offload fewer layers, or run on CPU (slow).
GPU driver/toolkit	GPU inference	AMD: ROCm 7.2 (rocminfo/rocm-smi on PATH). NVIDIA: recent driver + nvidia-smi. setup.sh auto-detects which is present. None needed for CPU/Metal.
python3 ≥ 3.8	runs router.py (stdlib only, no pip deps)	Preinstalled on macOS/Linux, or your package manager. Tested on 3.14.
claude CLI (Claude Code) logged into a subscription/Team seat	the cloud escalation target	npm install -g @anthropic-ai/claude-code, then claude auth login (browser OAuth for Pro/Max/Team). Verify with claude auth status — it must not say "API account" if you want flat-rate escalations.
curl + tar	setup.sh downloads/extracts	Preinstalled almost everywhere. unzip also needed on Windows/HIP .zip assets.
Hugging Face access to prism-ml/Ternary-Bonsai-27B-gguf	the model weights (~7 GB)	Public repo — no token needed. setup.sh pulls it via hf/aria2c/curl.
A running Hermes	the client that uses the router	Points its primary model at the router (provider: custom).

Recommended (auto-used if present, graceful fallback if not):

Tool	Speeds up	Install
aria2c	8-way parallel model download (much faster than curl)	apt install aria2 / brew install aria2 / pacman -S aria2
jq	robust GitHub-API release parsing in setup.sh	apt install jq / brew install jq / pacman -S jq
hf (huggingface_hub CLI)	resumable model download	pip install -U "huggingface_hub[cli]"

Run

cd ~/projects/ternary/router      # wherever you cloned LoFiCS
./setup.sh        # one-time: fetch prism binaries + the 27B GGUF (auto-detects GPU backend)
./run.sh          # starts both, tails the router log
./run.sh stop     # stops both

First-time setup (setup.sh)

setup.sh fetches the two big artifacts that are deliberately NOT in git: the prismML llama.cpp fork binaries and the ~7 GB GGUF. It's idempotent (skips anything already present) and auto-detects the right GPU backend:

• ROCm if rocminfo/rocm-smi exist, CUDA if nvidia-smi, else Vulkan/CPU; macOS gets the Apple-Silicon or Intel build.
• Pulls the latest prism release by default (so you always get the newest ROCm-capable build). Pin one with BONSAI_PRISM_TAG=prism-b9570-0ad1dab.
• Force a backend with BONSAI_BACKEND= (rocm | cuda-12.4 | cuda-12.8 | cuda-13.3 | vulkan | cpu | macos-arm64 | macos-x64).

Why the prism fork specifically? The ternary Q2_0 weights need prismML's custom kernels — upstream llama.cpp (and other forks) can't decode them yet (upstream PR planned). The 1-bit Q1_0 Bonsai is merged upstream if you ever want a stock-llama.cpp path, at a lower quality operating point.

run.sh reads machine-specific paths from env vars (with defaults), so the same repo works on any host. Override per machine as needed:

Var	Default	Meaning
BONSAI_BIN_DIR	~/projects/ternary/llama-prism-b8846-d104cf1	llama.cpp fork dir (has llama-server + .sos)
BONSAI_MODEL	Ternary-Bonsai-27B-Q2_0.gguf	GGUF filename (rel to bin dir) or absolute path
BONSAI_SERVER_PORT	8080	llama-server port
BONSAI_ROUTER_PORT	8000	router port
BONSAI_NGL / BONSAI_CTX_SIZE	99 / 8192	GPU layers / context size

Hermes is already configured to use it (~/.hermes/config.yaml):

model:
  default: bonsai-router
  provider: custom
  base_url: http://127.0.0.1:8000/v1
  key_env: BONSAI_ROUTER_KEY   # placeholder; localhost needs no real key

To revert Hermes to cloud Opus:

hermes config set model.default claude-opus-4-8
hermes config set model.provider anthropic
hermes config unset model.base_url
hermes config unset model.key_env
# or restore: cp ~/.hermes/config.yaml.bak.bonsai.<timestamp> ~/.hermes/config.yaml

How escalation is decided (balanced)

For each request the router calls the local 27B first (non-streaming, with a generous reasoning budget), then escalates to Opus if any of:

1. Empty / truncated — the model hit its token budget mid-reasoning and produced no final answer (finish_reason: length). Strong "out of depth" signal.
2. Sentinel — the model was instructed to emit <ESCALATE> when it can't answer well, and did.
3. Low self-confidence — a cheap second local call rates the answer 0–1; below CONF_THRESHOLD (0.6) it escalates.

Otherwise the local answer is returned. Escalations go through claude -p --model opus on your subscription (no metered API key).

Tuning (env vars, all optional)

Var	Default	Meaning
BONSAI_CONF_CHECK	1	enable the self-confidence backstop
BONSAI_CONF_THRESHOLD	0.6	escalate below this confidence
BONSAI_MIN_LOCAL_TOKENS	4096	reasoning headroom so thinking completes
BONSAI_SENTINEL	<ESCALATE>	the "I'm stuck" token
BONSAI_CLAUDE_MODEL	opus	cloud model alias for claude -p
BONSAI_ESCALATE_ON_ERROR	1	escalate if the local server errors
BONSAI_VERBOSE	1	log routing decisions

Make it more conservative (fewer Opus calls): BONSAI_CONF_CHECK=0 (sentinel + truncation only). More aggressive: raise BONSAI_CONF_THRESHOLD.

Known characteristics / gotchas

• Opus latency: a claude -p escalation is ~5–40s (CLI cold-start + reasoning). Fine because escalations are meant to be rare. Keep them rare.
• Reasoning budget: max_tokens on this model counts reasoning + answer together; it reasons a lot, so small budgets yield empty answers. That's why the router floors the local budget at BONSAI_MIN_LOCAL_TOKENS.
• llama.cpp fork flags differ from upstream: use llama-completion (not llama-cli -no-cnv) for one-shots; the server needs LD_LIBRARY_PATH set to its own dir. Supports qwen35 arch and --reasoning auto (splits thinking into reasoning_content).
• No logprobs in this fork, so confidence is self-assessment, not token probability.
• Reaching Opus: this only works because the claude CLI is logged into a Team/subscription seat (claude auth status). An external proxy cannot reach Nous-hosted Opus (fingerprint + hourly key rotation).

Verified working

• Easy prompt → stays local (correct answer)
• Hard prompt (finite integral domain proof) → escalates → Opus returns rigorous proof
• Streaming path → clean SSE with [DONE], terminates
• End-to-end through hermes -z → routed via local, confirmed in router log