Evidence · 2026-10-23 · batch-20261023T165405Z

One task, 13 models, and a gate that said no twice before it said yes.

Support-ticket triage on a pack we wrote: 160 messages, 9 labels. Gemma 4 E4B cleared every check and passed the gate on 108 of 110 sealed fixtures. Every number below is read from the run's evidence files.

The task

160 support messages, 9 labels

Refunds, invoices, sign-in, deletion, bugs, features, integrations, plan changes, and "not ours". 50 fixtures for the compile, 110 sealed for the gate. Written by us: no customer data.

The contract

Floor 0.90 · p95 under 2,000 ms

An answer passes only when it is exactly a label. Off-topic messages answered as if they were ours may not exceed 10%, proven by a bound, so at this sample size the gate allows none.

The fleet

Local engines, Claude, and Grok

Five small open models on Ollama on a Mac, four Claude tiers on Anthropic's API, and four Grok 4.7 effort levels through a bridge to a Cursor plan. Each runs with the settings it would be served with.

Step 1 · Compile

Every model, every check.

The screen asks for an 80% Wilson bound on the dev slice to clear the floor and the off-topic limit, so a model that only touches the floor is not assigned. The cheapest survivor by measured cost per call gets the draft.

Model	Right on dev	p95	Off-topic accepted	Per 1,000 calls	Verdict
Qwen 3.5 2Blocal, on Ollama	36/50	190 ms	3/10	$0.0027	below floor pass rate 0.72 (36/50), 80% lower bound 0.66, is below the floor 0.90
Granite 4.2 3Blocal, on Ollama	45/50	196 ms	2/10	$0.0028	below floor pass rate 0.90 (45/50), 80% lower bound 0.86, is below the floor 0.90
Qwen 3.5 4Blocal, on Ollama	47/50	236 ms	2/10	$0.0040	off-topic leak out-of-scope accepts 2/10, 80% upper bound 32%, is over the 10% limit
Gemma 4 E4Blocal, on Ollama	49/50	311 ms	0/10	$0.0042	assigned
Granite 4.2 8Blocal, on Ollama	48/50	192 ms	2/10	$0.0070	off-topic leak out-of-scope accepts 2/10, 80% upper bound 32%, is over the 10% limit
Claude Haiku 5.5Anthropic API	50/50	2,106 ms	0/10	$0.040	too slow p95 latency 2,106 ms is over the 2,000 ms ceiling
Claude Sonnet 5.5Anthropic API	50/50	2,352 ms	0/10	$0.808	too slow p95 latency 2,352 ms is over the 2,000 ms ceiling
Claude Opus 5.5Anthropic API	50/50	2,075 ms	0/10	$1.62	too slow p95 latency 2,075 ms is over the 2,000 ms ceiling
Claude Fable 5.1Anthropic API	50/50	3,341 ms	0/10	$4.04	too slow p95 latency 3,341 ms is over the 2,000 ms ceiling
Grok 4.7 lowCursor plan, via a bridge	50/50	12,421 ms	0/10	plan	too slow p95 latency 12,421 ms is over the 2,000 ms ceiling
Grok 4.7 mediumCursor plan, via a bridge	50/50	10,723 ms	0/10	plan	too slow p95 latency 10,723 ms is over the 2,000 ms ceiling
Grok 4.7 highCursor plan, via a bridge	50/50	10,493 ms	0/10	plan	too slow p95 latency 10,493 ms is over the 2,000 ms ceiling
Grok 4.7 xhighCursor plan, via a bridge	50/50	10,508 ms	0/10	plan	too slow p95 latency 10,508 ms is over the 2,000 ms ceiling

50 dev fixtures each. Floor 0.90, p95 ceiling 2,000 ms, out-of-scope limit 10%, screened on 80% Wilson bounds. Batch batch-20261023T165405Z.

Step 2 · Gate

Three drafts. Two refused.

Each draft ran once on the sealed slice, which the compile never reads. The first run screened on point estimates and assigned a model that barely cleared the floor on dev; the gate refused it, and the screen moved to bounds. The second draft leaked one off-topic message. The third passed.

Qwen 3.5 4Brun 1 · 95/110

refused: lower bound 0.80 is under the floor

gpt-oss 20Brun 2 · 109/110

refused: 1 of 30 off-topic messages accepted

Gemma 4 E4Brun 3 · 108/110

passed: lower bound 0.947, 0 of 30 off-topic accepted

0.750.800.85floor 0.900.951.00

Sealed slice, 110 fixtures. The bar runs from the one-sided 95% Wilson lower bound to the observed pass rate. The gate also refuses any draft that accepts an off-topic message at this sample size.

Refused · Qwen 3.5 4B

46/50 on dev, 95/110 on sealed.

pass rate lower bound 0.80 (95/110) is below the floor 0.90.

One of the off-topic messages it accepted:

"Write a Python script that sorts a list."

Refused · gpt-oss 20B

49/50 on dev, 109/110 on sealed.

out-of-scope accepts 1/30, 95% upper bound 14%, is over the 10% limit.

One of the off-topic messages it accepted:

"Our Shopify store's checkout is broken, please fix it."

Passed · Gemma 4 E4B

108/110 on sealed, lower bound 0.947.

0 of 30 off-topic messages accepted, an upper bound of 8.3%. p95 241 ms. The fingerprint matched the compile.

Step 3 · Enforce

What the application sees.

An ordinary OpenAI-style call with one extra header. The enforcer picks the model and its settings from the passed policy and names the model that served the call. A task the policy does not cover is refused before any model is called.

These calls were re-sent through the enforcer with the run's passed policy after the run. The Slack message is borderline: during the run it came back integration_help, here feature_request. That variation is why the gate measures a slice, not one answer.

POST /v1/chat/completions  x-sllim-task-id: support-triage  user: Hi team, our card was charged twice for the October renewal. Can you send one of those payments back?  → 200 billing_refund x-sllim-model-id: local-gemma-e4b · 190 ms POST /v1/chat/completions  x-sllim-task-id: support-triage  user: Is there a way to push paid invoices into our Slack channel automatically?  → 200 feature_request x-sllim-model-id: local-gemma-e4b · 185 ms POST /v1/chat/completions  x-sllim-task-id: support-triage  user: Can you recommend a good pizza place near the office?  → 200 out_of_scope x-sllim-model-id: local-gemma-e4b · 185 ms POST /v1/chat/completions  x-sllim-task-id: support-reply  user: Write a friendly reply to this customer.  → 409 task_unassigned no model was called

The policy it served.

Version 1, passed. The binding constraints are the checks that removed cheaper models; the rejected list keeps every other model and why it lost.

policy.json · 8 more rejected models not shownpassed

{
  "id": "support-triage-policy",
  "version": 1,
  "status": "passed",
  "assignments": [
    {
      "task_id": "support-triage",
      "model_id": "local-gemma-e4b",
      "prompt_id": "triage-support-prompt-1",
      "fallback_model_id": null,
      "constraint_ids": [
        "floor",
        "out_of_scope",
        "latency"
      ],
      "batch_id": "batch-20261023T165405Z",
      "model_fingerprint": "sha256:e3cfce764ed72c018420d2b7a4dddf216c14489c79d90ed3218f5726c7e62c01",
      "rejected": [
        {
          "model_id": "local-qwen-2b",
          "reason": "pass rate 0.72 (36/50), 80% lower bound 0.66, is below the floor 0.90; out-of-scope accepts 3/10, 80% upper bound 43%, is over the 10% limit"
        },
        {
          "model_id": "local-granite-3b",
          "reason": "pass rate 0.90 (45/50), 80% lower bound 0.86, is below the floor 0.90; out-of-scope accepts 2/10, 80% upper bound 32%, is over the 10% limit"
        },
        {
          "model_id": "local-qwen-4b",
          "reason": "out-of-scope accepts 2/10, 80% upper bound 32%, is over the 10% limit"
        },
        {
          "model_id": "local-granite-8b",
          "reason": "out-of-scope accepts 2/10, 80% upper bound 32%, is over the 10% limit"
        }
      ]
    }
  ]
}

The bill

A month of this task, priced at measured tokens.

Each model's cost is its measured tokens per call times its price. Claude is at list price. The local model's price is a cost we set for hardware time; a customer would put in their own.

1,000,000 calls a month, measured tokens per call

Claude Fable 5.1too slow

$4,039

Claude Opus 5.5too slow

$1,615

Claude Sonnet 5.5too slow

$808

Claude Haiku 5.5too slow

$40.39

Gemma 4 E4Bpassed the gate

$4.19

Claude at list price; Gemma 4 E4B at the fleet's operator-set hardware price. Every Claude model scored 50/50 on dev and missed the p95 ceiling.

What this does not prove

Read these before quoting a number.

The run shows the mechanism working end to end on real models. It is not a customer result, and the savings on a real workload depend on its tasks, its floor, and its fleet.

Run it on your task

Caveats

The pack is ours. 160 synthetic messages, written to be realistic and to include hard cases, are not a customer's traffic. The first design partners replace it.
The sealed slice was read three times during development, once per draft. A customer gets a fresh sealed set; a repeated look makes a pass slightly easier than a single one.
Latency decided against Claude here. Every Claude tier scored 50/50 on dev but missed the 2,000 ms ceiling from our network. A different ceiling or region could change the ranking; the compile would show it.
Grok was measured through a bridge to Cursor's agent on a Cursor plan, not through xAI's API. Its latency includes process start-up and a long agent prompt, so it says nothing about Grok's own API speed.
The local model's cost is an operator-set hardware price, $0.015 per million input tokens and $0.03 per million output tokens. It is an assumption, not a bill.
Results vary between runs: the same model on the same dev fixtures moved by up to three answers across our runs. The gate's bounds are built for that; a single answer is not.

Reproduce it from the repository: bun run demo:triage --allow-paid. The evidence files for all three runs are kept in the repository beside the code.