Decision models, one month after Jev: how to choose one
Cloudflare Clef, OpenAI's Decisions API, Liquid d1, and Microsoft-Decision-1 all followed Jev within weeks. A developer guide to comparing them on your own cases.
Last checked:
SectionsWhen I wrote about Jev for developers in September, there was one model in the category. TypeSafe called it a System One model: give it context and typed questions, get back a choice, a score, or a yes/no probability, with no generated prose.
A few weeks later there are at least five serious options, two of them from companies that run a large share of the internet's infrastructure. If you build software that routes, filters, or gates work, the question has moved from "is this a real category?" to "which one, and how do I tell?"
This post covers what shipped, what the vendors' numbers do and don't tell you, and a small harness design for comparing them on your own cases. The code here is a design to adapt. I haven't run a head-to-head benchmark for this post, and where I quote a number, it's the vendor's.
What shipped in four weeks
In release order:
Cloudflare Clef and Clef-flash, October 1. Open weights under Apache 2.0, hosted on Workers AI, and described as fully compatible with the Jev API. Clef sits on a frozen Qwen3.8-27B backbone and Clef-flash on Qwen3.5-9B, each with trained adapters and a routing head. Clef has a 64k context window and accepts images. A week later Cloudflare added Clef-omni for audio and video. See Cloudflare's announcement.
OpenAI's Decisions API, October 6, public beta. A hosted endpoint powered by GPT-6 Luna that returns three output types: predicates (the probability a statement is true), choices, and scores. It accepts text and images. Pricing is $0.10 per million input tokens with no output charge. See the beta announcement and the Decisions guide.
Liquid AI d1-3B and d1-omni-600M, October 7. Open-weight models built for small hardware. Liquid says they "don't produce tokens" at all: the answer comes out of a single forward pass. d1-3B takes text and images. The 600M model is an early research release that also takes audio. Both run under llama.cpp. See Liquid's post.
Microsoft-Decision-1, October 9. A hosted decision-scoring model on Microsoft Foundry and OpenRouter, post-trained from Qwen3.5-9B. It returns a probability for each option in a fixed set and costs $0.042 per million input tokens, with output free. See Microsoft's announcement.
- JevTypeSafe's System One model: Choice, Score, and Noul.Hosted
- Cloudflare Clef and Clef-flashJev-compatible API, 64k context, images. Clef-omni for audio and video a week later.HostedApache 2.0Run locally
- OpenAI Decisions APIPublic beta on GPT-6 Luna. Predicates, choices, and scores. $0.10 per million input, no output charge.Hosted
- Liquid d1-3B and d1-omni-600MOne forward pass, no tokens generated. Runs under llama.cpp.LFM LicenseRun locally
- Microsoft-Decision-1On Foundry and OpenRouter. A probability per option, $0.042 per million input.Hosted
Perplexity also published open-weight decider models in the same window, and community models keep appearing. I've kept this post to the four with first-party documentation I could read end to end.
The vocabulary differs. Jev says Choice, Score, and Noul. OpenAI says choices, scores, and predicates. Microsoft says yes/no, multiple-choice, and rating. Underneath, it's the same three shapes: pick one of a fixed set, place something on an ordered scale, or estimate whether a statement is true.
Why everyone built one at once
A generative model answers a classification question by writing tokens, and your code parses them. That's slow, it costs output tokens, and the "confidence" is whatever the model felt like saying.
A decision model skips the writing. It reads the context and the allowed answers, and returns a probability for each. That makes three things possible.
Speed. With no output to generate, latency is mostly the cost of reading the input. Liquid reports 30 ms for one question with d1-3B on an Apple M5 Pro and 8 ms on an RTX 4090. Cloudflare reports a median of about 39 ms for Clef-flash and 209 ms for Clef on its own network. Those are different machines and different measurements, so don't compare them to each other.
Price. Input-only billing at a few cents per million tokens means a decision can run on every request without anyone watching the bill.
A number your code can act on. A probability lets ordinary code decide what happens next: act, ask a person, or try a larger model. Microsoft states the goal plainly: "a 90% prediction should be right about nine times out of 10 on representative cases."
That last sentence has a condition in it. "Representative cases" means cases like the ones the model was tuned on. Yours might not be.
What the vendor numbers tell you
Every launch post has a table where the vendor's model wins. They can't all be right in general, and reading them closely shows why.
Cloudflare reports Clef well ahead of Jev on intent classification benchmarks such as BANKING77. The same post shows Jev ahead on When2Call, which tests whether a tool should be called at all, and on one retrieval benchmark. It also shows Clef-flash scoring 66.77 on a benchmark where Clef scores 97.43. The fast, cheap model isn't a slightly worse copy of the large one. On some tasks it's a different model.
Microsoft says its model had the highest accuracy across 36 benchmarks and changed its answer on 1.3% of perturbed inputs. Liquid says d1-3B leads every model under 10B on a public decision index. All of those are the vendors' own runs. None of them is an independent test.
What I take from this:
- Each model has task types where it's strong and others where it isn't. Routing, tool gating, and relevance filtering are different skills.
- A leaderboard built by one vendor tests the tasks that vendor cared about.
- "Compatible with the Jev API" means the request and response fit the same shape. It says nothing about whether a 0.8 from one model means what a 0.8 from another does.
So the comparison has to happen on your cases.
Put every model behind one interface
The first step is boring and worth doing: your application should never know which decision model answered. It should see one normalized result.
from dataclasses import dataclass
from typing import Protocol
@dataclass(frozen=True)
class Verdict:
option: str # the selected option
probabilities: dict[str, float] # every option, summing to about 1
model: str # exact model and version that answered
latency_ms: float
class Decider(Protocol):
def choose(
self,
state: str,
question: str,
options: dict[str, str], # option name -> description
) -> Verdict: ...
Each provider gets a thin adapter that implements choose. A yes/no question is a choice with two options, and a score is a choice over ordered levels, so one method covers the cases most applications need. Keep provider request details inside the adapter, and read each vendor's current API reference when you write it.
Two rules for the adapters:
- Record the exact model. An alias such as "latest" can change under you. Thresholds you tuned last week belong to the model you tuned them on.
- Always include a fallback option. "None of these" or "unclear" has to be a named option. Without it, the model is forced to spread its probability across answers that are all wrong.
The policy that turns a verdict into an action stays in plain code:
def route(verdict: Verdict, act_at: float, review_at: float) -> str:
top = verdict.probabilities[verdict.option]
if verdict.option == "unclear":
return "review"
if top >= act_at:
return f"act:{verdict.option}"
if top >= review_at:
return "review"
return "escalate"
act_at and review_at are per model. That's the point of the next section.
- PROVIDERSAny decision modelJev, Clef, Decisions API, d1, Microsoft-Decision-1
- ADAPTERchoose()State, question, named options, always including "unclear"
- VERDICToption + probabilitiesPlus the exact model version and the latency
- POLICYroute()act_at and review_at, tuned on your own labeled cases
- ActTop probability at or above act_at.
- ReviewBetween the two thresholds, or the answer is "unclear".
- EscalateBelow review_at. A person or a larger model takes it.
Check calibration on your own cases
A probability is only useful if it's honest. The check is simple: group your test cases by the confidence the model reported, then see how often it was right in each group.
from collections import defaultdict
def calibration(results: list[tuple[float, bool]], width: float = 0.1):
"""results: (reported probability of the chosen option, was it correct)."""
buckets: dict[float, list[bool]] = defaultdict(list)
for probability, correct in results:
floor = min(int(probability / width) * width, 1 - width)
buckets[round(floor, 2)].append(correct)
return {
floor: (len(hits), sum(hits) / len(hits))
for floor, hits in sorted(buckets.items())
}
Feed it a few hundred labeled cases and you get, for each confidence band, how many cases landed there and what share were right. If the 0.9 band is right 92% of the time, you can act on it. If it's right 70% of the time, the model is overconfident on your data and your act_at threshold needs to move up, or that model isn't the one.
calibration() looks like when drawn. Bars that reach the dashed line mean the reported confidence can be trusted.
Illustration, not measured data
Things I'd insist on:
- Use your own inputs. Real tickets, real documents, real tool-call traces, with the answer a person agreed on. Fifty cases is a start. A few hundred makes the bands meaningful.
- Include cases with no right option. They test whether the fallback gets used.
- Tune thresholds per model. A threshold copied from one model to another is a guess.
- Hold some cases back. Tune on one part of the set and confirm on the rest, or you'll fit the thresholds to noise.
The Jevidence companion project from the Jev post already has this structure: questions, a separate routing policy, and constructed cases. Adding a second backend there is a natural extension.
Measure latency where your code runs
Latency claims are the least portable numbers in any launch post. A median measured inside the vendor's network doesn't include your network hop, your cold starts, or your payload size.
Measure from your own environment, with your real context sizes, and record the slow tail as well as the median. A router that's fast at the median and slow one time in twenty will be felt by users.
Batching matters too. Jev and Clef let you ask several questions about one piece of context in one request, and the other APIs have their own batching rules. If your design asks three questions per item, compare one batched call against three single calls for each provider. Liquid's figures show the same effect on local hardware, where packing many items together raises throughput well above one-at-a-time speed.
For local models, measure on the machine that will actually serve them. A model that takes 30 ms on a new laptop may be fine on a workstation and too slow on a shared CPU server.
Test stability, not just accuracy
A decision that flips when the input is reworded is a problem even if the average accuracy looks fine. Microsoft reports this directly, which I appreciate: 1.3% of decisions changed under its perturbation tests, with none from reordered options or paraphrased descriptions.
You can run the same kind of check cheaply:
- Reorder the options. The answer shouldn't depend on which option came first.
- Reword the option descriptions. Keep the meaning and change the phrasing.
- Add irrelevant context. A signature block or a quoted earlier email shouldn't change a routing decision.
- Run it twice. The same request should give the same verdict.
Count how many verdicts change. For anything that triggers an action, I'd also look at how far the probability moved, because a case sitting right at your threshold will flip in production even if it didn't in the test.
Hosted or local
The four releases split evenly, and the split matters more than any benchmark.
Hosted (OpenAI Decisions, Microsoft-Decision-1, Clef on Workers AI, Jev) means no hardware, quick setup, and a per-token price. Your context leaves your network on every call. Read each provider's retention terms. Cloudflare says it doesn't read, store, or train on hosted requests. Check the others for your own account type.
Local (Clef and Clef-flash weights, d1-3B, d1-omni-600M) means the data stays put and the marginal cost is electricity. You take on serving, updates, and monitoring. Licenses differ: Clef is Apache 2.0, while Liquid's models use its own LFM License, which you should read before shipping a product on it.
A reasonable middle path is a small local model for the first pass and a hosted model for the cases it's unsure about. That only works if the local model's uncertainty is trustworthy, which is the calibration check again.
A shortlist by situation
Here's how I'd narrow it down before testing. This is a starting point for a trial, not a ranking.
- Already on Cloudflare Workers, or want open weights with a permissive license: start with Clef. Try Clef-flash only on tasks where you've confirmed it holds up.
- Already calling OpenAI's Responses API: the Decisions API is the smallest change. It's a public beta, so expect details to move.
- Microsoft shop with Foundry in place: Microsoft-Decision-1, especially for grading model outputs and gating agent actions, which its launch post focuses on.
- Data can't leave the building, or you need answers on a device: d1-3B, with the 600M model as an experiment if you need audio.
- Already built on Jev: stay until a test on your cases says otherwise. Clef's API compatibility makes that test cheap to run.
Whichever one you pick, keep the adapter. This category went from one model to five in a month, and it won't stop there.
What stays the same
The advice from the Jev post holds for all of them. The model judges. Your code decides. A verdict is evidence that a tested, ordinary function turns into an action, with a named fallback and a review path for the uncertain cases.
A typed answer can still be the wrong answer. Schemas guarantee the shape. Only your labeled cases tell you about the judgment.
If you're adding a decision step to a product, a retrieval pipeline, or an agent and want help designing the policy and the evaluation around it, that's the kind of custom software and integration work I do. Tell me what you're building.
Related: LangSmith and the rest of the eval stack for the wider testing picture, and prompt caching in production for where a cheap decision step fits in front of a larger model.