A faster model at the same price: what to check before you switch

Claude Sonnet 5.5 and GPT-6.1 Sol arrived at their predecessors' prices. Here's a small, repeatable way to decide whether a model upgrade is safe for your workflows.

Last checked:

Sections

Two of the models most businesses actually run their workflows on were replaced this week, and neither one got more expensive.

Anthropic released Claude Sonnet 5.5 on September 28. It says outputs are more than 30% faster than Sonnet 5, at the same price: $2 per million input tokens and $10 per million output tokens. OpenAI followed within days with GPT-6.1 Sol, also at its predecessor's price, with cached input cut in half.

If you have an inbox sorter, a document extractor, or a customer reply drafter running on either family, the obvious question is whether to switch. My answer is usually yes, after a short test. This post is about that test: what to save, what to compare, and what to do when the new model is better at most things and worse at one that matters.

What actually changed

Here's the short version of both releases, with the vendor's own pages linked. The claims are theirs. I haven't benchmarked either model for this post.

Claude Sonnet 5.5. Anthropic's announcement lists three things a business owner would care about:

Anthropic is also direct about the limit: Opus 5.5 is still "clearly stronger" on complex, open-ended work that needs sustained judgment.

GPT-6.1 Sol. OpenAI's developer announcement keeps GPT-6 Sol's prices, $2 in and $10 out per million tokens, and drops cached input from $0.20 to $0.10. OpenAI reports better coding and computer-use results at a lower cost per task.

Both releases follow the same pattern. The price on the rate card stays put, and the cost of a finished task goes down because the model is quicker and wastes fewer tokens. That's good news, and it's also the part you can't confirm by reading a rate card.

Why "same price" doesn't tell you your bill

You pay per token. What you care about is cost per completed task, and three things sit between those two numbers.

Tokens used per task. A model that reasons longer, writes longer answers, or calls a tool twice can cost more on the same rate card. A model that gets to the answer sooner costs less. The vendor's "up to 30% less per task" is an average over their tests. Yours might be 40% less or 5% more.

The effort setting. Sonnet 5.5 has five effort levels, from Low to Max. Lower settings answer faster with fewer tokens, and higher ones reason longer and check their work more. The default on Anthropic's developer platform is High, while the Claude apps default to Medium. If your workflow never set this explicitly, an upgrade can change how hard the model works on every request.

Retries and corrections. If the new model gets a field wrong more often, someone fixes it by hand or the workflow runs again. That cost never shows up on the model invoice.

Here's some rough math, using made-up numbers. Say a workflow reads 3,000 customer emails a month. Each one uses about 4,000 input tokens and 500 output tokens. At $2 and $10 per million tokens, that's about $39 a month. A 30% saving is roughly $12.

That's not why you'd switch. You'd switch because replies come back faster and fewer of them need a person to fix them. If one in fifty emails gets routed to the wrong team today and the new model makes that one in a hundred, the saved staff time is worth far more than $12. If it makes that one in twenty-five, the "free" upgrade just got expensive.

So the test has to measure correct results, not token prices.

What you actually pay / Per finished task The rate card is one input of four. Only the first box is on the vendor's page. The rest come from running your own work.
Price per tokenOn the rate card. "Same price" means only this didn't change.
Tokens per taskReasoning, answer length, tool calls. Moves with the effort setting.
RetriesMalformed responses that had to run again.
Human fixesWrong fields someone corrected. Never on the model invoice.
Cost per acceptable resultThe number to compare between models.
FIG. 01 A vendor's "up to 30% less per task" is an average over its tests. Your saved cases tell you what the second, third, and fourth boxes do on your work.

Save your test cases before you touch anything

The whole method depends on one habit: keep a small set of real examples with the answer you'd accept for each.

For most small workflows, 20 to 30 cases is enough to start. Pull them from real traffic and make sure the set includes:

For each case, write down what "right" means in a way you can check. "Category is Billing" is checkable. "The summary is good" is not. If a result needs judgment, write two or three specific things it must contain and one thing it must not, such as a promised appointment time the system never confirmed.

Strip or mask customer details you don't need for the test, and keep the file somewhere with the same access controls as the source data.

If you've never done this, the first set takes an afternoon. After that it's a few minutes whenever something goes wrong: add the case, add the right answer, move on. My guide to LangSmith and the rest of the eval stack covers the developer tooling for larger sets. For a small business workflow, a spreadsheet is a fine place to begin.

Run both models on the same set

Now the comparison is mechanical. Run every case through the current model and the candidate, with the same instructions and the same settings, and record four things for each run:

  1. Did it produce an acceptable result? Yes or no, by the rule you wrote down.
  2. How many tokens did it use? Input and output, from the provider's usage numbers.
  3. How long did it take? Wall-clock time for the whole task, tool calls included.
  4. Did it need a retry? A malformed response that had to be re-run counts against the model.

Run each case more than once. Models don't give identical answers every time, and a case that passes four runs out of five is a different problem from one that always fails.

Then put the results side by side. These numbers are invented to show the shape, not measured.

The comparison / 30 saved cases, both models Put the results side by side. The shape of a comparison, with numbers invented to show it. Example numbers, not measured
  • Acceptable resultshigher is better
    27 of 30
    28 of 30
  • Cost per acceptable resultlower is better
    $0.014
    $0.010
  • Median time per tasklower is better
    6.1 s
    4.2 s
  • Retries neededlower is better
    2
    0
  • Cases that got worsethe line people skip
    1 on the candidate
FIG. 02 The candidate wins on every total, and one case still got worse. Read every case whose result changed, in either direction, before you decide.

The line people skip is "cases that got worse." A candidate can fix three failures and introduce one new one, and the totals will hide it. Look at every case whose result changed, in either direction.

Decide with rules you wrote in advance

It's easy to talk yourself into a switch after you've seen a faster response. Write the rules down first. These are the ones I'd use for most business workflows:

The rules / Written before you see the results Four outcomes, decided in advance. Start at the top with your comparison in hand.
Did any case get worse on the candidate?
No
SwitchAt least as accurate, and cost or speed improved.
Yes
Is it an expensive mistake? Money, a customer commitment, a compliance step.
Yes
HoldFaster isn't worth a new failure there.
No
Fix, then switchAdjust instructions you control, re-run the full set.
Better at routine work, worse at the hard cases? Split: routine work to the new model, hard cases stay where they are.
FIG. 03 The person who owns the outcome signs off, with the comparison in front of them.

Also decide who signs off. For a workflow that touches customers, it should be the person who owns the outcome, with the comparison in front of them.

Things that break quietly in an upgrade

Most upgrade problems aren't dramatic. They're small changes in behavior that pass a casual look. These are the ones I check for.

Output format. If your workflow expects a specific structure, test that every response still parses. Anthropic's announcement notes that the pre-release version of Sonnet 5.5 given to an outside evaluator had a structured-output bug, since fixed. That's a normal thing to happen around a launch, and it's exactly the kind of problem a saved test set catches on day one.

Settings that no longer mean the same thing. Anthropic's migration notes say that running Sonnet 5.5 with thinking turned off requires a new setting first. If a developer set your workflow up months ago, ask them to read the migration notes before changing the model name.

More effort isn't always better. Anthropic reports that Sonnet 5.5 scored lower at its Max effort setting than at the next level down on one coding benchmark, because it took on extra work that ran out of time or edited things outside the task. Test the setting you'll actually run. Don't assume the highest one is the safest.

Length and tone. A new model may write shorter replies, longer ones, or more formal ones. If the output goes to customers, read ten of them side by side.

Refusals. Newer models ship with updated safeguards. Anthropic notes that Sonnet 5.5's biology safeguards may flag some legitimate microbiology and virology requests. If you work in a field near a sensitive topic, such as pest control chemicals, medical offices, or security, include those requests in your set.

Caching. If your workflow uses prompt caching to save money, a model change starts the cache from empty, and the cache prices changed too. My post on prompt caching in production covers how to measure that.

Pin the version and keep a way back

Two habits make every future upgrade cheaper.

Name the exact model. Use the specific model name, such as claude-sonnet-5-5, and avoid an alias that moves when the vendor ships something new. With an alias, the upgrade happens to you on the vendor's schedule, and the first you hear of it is a customer complaint.

Make the model a setting. The model name should live in one configuration value, not scattered through the workflow. Then switching is one change, and switching back is the same change in reverse.

When you do switch, don't move everything at once if the workflow is busy. Send a small share of live work to the new model for a few days, or run it in the background on real requests without letting it act, and compare. Keep the old model available until you've seen a normal week go by.

Vendors retire old models on a published schedule, so "never upgrade" isn't an option. A saved test set means the day a model is retired is a routine afternoon and not an emergency.

When you don't need to do any of this

Not every use of AI needs a test set. If you use a chat app to draft a post or think through a decision, and you read the result before anything happens, just use the new model. The apps usually move you to it automatically.

The test matters when a model is wired into a process that runs without someone reading each result: sorting, extracting, routing, replying. The less a person looks at each output, the more the saved cases are doing the looking for you.

What I'd do this week

If you have a workflow running on Sonnet 5 or GPT-6 Sol today:

  1. Find out which exact model name it uses and where that's set.
  2. Save 20 to 30 real cases with the result you'd accept.
  3. Run both models and write down the comparison.
  4. Decide by the rules above, and keep the old model one setting away.

The whole exercise is a few hours the first time. After that, each new release is a quick re-run and a decision you can explain.

This is part of what I do when I build or look after workflow automation and integrations: the test cases come with the workflow, so the next model release is a small job. I'm in Southwest Florida and work remotely across the US. If you have a process running on a model you've never re-checked, tell me what it does.

For where these workflows usually start, see business automation with AI.

More notes

All notes · Contact