Home / Blog / Prompt caching in production: stable context, real costs, and where Jev fits

Prompt caching in production: stable context, real costs, and where Jev fits

By Peak Evergreen LLC

A practical developer guide to prompt caching: stable prefixes, provider differences, cost accounting, invalidation, testing, and optional Jev routing.

Last checked: September 22, 2026

An assistant receives a new question. The request includes instructions, reference material, conversation history, and the question itself. On the next turn, much of that input may be repeated.

Some of that repeated work is necessary. Some can be reused. The difficult part is knowing which is which, and proving that the reuse actually helps.

Prompt caching becomes useful when repeated input is organized into a stable prefix that the provider can reuse. Making that work in production requires more than enabling an API option: prompt assembly, expiration, usage accounting, and evaluation all affect the result. The examples below illustrate general engineering patterns and hypothetical costs.

A small classifier such as Jev can also help choose a request profile or select relevant evidence before a larger model runs. Later sections explore when that extra step helps, how to test it, and when ordinary code is enough.

First, name the thing you are caching

“We added caching” is too vague to evaluate. Three common forms of reuse solve different problems:

• Embedding caching: Reuse a previously computed vector for the same input and compatible embedding model. This avoids another embedding request, not completion input processing.
• Provider prompt caching: The model provider can reuse computation for a matching prefix of a new request. The application still requests a new completion.
• Response caching: An application could return a previously generated answer. That needs its own rules for identity, authorization, freshness, and invalidation. It is a separate design from the prompt-caching path discussed here.

A semantic response cache adds another question: is a similar request equivalent enough to reuse the answer? “What is my balance?” cannot be answered from a neighbor's cached response just because the wording matches. Provider prefix reuse does not solve that application problem.

Keep these layers named separately in code and telemetry. A high embedding-cache hit rate says nothing about how many prompt tokens the completion provider reused.

Build a request around a reusable prefix

A general request flow makes the caching boundary explicit:

1. Choose the model and a versioned prompt template.
2. Assemble stable instructions, tool definitions, and reusable reference material in a consistent order.
3. Place changing context and the current question after the reusable prefix, using appropriate message roles.
4. Configure cache boundaries where the provider supports them.
5. Send the request and generate a new answer.
6. Record actual cache reads, writes, total cost, and latency.

The important separation is between what the application knows, what it asks the model, and what the provider reports it processed. Prompt assembly can be deterministic even when provider cache availability is not. A local token estimate helps with planning, but reported usage is needed to assess actual reuse.

Keep prompt versions visible in these measurements. Otherwise a rollout that changes the reusable prefix can look like an unexplained drop in cache performance.

Put stable instructions ahead of changing context

A useful starting pattern is to separate stable instructions from changing evidence. Keep the reusable content in an unchanged opening segment, then add retrieved passages, conversation updates, and the current question in later segments. Use the roles and content-block structure appropriate to the model API.

For APIs with explicit cache boundaries, place a supported breakpoint after the content you expect to reuse. For automatic caching, preserve the matching prefix and inspect reported usage. Avoid flattening stable instructions and fresh evidence into a single changing instruction block when that prevents a useful boundary.

The goal is to let a new question or retrieved passage change the suffix without rewriting the reusable prefix. This does not guarantee a hit: the prefix must still meet the provider's requirements and remain available.

“Static” is relative to a request family. Different tasks, languages, or tool sets can require different prefixes. Each distinct prefix can have a different reuse pattern. A service with ten thousand requests may still have little repetition within any one prefix.

This layout favors reuse of the shared instructions. It does not automatically maximize reuse of conversation history: if changing retrieval context appears before that history, a longer matching prefix can end at the changed context. Consider additional supported boundaries or a different message layout only after checking model behavior and measuring the benefit. Do not move evidence into a misleading role just to improve cache statistics.

Provider details belong in the adapter

The shared service should express a reuse intention. The adapter must translate that intention into the selected provider's actual API. Forwarding one vendor's cache field to every model is not portability.

OpenAI: Current documentation distinguishes GPT-5.6 and later from earlier models. The newer family supports explicit breakpoints and caching modes, has a 1,024-token minimum, and bills cache writes as well as reads. Earlier model behavior differs. Chat Completions and Responses integrations also expose different request and usage shapes. Verify the exact model, endpoint, SDK, and usage fields when implementing caching. OpenAI prompt caching.

Anthropic: A cache breakpoint covers the prefix through that content block, including preceding tool definitions and system content. The default lifetime is five minutes; a one-hour option has different write pricing. Minimum cacheable length depends on the model. Choose a breakpoint that includes useful repeated content and meets the selected model's requirements. Claude prompt caching.

Google: Gemini distinguishes implicit caching from explicit cache resources. Its current Interactions API documentation supports implicit caching, while explicit caching is available through generateContent. Match the integration to the endpoint you actually use. Normalize the reported usage before comparing cache performance across providers. Gemini context caching.

Put these capabilities in a versioned provider contract: supported cache controls, minimum qualifying length, retention choices, usage normalization, and pricing source. An unsupported field should not silently become a promise of reuse.

Make prompt stability testable without a model call

Start with a pure assembly test. It should establish that changing the evidence or question leaves the intended static segment unchanged, and that changing the instruction version changes it.

This small example demonstrates that invariant. It is not an SDK integration or a full prompt serializer, and its short strings do not meet a provider's cache minimum. The digest is a diagnostic fingerprint, not a response-cache key or proof of a provider hit.

from dataclasses import dataclass
from hashlib import sha256


@dataclass(frozen=True)
class PromptParts:
    static: str
    evidence: str
    question: str

    @property
    def static_fingerprint(self):
        return sha256(self.static.encode("utf-8")).hexdigest()


def assemble(*, instructions, revision, evidence, question):
    return PromptParts(
        static=f"Instruction revision: {revision}\n{instructions}",
        evidence=evidence,
        question=question,
    )


def check(condition, message):
    if not condition:
        raise AssertionError(message)


base = {"instructions": "Answer from authorized evidence.", "revision": "v3"}
first = assemble(**base, evidence="Document A", question="How do I reset it?")
second = assemble(**base, evidence="Document B", question="What changed?")
changed = assemble(
    **{**base, "revision": "v4"},
    evidence="Document B",
    question="What changed?",
)
check(first.static == second.static, "Dynamic input changed the prefix")
check(first.evidence != second.evidence, "Fresh evidence was lost")
check(first.static_fingerprint != changed.static_fingerprint,
      "Instruction revision did not change the prefix")
print("3 prompt assembly checks passed")

At the real adapter boundary, extend this to the serialized request. Test tool-definition order, system-block order, cache markers, model settings, and conversation history. Stable JSON object construction is useful, but never sort a message sequence whose order carries meaning.

Also include a test that would fail if changing evidence were accidentally concatenated into the reusable instruction block. A cache optimization is much easier to maintain when a future refactor can break a specific test.

Invalidation is part of correctness

A prompt revision should change when its instructions change. A tool schema revision should change when its accepted arguments or behavior change. A model migration should create a distinct measurement group. Record these versions together so a sudden drop in reuse is explainable.

Keep per-request timestamps, trace IDs, random experiment names, and current user data outside the reusable instruction segment. If the model needs today's date, include it where the adapter and task semantics allow it without rewriting unrelated instructions. Do not remove necessary context to manufacture a cache hit.

Provider caching and application caches need different invalidation rules. A provider receiving changed evidence should process the changed suffix. An application returning a stored answer may never reach the provider, so it must check whether permissions, source documents, or the underlying business state have changed.

For example, a support answer about a return policy can become invalid after a policy update. A cached embedding for the old policy does not become the new policy's embedding. A response cache needs a source revision or an invalidation event; a TTL alone expresses how long you tolerate possible staleness, not whether the answer is correct.

Normalize usage before calculating savings

A single “cached tokens” counter is not a complete cost model. Providers can distinguish cache reads from writes and define total input differently. Normalize those meanings before applying prices or comparing providers.

Claude provides a concrete trap: its input_tokens excludes cache-read and cache-creation tokens. Total input is the sum of all three fields. Treating cache reads as a subset of input_tokens produces the wrong arithmetic. Claude Messages usage fields.

Use disjoint accounting categories: uncached input, cache-read input, cache-write input, and output, with separate write categories when retention affects the price. Keep raw provider usage alongside the normalized record so a mapping can be audited or corrected.

The calculation then becomes:

estimated cost =
    uncached_input_tokens * uncached_input_rate
  + cache_read_tokens     * cache_read_rate
  + cache_write_tokens    * applicable_cache_write_rate
  + output_tokens         * output_rate

Use rates in the same units, and add any applicable storage, tool, or other charges separately. Price the actual provider and model used after fallback, not the originally requested one. Keep pricing revisions so old estimates can be explained.

If a dashboard estimates the cost of individual prompt components, label that allocation as an estimate. A token-count-based split does not establish exactly which passages the provider reused.

Unknown cache usage also needs its own state. Retain “not reported” separately from a confirmed zero-read request; otherwise an integration that omits a usage field can look like a consistently cold cache.

Calculate the break-even point for your traffic

For a fixed reusable prefix, let B be its ordinary input-processing cost, W the cost of writing it once, R the cost of reading it once, and n the total number of requests served by one write and subsequent reads before expiry. Ignoring unchanged suffix and output costs, caching helps when:

W + (n - 1) * R < n * B

As a hypothetical example, suppose B = 1, W = 1.5, and R = 0.2 cost units. Five uncached requests cost 5 units for that prefix. One write and four reads cost 2.3. These are illustrative numbers, not a vendor price quote.

Now change the traffic: each of the five requests uses a different prefix and none is reused. Five writes cost 7.5 units under those assumptions. The price of a read is attractive only when a read happens. Longer retention or prewarming needs the same expected-reuse calculation.

Measure token-weighted reuse as well as request-level hits. Reusing a small prefix on every request can matter less than reusing most of a large prompt on half the requests. Keep cold starts, expired entries, and changed-prefix requests visible instead of reporting only a warmed benchmark.

Time to first token and total task latency also differ. Reusing input computation will not eliminate retrieval latency, a slow external tool, long output generation, or a second model call. Report the full task's p50 and p95 latency beside provider timing and cost per successful outcome.

Where Jev could fit before the main completion

Jev is useful to consider when selecting the next workflow requires interpreting text. TypeSafe's intent-routing pattern separates a model's classification from the program that chooses a handler. The Jev developer guide develops the same idea: let Jev judge and let your code decide. TypeSafe intent routing.

For example, a document assistant could use a small set of request profiles, such as summarize, compare, and answer-question. Each profile selects a versioned instruction template; model selection, tools, and generation settings are explicit configuration choices. Application code maps an accepted classification to one of those profiles.

The proposed sequence is: ask Jev for a typed classification, validate the result, apply a tested routing policy, then assemble the chosen profile's request. A low-confidence or unsupported result takes a defined fallback. A transport error takes an error path, not a fabricated classification.

Keep the profiles finite and controlled by code. Treat the classifier's choice as a lookup into an allowlist rather than letting it generate a new instruction prefix for every request. Its confidence is a statistic derived from its answer distribution, not a guaranteed probability that your business decision is correct. Set thresholds from reviewed cases and the cost of mistakes. TypeSafe confidence.

Name the baseline. Compared with generating a custom instruction prefix for every request, routing to a few stable profiles can improve opportunities for exact-prefix reuse. Compared with one suitable universal prefix on the same model and compatible configuration, separate profile prefixes divide that opportunity. Routing is not a caching improvement by itself, and even a shared prefix does not guarantee a hit.

Three conditions determine whether the profile-specific content is reusable:

• Traffic within the cache lifetime: Each profile needs repeat requests while its entry is available. With a five-minute lifetime, a profile requested every twenty minutes is a poor reuse candidate. Anthropic defaults to five minutes and offers a one-hour option with different write pricing.
• Enough eligible input: The prefix through the intended boundary must meet the selected model's minimum. Do not assume 1,024 tokens universally; Claude's documented thresholds vary by model. A short profile instruction can still qualify when preceded by enough shared content.
• An unchanged prefix: Keep content and ordering identical between revisions. A timestamp, instruction edit, or reordered tool definition can prevent matching from the changed point onward. An earlier unchanged prefix may still be reusable at an eligible boundary.

Check the selected model's cache limits and lifetime. Too many profiles, or equivalent requests split across profiles, spreads traffic across more entries. Labels changing appropriately between different tasks is not itself the problem.

Put shared content first. Arrange common instructions and tool schemas before profile-specific material, within the API's supported ordering. Add a breakpoint at the common boundary where supported and eligible, then another at a useful profile boundary. This preserves opportunities for reuse across routes as well as within each route. Different models or tool schemas may prevent that sharing.

Distinguish cache-affecting configuration from sampling controls. Anthropic documents that changing tool definitions invalidates subsequent content; tool_choice changes affect message blocks, while thinking changes have model-dependent effects on earlier blocks. A sampling-only option such as temperature does not inherently change the instruction prefix, so profiles need not duplicate it merely to sample differently. Verify the selected API's behavior rather than treating every setting as a separate cache identity. Claude cache invalidation.

Count the classifier request too. Keep its question definitions and option descriptions stable, with changing input separate. For a classifier API that supports prompt caching, apply its own length, boundary, and lifetime rules. This is not an assertion that Jev exposes that capability; budget its full call unless documented support and reported usage establish a discount. Include classifier latency and retries in the comparison. The benefit must come from a better workflow, not from adding a model call just to label requests.

Jev can also help select evidence, with limits

A second candidate is passage filtering after authorized retrieval. Give Jev a bounded set of candidate passages and typed relevance questions, then let code select evidence within a context budget. TypeSafe provides a RAG passage classification cookbook.

This can reduce irrelevant context sent to the larger model. It also creates a new failure mode: discarding the one passage needed for the answer. Evaluate recall of necessary evidence, including exceptions and qualifications, before celebrating fewer tokens. Preserve document identifiers and reasons for selection so failures can be inspected.

With a stable-prefix layout, different passage selections can remain after the reusable instruction boundary. Jev does not need to make every suffix identical. Relevant, current evidence is more important than forcing repeatable context.

Pin Jev's version when tuning this policy, version its questions, and include retry behavior in the budget. TypeSafe documents model aliases and SDK retries. I am not assuming a Jev prompt-cache feature or a cache discount, and a cache at one model provider cannot be treated as reusable computation at another. TypeSafe models and operational details.

Compare the added classification cost, latency, and errors with the downstream work avoided. If an existing route field or a deterministic rule already identifies the profile, use it. If Jev mostly forwards every request to the same expensive path, the extra step has not justified itself.

Test four designs against the same tasks

Start with a reviewed dataset and compare a baseline, stable-prefix assembly, Jev-assisted routing, and Jev-assisted evidence selection. Change one variable at a time before combining them. Use a supported uncached configuration where possible; otherwise label a comparison as cold versus warm instead of claiming caching was disabled.

For each design, record task correctness, authorized tool behavior, retained evidence, fallback frequency, end-to-end latency, total provider cost, and cache-read and write tokens. A cheaper request that requires a human to repair its answer may be a more expensive task.

The test layers should include:

• Pure assembly tests: Changing evidence and questions do not change the intended static block. Instruction and tool revisions do change the appropriate serialized prefix.
• Adapter contract tests: Recorded usage fixtures cover cold requests, read hits, writes, mixed reads and writes, missing fields, and each supported provider's different totals.
• Accounting tests: Prices apply to disjoint token buckets, fallback uses the actual model, and unknown usage remains distinguishable from a miss.
• Expiration and recovery tests: Updated reference material, expired entries, retries, and partial failures preserve the right behavior.
• Live integration experiments: Qualifying prompts are tested cold, warm, after expiry, and under expected concurrency. Provider availability makes cache hits an observed result rather than a deterministic unit-test assertion.
• Jev evaluations: Measure routing errors, unnecessary fallback, lost evidence, retry overhead, and the effect on the final answer, not only classification agreement.

Mock provider responses for routine CI. Run bounded live experiments separately with synthetic or approved data and a spending limit. Repeated calls intended to measure variability must still generate fresh answers; accidentally returning an application-cached response invalidates that experiment. The LangSmith and evaluation guide explains how to connect these checks to datasets, traces, and release decisions.

Start with one repeated request pattern

Choose one task with genuinely repeated input. Establish a correctness baseline, separate stable content from changing context, and verify provider usage with recorded fixtures. Measure how often that request family reuses its prefix and whether successful tasks become cheaper or faster.

If text interpretation is a bottleneck, test a Jev-assisted route in shadow mode: record what it would choose without changing the live path. Review disagreements, compare the extra cost and latency, then trial a bounded profile with a clear fallback. Evaluate evidence filtering as a separate experiment.

Keep a small operational record with each rollout: adapter version, model, prompt and tool revisions, normalized usage schema, price revision, dataset version, and observed quality and latency. That makes a cache regression a diagnosable change instead of a mysterious bill.

If you are building an assistant or business workflow with repeated context, Peak Evergreen can help inspect the request path, test the tradeoffs, and connect the result to an application you can maintain. Discuss your idea.

All notes · Contact