Inference

Cut input cost with prompt caching on Token Factory

Measure prompt caching on Kimi K3.1 Pro, GLM-5.3-Flash and DeepSeek V4.1 Flash, compare cached tokens and cost side by side, see what breaks a hit, and read every model's cached-input discount.

Build it with an agent

Paste into Claude Code, Cursor, or any coding agent — it builds the recipe for you.

Build a small Python script that measures prompt caching on the Nebius Token Factory OpenAI-compatible chat completions API. Deliver `measure.py`, `requirements.txt`, pytest tests under `tests/`, and a `README.md`.

Read the credential from `NEBIUS_API_KEY`, the API base URL from `NEBIUS_BASE_URL` (default `https://api.tokenfactory.nebius.com/v1`), and the model ID from `NEBIUS_MODEL` (default `zai-org/GLM-5.3-Flash`); never print or commit the credential. Generate a deterministic document of `CACHE_PARAGRAPHS` paragraphs (default 400) and put it in the system prompt. Send one warm-up call, then `CACHE_QUESTIONS` (default 6) short questions that share that system prompt, and print for every call the prompt tokens, `usage.prompt_tokens_details.cached_tokens`, whether it hit, the wall time, and the list-price cost. Then send one more call with a timestamp line placed in front of the document and print its cached count under the prefix `anti-pattern:`. Read list prices from the public catalog at `https://tokenfactory.nebius.com/api/public/models_info` (override with `NEBIUS_INPUT_PRICE`, `NEBIUS_CACHED_PRICE`, `NEBIUS_OUTPUT_PRICE` in USD per 1M) and print a table of every model's cached input discount sorted by discount.

The command `python measure.py` must exit zero and print a line prefixed exactly with `summary:` giving calls, hits, cached tokens, total cost, uncached cost, and percent saved. Tests must cover the cost math, the summary, the catalog parsing, and the full run with a fake client, and must not call the network. Document local `.venv` setup, run the tests, then run the script once and record its dated output in the README.

Recipe

Nebius Token Factory caches the prefix of recently processed prompts. Tokens served from that cache are billed at the model’s cached input price, which is usually lower than the input price, by an amount set per model. The cache is implicit: there is no cache key, no opt-in and no change to your request. According to the cached tokens documentation as read on November 5, 2026, it is best effort, it matches on the prefix of the prompt, and on Serverless it is shared per model.

This cookbook measures the cache on three models with a small script, breaks it on purpose, compares the models, and reads the cached input discount of every model in the catalog.

How a hit shows up

The response’s usage block tells you how many prompt tokens came from the cache. Chat Completions and Completions report it in usage.prompt_tokens_details.cached_tokens. The Responses API reports it in usage.input_tokens_details.cached_tokens.

  • Prompt tokens are the sum of uncached input and cached input. Only the uncached part is billed at the full input price.
  • On November 5, 2026 the cached counts were multiples of 64, so up to the last partial block of a prompt stayed uncached.
  • When you stream, pass stream_options={"include_usage": True}. Without it the usage block, and the cached count with it, may not come back.
  • cached_tokens: 0 on a repeated prompt is a valid answer. The cache gives no guarantee of a hit.
import os
from openai import OpenAI

client = OpenAI(base_url="https://api.tokenfactory.nebius.com/v1", api_key=os.environ["NEBIUS_API_KEY"])
long_system_prompt = open("document.txt").read()  # several thousand tokens of static text
response = client.chat.completions.create(
    model="zai-org/GLM-5.3-Flash",
    messages=[{"role": "system", "content": long_system_prompt}, {"role": "user", "content": "Summarize section 2."}],
)
details = response.usage.prompt_tokens_details
print(response.usage.prompt_tokens, details.cached_tokens if details else 0)

Send the same call twice. The first call misses unless the same prefix was sent recently. The second should report most of the prompt as cached.

Prerequisites

git clone https://github.com/nebius/token-factory-cookbook.git
cd token-factory-cookbook/api/prompt-caching
python3 -m venv .venv && .venv/bin/pip install -r requirements.txt

Create a file named .env next to the script in your editor, containing one line: NEBIUS_API_KEY=your-key. Then load it into your shell. Never print the key or commit the file.

set -a; . .env; set +a

Measure the savings

The script runs three phases for each model in NEBIUS_MODEL, which lists three models by default. First it builds a deterministic document of about 35,000 tokens, puts it in the system prompt, and sends one warm-up call followed by six short questions that share that prefix. For every call it prints the prompt tokens, the cached tokens, hit or miss, the wall time and the list-price cost. Second, it sends one more call with a timestamp in front of the document, to show what that does to the cache. Each model’s block ends with a summary line and that anti-pattern line. After the last model, a compare block lines up the summaries. Third, it prints the cached input discount of every model, from the public model catalog it fetched before the first call.

"""Measure prompt caching on Nebius Token Factory.

For every model in NEBIUS_MODEL (comma separated), phase 1 sends one warm-up call
and then several questions that share a long prefix, printing cached tokens, latency
and list-price cost per call, and phase 2 shows what a timestamp in the system prompt
does to the cache. Then a comparison table lines the models up, and phase 3 prints
the cached-input discount of every model in the public catalog.
"""

from __future__ import annotations

import json
import os
import sys
import time
import urllib.request
from dataclasses import dataclass

DEFAULT_BASE_URL = "https://api.tokenfactory.nebius.com/v1"
DEFAULT_MODELS = "moonshotai/Kimi-K3.1-Pro,zai-org/GLM-5.3-Flash,deepseek-ai/DeepSeek-V4.1-Flash-1017"
DEFAULT_CATALOG_URL = "https://tokenfactory.nebius.com/api/public/models_info"
SENTENCE = (
    "The Northwind parts catalog lists item {i} with its unit price, warehouse shelf, "
    "supplier code, reorder point and preferred shipping lane. "
)


class CatalogError(Exception):
    """The public model catalog could not be read."""


@dataclass
class CallResult:
    prompt_tokens: int
    cached_tokens: int
    completion_tokens: int
    seconds: float
    answer: str

    @property
    def hit(self) -> bool:
        return self.cached_tokens > 0


@dataclass
class Prices:
    """USD per 1M tokens."""

    input: float
    cached: float
    output: float


@dataclass
class Summary:
    calls: int
    hits: int
    cached_tokens: int
    cost_usd: float
    uncached_cost_usd: float


def build_document(paragraphs: int) -> str:
    """A deterministic document: same input, same bytes, every run."""
    return "\n".join(f"Paragraph {i}: " + SENTENCE.format(i=i) * 3 for i in range(paragraphs))


def system_prompt(document: str, salt: str = "") -> str:
    """The static prefix. A salt in front of it makes every call a fresh prefix."""
    header = f"Run: {salt}\n" if salt else ""
    return header + "You answer briefly from the document.\n\nDOCUMENT:\n" + document


def questions(n: int, paragraphs: int) -> list[str]:
    """n questions spread across the document; the first one is the warm-up question."""
    return [f"What is paragraph {i * paragraphs // n} about?" for i in range(n)]


def parse_catalog(data: list[dict]) -> list[dict]:
    """One row per live flavor that has a cached price."""
    rows = []
    for model in data:
        for flavor in model.get("flavors", []):
            cached = flavor.get("input_cached_price_per_million_tokens")
            if flavor.get("deprecated") or cached is None:
                continue
            rows.append(
                {
                    "model_id": flavor["model_id"],
                    "input": float(flavor["input_price_per_million_tokens"]),
                    "cached": float(cached),
                    "output": float(flavor["output_price_per_million_tokens"]),
                }
            )
    return rows


def fetch_catalog(url: str, timeout: float = 10) -> list[dict]:
    try:
        with urllib.request.urlopen(url, timeout=timeout) as response:
            data = json.load(response)
    except (OSError, ValueError) as exc:
        raise CatalogError(f"could not read {url}: {exc}") from exc
    return parse_catalog(data)


def price_of(catalog: list[dict], model: str) -> Prices:
    for row in catalog:
        if row["model_id"] == model:
            return Prices(row["input"], row["cached"], row["output"])
    raise KeyError(model)


def prices_from_env(env: dict) -> Prices | None:
    keys = ("NEBIUS_INPUT_PRICE", "NEBIUS_CACHED_PRICE", "NEBIUS_OUTPUT_PRICE")
    if not all(key in env for key in keys):
        return None
    return Prices(*(float(env[key]) for key in keys))


def cost(result: CallResult, prices: Prices) -> float:
    uncached = result.prompt_tokens - result.cached_tokens
    total = uncached * prices.input + result.cached_tokens * prices.cached + result.completion_tokens * prices.output
    return total / 1_000_000


def uncached_cost(result: CallResult, prices: Prices) -> float:
    total = result.prompt_tokens * prices.input + result.completion_tokens * prices.output
    return total / 1_000_000


def summarize(results: list[CallResult], prices: Prices) -> Summary:
    return Summary(
        calls=len(results),
        hits=sum(1 for r in results if r.hit),
        cached_tokens=sum(r.cached_tokens for r in results),
        cost_usd=sum(cost(r, prices) for r in results),
        uncached_cost_usd=sum(uncached_cost(r, prices) for r in results),
    )


def break_the_cache(system: str, now: str) -> str:
    """Put something that changes every call in front of the static prefix."""
    return f"Current time: {now}\n" + system


def discount_table(catalog: list[dict]) -> list[tuple[str, float, float, int]]:
    rows = []
    for row in catalog:
        discount = 0 if row["input"] == 0 else round(100 * (1 - row["cached"] / row["input"]))
        rows.append((row["model_id"], row["input"], row["cached"], discount))
    return sorted(rows, key=lambda r: (-r[3], r[0]))


def ask(client, model: str, system: str, question: str, max_tokens: int = 40) -> CallResult:
    """One chat call. Cached tokens come from usage.prompt_tokens_details.cached_tokens."""
    started = time.perf_counter()
    response = client.chat.completions.create(
        model=model,
        messages=[{"role": "system", "content": system}, {"role": "user", "content": question}],
        max_tokens=max_tokens,
    )
    seconds = time.perf_counter() - started
    usage = response.usage
    details = getattr(usage, "prompt_tokens_details", None)
    cached = getattr(details, "cached_tokens", None) or 0
    return CallResult(
        prompt_tokens=usage.prompt_tokens,
        cached_tokens=cached,
        completion_tokens=usage.completion_tokens,
        seconds=seconds,
        answer=response.choices[0].message.content or "",
    )


def _call_line(index: int, result: CallResult, prices: Prices) -> str:
    verdict = "hit " if result.hit else "miss"
    line = (
        f"call {index:<2} prompt={result.prompt_tokens:<6} cached={result.cached_tokens:<6} "
        f"{verdict}  {result.seconds:4.1f}s  ${cost(result, prices):.6f}"
    )
    return line + "  (warm-up)" if index == 0 else line


def _saved_pct(summary: Summary) -> int:
    if summary.uncached_cost_usd == 0:
        return 0
    return round(100 * (1 - summary.cost_usd / summary.uncached_cost_usd))


def measure_model(client, model: str, prices: Prices, source: str, system: str, asks: list[str], now: str) -> Summary | None:
    """Phases 1 and 2 for one model. Prints the per-call lines; returns None after an API error."""
    print(f"model: {model}  prices: in=${prices.input:g} cached=${prices.cached:g} out=${prices.output:g} per 1M ({source})")
    results = []
    for index, question in enumerate([asks[0], *asks]):
        try:
            result = ask(client, model, system, question)
        except Exception as exc:
            print(f"error: {type(exc).__name__}: {exc}")
            return None
        results.append(result)
        print(_call_line(index, result, prices))

    summary = summarize(results, prices)
    print(
        f"summary: {summary.calls} calls, {summary.hits} hits, {summary.cached_tokens} cached tokens, "
        f"${summary.cost_usd:.4f} total vs ${summary.uncached_cost_usd:.4f} uncached, {_saved_pct(summary)}% saved"
    )

    try:
        broken = ask(client, model, break_the_cache(system, now), asks[0])
    except Exception as exc:
        print(f"error: {type(exc).__name__}: {exc}")
        return None
    verdict = "hit" if broken.hit else "miss"
    print(
        f"anti-pattern: timestamp in system prompt -> prompt={broken.prompt_tokens} "
        f"cached={broken.cached_tokens} {verdict} {broken.seconds:.1f}s"
    )
    return summary


def main(argv=None, client=None, catalog=None, now=None, env=None) -> int:
    env = os.environ if env is None else env
    now = now or time.strftime("%Y-%m-%d %H:%M:%S")
    models = [m.strip() for m in env.get("NEBIUS_MODEL", DEFAULT_MODELS).split(",") if m.strip()]
    try:
        paragraphs = int(env.get("CACHE_PARAGRAPHS", "400"))
        count = int(env.get("CACHE_QUESTIONS", "6"))
    except ValueError:
        print("error: CACHE_PARAGRAPHS and CACHE_QUESTIONS must be integers")
        return 2
    if count < 1:
        print("error: CACHE_QUESTIONS must be at least 1")
        return 2

    if catalog is None:
        try:
            catalog = fetch_catalog(env.get("CATALOG_URL", DEFAULT_CATALOG_URL))
        except CatalogError as exc:
            print(f"warning: {exc}")
            catalog = []

    override = prices_from_env(env)
    prices = {}
    for model in models:
        if override is not None:
            prices[model] = (override, "env")
            continue
        try:
            prices[model] = (price_of(catalog, model), f"catalog, {now[:10]}")
        except KeyError:
            print(
                f"error: no catalog price for {model}; set NEBIUS_INPUT_PRICE, "
                "NEBIUS_CACHED_PRICE and NEBIUS_OUTPUT_PRICE (USD per 1M tokens)"
            )
            return 2

    if client is None:
        if "NEBIUS_API_KEY" not in env:
            print("error: NEBIUS_API_KEY is not set")
            return 2
        from openai import OpenAI

        client = OpenAI(api_key=env["NEBIUS_API_KEY"], base_url=env.get("NEBIUS_BASE_URL", DEFAULT_BASE_URL))

    system = system_prompt(build_document(paragraphs), env.get("CACHE_SALT", ""))
    asks = questions(count, paragraphs)
    summaries = {}
    for model in models:
        model_prices, source = prices[model]
        summary = measure_model(client, model, model_prices, source, system, asks, now)
        if summary is None:
            return 1
        summaries[model] = summary
        print()

    if len(models) > 1:
        print("compare: hits, cached tokens and cost per model")
        for model, summary in summaries.items():
            print(
                f"  {model:<40} {summary.hits}/{summary.calls} hits  cached={summary.cached_tokens:<7} "
                f"${summary.cost_usd:.4f} vs ${summary.uncached_cost_usd:.4f}  {_saved_pct(summary):>3}% saved"
            )
        print()

    if catalog:
        print("catalog: cached input discounts (USD per 1M, read live)")
        for model_id, input_price, cached_price, discount in discount_table(catalog):
            print(f"  {model_id:<40} in={input_price:<6g} cached={cached_price:<7g} {discount:>3}%")
    else:
        print("catalog: skipped (catalog unavailable)")
    return 0


if __name__ == "__main__":
    sys.exit(main())

ask makes one chat call and times it. It reads the cached count from usage.prompt_tokens_details.cached_tokens and treats a missing field as zero, so the script still runs against an endpoint that does not report it. A call is a hit when the cached count is above zero.

cost splits the prompt into its uncached and cached parts and prices each one at its own rate, then adds the output tokens at the output price. uncached_cost prices the same call as if nothing had been cached. The summary compares the two totals, which gives the saving at list price.

main reads its settings from the environment, splits NEBIUS_MODEL on commas, and fetches the catalog before any call. It resolves a price for every model first. If all three price variables are set, it uses those prices for every model; otherwise it uses each model’s catalog price, and one model missing from the catalog stops the run before any call. Then it creates the OpenAI client pointed at Token Factory and calls measure_model once per model. measure_model runs the warm-up, the questions and the anti-pattern call, and prints that model’s lines. With more than one model, main then prints the compare block, one row per model with hits, cached tokens, cost against the uncached cost, and the saving. The document is the same bytes on every run, so the only thing that changes between calls is the question at the end. Call 1 repeats the warm-up question, so the first hit is a pure repeat; calls 2 to 6 change only the question at the end. The anti-pattern call reuses the same document with one line in front of it.

Run it

.venv/bin/python measure.py

Measured on November 5, 2026 at list prices, with the default settings:

model: moonshotai/Kimi-K3.1-Pro  prices: in=$2 cached=$0.2 out=$10 per 1M (catalog, 2026-11-05)
call 0  prompt=34512  cached=0      miss   5.1s  $0.069424  (warm-up)
call 1  prompt=34512  cached=33408  hit    2.3s  $0.009290
call 2  prompt=34512  cached=33408  hit    1.8s  $0.009290
call 3  prompt=34512  cached=33408  hit    1.6s  $0.009290
call 4  prompt=34512  cached=33408  hit    1.5s  $0.009290
call 5  prompt=34512  cached=33408  hit    1.5s  $0.009290
call 6  prompt=34512  cached=33408  hit    4.2s  $0.009290
summary: 7 calls, 6 hits, 200448 cached tokens, $0.1252 total vs $0.4860 uncached, 74% saved
anti-pattern: timestamp in system prompt -> prompt=34529 cached=0 miss 3.3s

model: zai-org/GLM-5.3-Flash  prices: in=$0.15 cached=$0.03 out=$0.5 per 1M (catalog, 2026-11-05)
call 0  prompt=35029  cached=35008  hit    1.3s  $0.001073  (warm-up)
call 1  prompt=35029  cached=35008  hit    1.3s  $0.001073
call 2  prompt=35029  cached=35008  hit    1.3s  $0.001073
call 3  prompt=35029  cached=35008  hit    1.4s  $0.001073
call 4  prompt=35029  cached=35008  hit    1.0s  $0.001073
call 5  prompt=35030  cached=35008  hit    7.3s  $0.001074
call 6  prompt=35029  cached=35008  hit    1.4s  $0.001073
summary: 7 calls, 7 hits, 245056 cached tokens, $0.0075 total vs $0.0369 uncached, 80% saved
anti-pattern: timestamp in system prompt -> prompt=35046 cached=0 miss 2.4s

model: deepseek-ai/DeepSeek-V4.1-Flash-1017  prices: in=$0.3 cached=$0.006 out=$1.2 per 1M (catalog, 2026-11-05)
call 0  prompt=35647  cached=35584  hit    1.6s  $0.000280  (warm-up)
call 1  prompt=35647  cached=35584  hit    0.9s  $0.000280
call 2  prompt=35647  cached=35584  hit    1.2s  $0.000280
call 3  prompt=35647  cached=35584  hit    0.9s  $0.000280
call 4  prompt=35647  cached=35584  hit    2.4s  $0.000280
call 5  prompt=35647  cached=35584  hit    0.9s  $0.000280
call 6  prompt=35647  cached=35584  hit    1.9s  $0.000280
summary: 7 calls, 7 hits, 249088 cached tokens, $0.0020 total vs $0.0752 uncached, 97% saved
anti-pattern: timestamp in system prompt -> prompt=35664 cached=0 miss 1.4s

compare: hits, cached tokens and cost per model
  moonshotai/Kimi-K3.1-Pro                 6/7 hits  cached=200448  $0.1252 vs $0.4860   74% saved
  zai-org/GLM-5.3-Flash                    7/7 hits  cached=245056  $0.0075 vs $0.0369   80% saved
  deepseek-ai/DeepSeek-V4.1-Flash-1017     7/7 hits  cached=249088  $0.0020 vs $0.0752   97% saved

The script then prints the catalog header and one line per catalog model. That table appears in full in its own section, Which models discount cached input.

Kimi K3.1 Pro shows the cold pattern. Its warm-up had nothing cached yet, took 5.1 s and cost $0.069 at list price. Each of the six calls after it hit with 33,408 of 34,512 prompt tokens served from the cache and cost $0.0093. That is 522 blocks of 64 tokens. The other 1,104 prompt tokens on each hit were billed at the input price.

GLM-5.3-Flash and DeepSeek V4.1 Flash 1017 hit on the warm-up too, 7 calls out of 7. The document is the same bytes every run, and earlier runs on November 5, 2026 had sent it to both models, so the prefix was still in the shared cache. GLM cached 35,008 of 35,029 tokens on every call, and DeepSeek 35,584 of 35,647. In an earlier single-model run the same day, GLM’s warm-up missed and took 3.8 s before six hits.

The compare rows add it up per model. On November 5, 2026 Kimi cost $0.1252 against $0.4860 without the cache, a 74% saving. GLM cost $0.0075 against $0.0369, 80%. DeepSeek cost $0.0020 against $0.0752, 97%. Each saving reflects the model’s cached price and how many of its calls hit. All three cached counts were multiples of 64. The settings (models, document size, question count, catalog URL and price overrides) are environment variables listed in the companion README.

A longer document reuses the shorter prefix

A second run on November 5, 2026 set CACHE_PARAGRAPHS=401. No earlier run had sent that document. Every warm-up still hit, with the cached counts of the 400-paragraph document:

call 0  prompt=34598  cached=33408  hit    2.1s  $0.009462  (warm-up)
call 0  prompt=35115  cached=35008  hit    1.0s  $0.001086  (warm-up)
call 0  prompt=35736  cached=35584  hit    1.0s  $0.000307  (warm-up)

Those lines are Kimi, GLM and DeepSeek in that order. The first 400 paragraphs are a prefix of the 401-paragraph document, so the cache served the longest prefix it held. Later calls in the same run cached the longer prefix: 34,560 tokens on Kimi, 35,072 on GLM and 35,712 on DeepSeek.

What breaks the cache

The anti-pattern lines from the same run, one per model:

anti-pattern: timestamp in system prompt -> prompt=34529 cached=0 miss 3.3s
anti-pattern: timestamp in system prompt -> prompt=35046 cached=0 miss 2.4s
anti-pattern: timestamp in system prompt -> prompt=35664 cached=0 miss 1.4s

The script put Current time: ... on the first line of the system prompt and sent the same document after it. On each model the prompt grew by 17 tokens and the cached count fell to zero. The cache matches from the first byte of the prompt. A timestamp at the start changes those first bytes, so nothing after it matches either, even though the 35,000-token document behind it is identical.

Other changes that kill a prefix in the same way:

  • Dynamic content placed before static content: request IDs, user names, or today’s date in the system prompt.
  • Tool definitions that are reordered or edited between calls.
  • Whitespace or formatting changes to the system prompt, such as a template that renders differently.
  • Switching models. The cache is per model, so a prefix warmed on one model does nothing for another.
  • Trimming conversation history from the front to fit a budget. Append new turns to the end instead.

Which models discount cached input

The catalog table from the same run, read on November 5, 2026:

catalog: cached input discounts (USD per 1M, read live)
  deepseek-ai/DeepSeek-V4.1-Flash-1017     in=0.3    cached=0.006    98%
  deepseek-ai/DeepSeek-V4-Pro-0813         in=1.32   cached=0.04     97%
  deepseek-ai/DeepSeek-V4.1-Flash          in=0.3    cached=0.01     97%
  deepseek-ai/DeepSeek-V4-Flash-0731       in=0.14   cached=0.01     93%
  MiniMaxAI/MiniMax-M3.1-Flash             in=0.15   cached=0.015    90%
  moonshotai/Kimi-K3                       in=3      cached=0.3      90%
  moonshotai/Kimi-K3.1-Pro                 in=2      cached=0.2      90%
  zai-org/GLM-5.2                          in=1.4    cached=0.14     90%
  meta-models/muse-spark-open              in=1.25   cached=0.15     88%
  nvidia/Nemotron-3-Ultra-550b-a55b        in=1      cached=0.12     88%
  moonshotai/Kimi-K2.7-Code                in=0.95   cached=0.15     84%
  moonshotai/Kimi-K2.6                     in=0.95   cached=0.16     83%
  zai-org/GLM-5.3                          in=1.4    cached=0.26     81%
  MiniMaxAI/MiniMax-M3                     in=0.3    cached=0.06     80%
  NousResearch/Hermes-4-405B               in=1      cached=0.2      80%
  Qwen/Qwen3.5-397B-A17B                   in=0.6    cached=0.12     80%
  nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B    in=0.06   cached=0.012    80%
  nvidia/Nemotron-3.5-Super-VL             in=0.3    cached=0.06     80%
  nvidia/nemotron-3-super-120b-a12b        in=0.3    cached=0.06     80%
  reflection/beam                          in=0.25   cached=0.05     80%
  zai-org/GLM-5.3-Flash                    in=0.15   cached=0.03     80%
  Qwen/Qwen3.8-27B                         in=0.45   cached=0.1      78%
  google/gemma-3-27b-it                    in=0.1    cached=0.04     60%
  nvidia/Nemotron-3_5-Lightning            in=0.06   cached=0.03     50%
  openai/gpt-oss-120b                      in=0.15   cached=0.1      33%
  Qwen/Qwen3-30B-A3B-Instruct-2507         in=0.1    cached=0.1       0%

The cached input discount is set per model. The script reads the catalog at run time, so your table may differ from this one. A model without a cached price bills every input token at the input price. Prices change, so check the model catalog before you plan a budget around them.

Design rules

  • Put static content first: the system prompt, tool definitions and documents. Put the changing question last.
  • Append turns to the conversation history. Do not rewrite or summarize earlier turns in place.
  • Keep the prefix byte identical between calls. Render templates the same way every time.
  • Send requests that share a prefix to the same model.
  • Warm a prefix with one call before you fan out parallel requests.
  • Budget at the uncached price and treat hits as savings.
  • Caching matters most for prompts of several thousand tokens or more. On November 5, 2026 a 300-token prefix hit once in four calls while the 35k-token prefix in this cookbook hit six times out of six after the warm-up. The same day, DeepSeek V4.1 Flash 1017 missed twice before it hit on a 6k-token prefix.

Verify the result

The cache is working when:

  1. cached is nonzero on the calls after each model’s warm-up; rerun once if a call shows 0. A warm-up that hits means the prefix was still cached from an earlier run. Set CACHE_SALT to any new value to force a cold start: it goes in front of the system prompt, so nothing matches.
  2. The per-call cost on the hits is well below the cost of a miss.
  3. Each anti-pattern: line shows cached=0.

Troubleshooting

  • error: NEBIUS_API_KEY is not set, exit code 2: load the .env file into the shell you run from.
  • error: AuthenticationError: ..., exit code 1: the API returned 401. The key is wrong or revoked. Check NEBIUS_API_KEY.
  • error: RateLimitError: ..., exit code 1: the API returned 429. Wait and rerun. The cached prefix may have been evicted by the time you retry, so the first call may miss again.
  • error: CACHE_PARAGRAPHS and CACHE_QUESTIONS must be integers or error: CACHE_QUESTIONS must be at least 1, exit code 2: fix the variable the line names.
  • Every call shows cached=0: the prefix was evicted, another replica served the call, the prefix is too short, or something changed it between calls. Rerun once before you debug.
  • warning: could not read ... followed by error: no catalog price for <model>, exit code 2: the public catalog was unreachable. Set NEBIUS_INPUT_PRICE, NEBIUS_CACHED_PRICE and NEBIUS_OUTPUT_PRICE in USD per 1M tokens. With those set, the run completes and ends with catalog: skipped (catalog unavailable).
  • error: no catalog price for <model> with no warning before it, exit code 2: NEBIUS_MODEL names a model that the catalog does not list with a cached price. With several models, any one missing model stops the run before the first call, and the line names it. Fix the name or set the three price variables, which then apply to every listed model.
  • Usage is missing when you stream in your own code: pass stream_options={"include_usage": True}.

Clean up

The script creates no cloud resources. Remove the virtual environment and the key file.

rm -rf .venv
rm .env

Going further