Nebius Token Factory caches the prefix of recently processed prompts. Tokens served from that cache are billed at the model’s cached input price, which is usually lower than the input price, by an amount set per model. The cache is implicit: there is no cache key, no opt-in and no change to your request. According to the cached tokens documentation as read on November 5, 2026, it is best effort, it matches on the prefix of the prompt, and on Serverless it is shared per model.
This cookbook measures the cache on three models with a small script, breaks it on purpose, compares the models, and reads the cached input discount of every model in the catalog.
How a hit shows up
The response’s usage block tells you how many prompt tokens came from the cache. Chat Completions and Completions report it in usage.prompt_tokens_details.cached_tokens. The Responses API reports it in usage.input_tokens_details.cached_tokens.
- Prompt tokens are the sum of uncached input and cached input. Only the uncached part is billed at the full input price.
- On November 5, 2026 the cached counts were multiples of 64, so up to the last partial block of a prompt stayed uncached.
- When you stream, pass
stream_options={"include_usage": True}. Without it the usage block, and the cached count with it, may not come back. cached_tokens: 0on a repeated prompt is a valid answer. The cache gives no guarantee of a hit.
import os
from openai import OpenAI
client = OpenAI(base_url="https://api.tokenfactory.nebius.com/v1", api_key=os.environ["NEBIUS_API_KEY"])
long_system_prompt = open("document.txt").read() # several thousand tokens of static text
response = client.chat.completions.create(
model="zai-org/GLM-5.3-Flash",
messages=[{"role": "system", "content": long_system_prompt}, {"role": "user", "content": "Summarize section 2."}],
)
details = response.usage.prompt_tokens_details
print(response.usage.prompt_tokens, details.cached_tokens if details else 0)
Send the same call twice. The first call misses unless the same prefix was sent recently. The second should report most of the prompt as cached.
Prerequisites
- A Nebius Token Factory account and an API key.
- Python 3.10 or newer; the companion was tested on 3.12.
- A clone of the companion repository:
git clone https://github.com/nebius/token-factory-cookbook.git
cd token-factory-cookbook/api/prompt-caching
python3 -m venv .venv && .venv/bin/pip install -r requirements.txt
Create a file named .env next to the script in your editor, containing one line: NEBIUS_API_KEY=your-key. Then load it into your shell. Never print the key or commit the file.
set -a; . .env; set +a
Measure the savings
The script runs three phases for each model in NEBIUS_MODEL, which lists three models by default. First it builds a deterministic document of about 35,000 tokens, puts it in the system prompt, and sends one warm-up call followed by six short questions that share that prefix. For every call it prints the prompt tokens, the cached tokens, hit or miss, the wall time and the list-price cost. Second, it sends one more call with a timestamp in front of the document, to show what that does to the cache. Each model’s block ends with a summary line and that anti-pattern line. After the last model, a compare block lines up the summaries. Third, it prints the cached input discount of every model, from the public model catalog it fetched before the first call.
"""Measure prompt caching on Nebius Token Factory.
For every model in NEBIUS_MODEL (comma separated), phase 1 sends one warm-up call
and then several questions that share a long prefix, printing cached tokens, latency
and list-price cost per call, and phase 2 shows what a timestamp in the system prompt
does to the cache. Then a comparison table lines the models up, and phase 3 prints
the cached-input discount of every model in the public catalog.
"""
from __future__ import annotations
import json
import os
import sys
import time
import urllib.request
from dataclasses import dataclass
DEFAULT_BASE_URL = "https://api.tokenfactory.nebius.com/v1"
DEFAULT_MODELS = "moonshotai/Kimi-K3.1-Pro,zai-org/GLM-5.3-Flash,deepseek-ai/DeepSeek-V4.1-Flash-1017"
DEFAULT_CATALOG_URL = "https://tokenfactory.nebius.com/api/public/models_info"
SENTENCE = (
"The Northwind parts catalog lists item {i} with its unit price, warehouse shelf, "
"supplier code, reorder point and preferred shipping lane. "
)
class CatalogError(Exception):
"""The public model catalog could not be read."""
@dataclass
class CallResult:
prompt_tokens: int
cached_tokens: int
completion_tokens: int
seconds: float
answer: str
@property
def hit(self) -> bool:
return self.cached_tokens > 0
@dataclass
class Prices:
"""USD per 1M tokens."""
input: float
cached: float
output: float
@dataclass
class Summary:
calls: int
hits: int
cached_tokens: int
cost_usd: float
uncached_cost_usd: float
def build_document(paragraphs: int) -> str:
"""A deterministic document: same input, same bytes, every run."""
return "\n".join(f"Paragraph {i}: " + SENTENCE.format(i=i) * 3 for i in range(paragraphs))
def system_prompt(document: str, salt: str = "") -> str:
"""The static prefix. A salt in front of it makes every call a fresh prefix."""
header = f"Run: {salt}\n" if salt else ""
return header + "You answer briefly from the document.\n\nDOCUMENT:\n" + document
def questions(n: int, paragraphs: int) -> list[str]:
"""n questions spread across the document; the first one is the warm-up question."""
return [f"What is paragraph {i * paragraphs // n} about?" for i in range(n)]
def parse_catalog(data: list[dict]) -> list[dict]:
"""One row per live flavor that has a cached price."""
rows = []
for model in data:
for flavor in model.get("flavors", []):
cached = flavor.get("input_cached_price_per_million_tokens")
if flavor.get("deprecated") or cached is None:
continue
rows.append(
{
"model_id": flavor["model_id"],
"input": float(flavor["input_price_per_million_tokens"]),
"cached": float(cached),
"output": float(flavor["output_price_per_million_tokens"]),
}
)
return rows
def fetch_catalog(url: str, timeout: float = 10) -> list[dict]:
try:
with urllib.request.urlopen(url, timeout=timeout) as response:
data = json.load(response)
except (OSError, ValueError) as exc:
raise CatalogError(f"could not read {url}: {exc}") from exc
return parse_catalog(data)
def price_of(catalog: list[dict], model: str) -> Prices:
for row in catalog:
if row["model_id"] == model:
return Prices(row["input"], row["cached"], row["output"])
raise KeyError(model)
def prices_from_env(env: dict) -> Prices | None:
keys = ("NEBIUS_INPUT_PRICE", "NEBIUS_CACHED_PRICE", "NEBIUS_OUTPUT_PRICE")
if not all(key in env for key in keys):
return None
return Prices(*(float(env[key]) for key in keys))
def cost(result: CallResult, prices: Prices) -> float:
uncached = result.prompt_tokens - result.cached_tokens
total = uncached * prices.input + result.cached_tokens * prices.cached + result.completion_tokens * prices.output
return total / 1_000_000
def uncached_cost(result: CallResult, prices: Prices) -> float:
total = result.prompt_tokens * prices.input + result.completion_tokens * prices.output
return total / 1_000_000
def summarize(results: list[CallResult], prices: Prices) -> Summary:
return Summary(
calls=len(results),
hits=sum(1 for r in results if r.hit),
cached_tokens=sum(r.cached_tokens for r in results),
cost_usd=sum(cost(r, prices) for r in results),
uncached_cost_usd=sum(uncached_cost(r, prices) for r in results),
)
def break_the_cache(system: str, now: str) -> str:
"""Put something that changes every call in front of the static prefix."""
return f"Current time: {now}\n" + system
def discount_table(catalog: list[dict]) -> list[tuple[str, float, float, int]]:
rows = []
for row in catalog:
discount = 0 if row["input"] == 0 else round(100 * (1 - row["cached"] / row["input"]))
rows.append((row["model_id"], row["input"], row["cached"], discount))
return sorted(rows, key=lambda r: (-r[3], r[0]))
def ask(client, model: str, system: str, question: str, max_tokens: int = 40) -> CallResult:
"""One chat call. Cached tokens come from usage.prompt_tokens_details.cached_tokens."""
started = time.perf_counter()
response = client.chat.completions.create(
model=model,
messages=[{"role": "system", "content": system}, {"role": "user", "content": question}],
max_tokens=max_tokens,
)
seconds = time.perf_counter() - started
usage = response.usage
details = getattr(usage, "prompt_tokens_details", None)
cached = getattr(details, "cached_tokens", None) or 0
return CallResult(
prompt_tokens=usage.prompt_tokens,
cached_tokens=cached,
completion_tokens=usage.completion_tokens,
seconds=seconds,
answer=response.choices[0].message.content or "",
)
def _call_line(index: int, result: CallResult, prices: Prices) -> str:
verdict = "hit " if result.hit else "miss"
line = (
f"call {index:<2} prompt={result.prompt_tokens:<6} cached={result.cached_tokens:<6} "
f"{verdict} {result.seconds:4.1f}s ${cost(result, prices):.6f}"
)
return line + " (warm-up)" if index == 0 else line
def _saved_pct(summary: Summary) -> int:
if summary.uncached_cost_usd == 0:
return 0
return round(100 * (1 - summary.cost_usd / summary.uncached_cost_usd))
def measure_model(client, model: str, prices: Prices, source: str, system: str, asks: list[str], now: str) -> Summary | None:
"""Phases 1 and 2 for one model. Prints the per-call lines; returns None after an API error."""
print(f"model: {model} prices: in=${prices.input:g} cached=${prices.cached:g} out=${prices.output:g} per 1M ({source})")
results = []
for index, question in enumerate([asks[0], *asks]):
try:
result = ask(client, model, system, question)
except Exception as exc:
print(f"error: {type(exc).__name__}: {exc}")
return None
results.append(result)
print(_call_line(index, result, prices))
summary = summarize(results, prices)
print(
f"summary: {summary.calls} calls, {summary.hits} hits, {summary.cached_tokens} cached tokens, "
f"${summary.cost_usd:.4f} total vs ${summary.uncached_cost_usd:.4f} uncached, {_saved_pct(summary)}% saved"
)
try:
broken = ask(client, model, break_the_cache(system, now), asks[0])
except Exception as exc:
print(f"error: {type(exc).__name__}: {exc}")
return None
verdict = "hit" if broken.hit else "miss"
print(
f"anti-pattern: timestamp in system prompt -> prompt={broken.prompt_tokens} "
f"cached={broken.cached_tokens} {verdict} {broken.seconds:.1f}s"
)
return summary
def main(argv=None, client=None, catalog=None, now=None, env=None) -> int:
env = os.environ if env is None else env
now = now or time.strftime("%Y-%m-%d %H:%M:%S")
models = [m.strip() for m in env.get("NEBIUS_MODEL", DEFAULT_MODELS).split(",") if m.strip()]
try:
paragraphs = int(env.get("CACHE_PARAGRAPHS", "400"))
count = int(env.get("CACHE_QUESTIONS", "6"))
except ValueError:
print("error: CACHE_PARAGRAPHS and CACHE_QUESTIONS must be integers")
return 2
if count < 1:
print("error: CACHE_QUESTIONS must be at least 1")
return 2
if catalog is None:
try:
catalog = fetch_catalog(env.get("CATALOG_URL", DEFAULT_CATALOG_URL))
except CatalogError as exc:
print(f"warning: {exc}")
catalog = []
override = prices_from_env(env)
prices = {}
for model in models:
if override is not None:
prices[model] = (override, "env")
continue
try:
prices[model] = (price_of(catalog, model), f"catalog, {now[:10]}")
except KeyError:
print(
f"error: no catalog price for {model}; set NEBIUS_INPUT_PRICE, "
"NEBIUS_CACHED_PRICE and NEBIUS_OUTPUT_PRICE (USD per 1M tokens)"
)
return 2
if client is None:
if "NEBIUS_API_KEY" not in env:
print("error: NEBIUS_API_KEY is not set")
return 2
from openai import OpenAI
client = OpenAI(api_key=env["NEBIUS_API_KEY"], base_url=env.get("NEBIUS_BASE_URL", DEFAULT_BASE_URL))
system = system_prompt(build_document(paragraphs), env.get("CACHE_SALT", ""))
asks = questions(count, paragraphs)
summaries = {}
for model in models:
model_prices, source = prices[model]
summary = measure_model(client, model, model_prices, source, system, asks, now)
if summary is None:
return 1
summaries[model] = summary
print()
if len(models) > 1:
print("compare: hits, cached tokens and cost per model")
for model, summary in summaries.items():
print(
f" {model:<40} {summary.hits}/{summary.calls} hits cached={summary.cached_tokens:<7} "
f"${summary.cost_usd:.4f} vs ${summary.uncached_cost_usd:.4f} {_saved_pct(summary):>3}% saved"
)
print()
if catalog:
print("catalog: cached input discounts (USD per 1M, read live)")
for model_id, input_price, cached_price, discount in discount_table(catalog):
print(f" {model_id:<40} in={input_price:<6g} cached={cached_price:<7g} {discount:>3}%")
else:
print("catalog: skipped (catalog unavailable)")
return 0
if __name__ == "__main__":
sys.exit(main())
ask makes one chat call and times it. It reads the cached count from usage.prompt_tokens_details.cached_tokens and treats a missing field as zero, so the script still runs against an endpoint that does not report it. A call is a hit when the cached count is above zero.
cost splits the prompt into its uncached and cached parts and prices each one at its own rate, then adds the output tokens at the output price. uncached_cost prices the same call as if nothing had been cached. The summary compares the two totals, which gives the saving at list price.
main reads its settings from the environment, splits NEBIUS_MODEL on commas, and fetches the catalog before any call. It resolves a price for every model first. If all three price variables are set, it uses those prices for every model; otherwise it uses each model’s catalog price, and one model missing from the catalog stops the run before any call. Then it creates the OpenAI client pointed at Token Factory and calls measure_model once per model. measure_model runs the warm-up, the questions and the anti-pattern call, and prints that model’s lines. With more than one model, main then prints the compare block, one row per model with hits, cached tokens, cost against the uncached cost, and the saving. The document is the same bytes on every run, so the only thing that changes between calls is the question at the end. Call 1 repeats the warm-up question, so the first hit is a pure repeat; calls 2 to 6 change only the question at the end. The anti-pattern call reuses the same document with one line in front of it.
Run it
.venv/bin/python measure.py
Measured on November 5, 2026 at list prices, with the default settings:
model: moonshotai/Kimi-K3.1-Pro prices: in=$2 cached=$0.2 out=$10 per 1M (catalog, 2026-11-05)
call 0 prompt=34512 cached=0 miss 5.1s $0.069424 (warm-up)
call 1 prompt=34512 cached=33408 hit 2.3s $0.009290
call 2 prompt=34512 cached=33408 hit 1.8s $0.009290
call 3 prompt=34512 cached=33408 hit 1.6s $0.009290
call 4 prompt=34512 cached=33408 hit 1.5s $0.009290
call 5 prompt=34512 cached=33408 hit 1.5s $0.009290
call 6 prompt=34512 cached=33408 hit 4.2s $0.009290
summary: 7 calls, 6 hits, 200448 cached tokens, $0.1252 total vs $0.4860 uncached, 74% saved
anti-pattern: timestamp in system prompt -> prompt=34529 cached=0 miss 3.3s
model: zai-org/GLM-5.3-Flash prices: in=$0.15 cached=$0.03 out=$0.5 per 1M (catalog, 2026-11-05)
call 0 prompt=35029 cached=35008 hit 1.3s $0.001073 (warm-up)
call 1 prompt=35029 cached=35008 hit 1.3s $0.001073
call 2 prompt=35029 cached=35008 hit 1.3s $0.001073
call 3 prompt=35029 cached=35008 hit 1.4s $0.001073
call 4 prompt=35029 cached=35008 hit 1.0s $0.001073
call 5 prompt=35030 cached=35008 hit 7.3s $0.001074
call 6 prompt=35029 cached=35008 hit 1.4s $0.001073
summary: 7 calls, 7 hits, 245056 cached tokens, $0.0075 total vs $0.0369 uncached, 80% saved
anti-pattern: timestamp in system prompt -> prompt=35046 cached=0 miss 2.4s
model: deepseek-ai/DeepSeek-V4.1-Flash-1017 prices: in=$0.3 cached=$0.006 out=$1.2 per 1M (catalog, 2026-11-05)
call 0 prompt=35647 cached=35584 hit 1.6s $0.000280 (warm-up)
call 1 prompt=35647 cached=35584 hit 0.9s $0.000280
call 2 prompt=35647 cached=35584 hit 1.2s $0.000280
call 3 prompt=35647 cached=35584 hit 0.9s $0.000280
call 4 prompt=35647 cached=35584 hit 2.4s $0.000280
call 5 prompt=35647 cached=35584 hit 0.9s $0.000280
call 6 prompt=35647 cached=35584 hit 1.9s $0.000280
summary: 7 calls, 7 hits, 249088 cached tokens, $0.0020 total vs $0.0752 uncached, 97% saved
anti-pattern: timestamp in system prompt -> prompt=35664 cached=0 miss 1.4s
compare: hits, cached tokens and cost per model
moonshotai/Kimi-K3.1-Pro 6/7 hits cached=200448 $0.1252 vs $0.4860 74% saved
zai-org/GLM-5.3-Flash 7/7 hits cached=245056 $0.0075 vs $0.0369 80% saved
deepseek-ai/DeepSeek-V4.1-Flash-1017 7/7 hits cached=249088 $0.0020 vs $0.0752 97% saved
The script then prints the catalog header and one line per catalog model. That table appears in full in its own section, Which models discount cached input.
Kimi K3.1 Pro shows the cold pattern. Its warm-up had nothing cached yet, took 5.1 s and cost $0.069 at list price. Each of the six calls after it hit with 33,408 of 34,512 prompt tokens served from the cache and cost $0.0093. That is 522 blocks of 64 tokens. The other 1,104 prompt tokens on each hit were billed at the input price.
GLM-5.3-Flash and DeepSeek V4.1 Flash 1017 hit on the warm-up too, 7 calls out of 7. The document is the same bytes every run, and earlier runs on November 5, 2026 had sent it to both models, so the prefix was still in the shared cache. GLM cached 35,008 of 35,029 tokens on every call, and DeepSeek 35,584 of 35,647. In an earlier single-model run the same day, GLM’s warm-up missed and took 3.8 s before six hits.
The compare rows add it up per model. On November 5, 2026 Kimi cost $0.1252 against $0.4860 without the cache, a 74% saving. GLM cost $0.0075 against $0.0369, 80%. DeepSeek cost $0.0020 against $0.0752, 97%. Each saving reflects the model’s cached price and how many of its calls hit. All three cached counts were multiples of 64. The settings (models, document size, question count, catalog URL and price overrides) are environment variables listed in the companion README.
A longer document reuses the shorter prefix
A second run on November 5, 2026 set CACHE_PARAGRAPHS=401. No earlier run had sent that document. Every warm-up still hit, with the cached counts of the 400-paragraph document:
call 0 prompt=34598 cached=33408 hit 2.1s $0.009462 (warm-up)
call 0 prompt=35115 cached=35008 hit 1.0s $0.001086 (warm-up)
call 0 prompt=35736 cached=35584 hit 1.0s $0.000307 (warm-up)
Those lines are Kimi, GLM and DeepSeek in that order. The first 400 paragraphs are a prefix of the 401-paragraph document, so the cache served the longest prefix it held. Later calls in the same run cached the longer prefix: 34,560 tokens on Kimi, 35,072 on GLM and 35,712 on DeepSeek.
What breaks the cache
The anti-pattern lines from the same run, one per model:
anti-pattern: timestamp in system prompt -> prompt=34529 cached=0 miss 3.3s
anti-pattern: timestamp in system prompt -> prompt=35046 cached=0 miss 2.4s
anti-pattern: timestamp in system prompt -> prompt=35664 cached=0 miss 1.4s
The script put Current time: ... on the first line of the system prompt and sent the same document after it. On each model the prompt grew by 17 tokens and the cached count fell to zero. The cache matches from the first byte of the prompt. A timestamp at the start changes those first bytes, so nothing after it matches either, even though the 35,000-token document behind it is identical.
Other changes that kill a prefix in the same way:
- Dynamic content placed before static content: request IDs, user names, or today’s date in the system prompt.
- Tool definitions that are reordered or edited between calls.
- Whitespace or formatting changes to the system prompt, such as a template that renders differently.
- Switching models. The cache is per model, so a prefix warmed on one model does nothing for another.
- Trimming conversation history from the front to fit a budget. Append new turns to the end instead.
Which models discount cached input
The catalog table from the same run, read on November 5, 2026:
catalog: cached input discounts (USD per 1M, read live)
deepseek-ai/DeepSeek-V4.1-Flash-1017 in=0.3 cached=0.006 98%
deepseek-ai/DeepSeek-V4-Pro-0813 in=1.32 cached=0.04 97%
deepseek-ai/DeepSeek-V4.1-Flash in=0.3 cached=0.01 97%
deepseek-ai/DeepSeek-V4-Flash-0731 in=0.14 cached=0.01 93%
MiniMaxAI/MiniMax-M3.1-Flash in=0.15 cached=0.015 90%
moonshotai/Kimi-K3 in=3 cached=0.3 90%
moonshotai/Kimi-K3.1-Pro in=2 cached=0.2 90%
zai-org/GLM-5.2 in=1.4 cached=0.14 90%
meta-models/muse-spark-open in=1.25 cached=0.15 88%
nvidia/Nemotron-3-Ultra-550b-a55b in=1 cached=0.12 88%
moonshotai/Kimi-K2.7-Code in=0.95 cached=0.15 84%
moonshotai/Kimi-K2.6 in=0.95 cached=0.16 83%
zai-org/GLM-5.3 in=1.4 cached=0.26 81%
MiniMaxAI/MiniMax-M3 in=0.3 cached=0.06 80%
NousResearch/Hermes-4-405B in=1 cached=0.2 80%
Qwen/Qwen3.5-397B-A17B in=0.6 cached=0.12 80%
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B in=0.06 cached=0.012 80%
nvidia/Nemotron-3.5-Super-VL in=0.3 cached=0.06 80%
nvidia/nemotron-3-super-120b-a12b in=0.3 cached=0.06 80%
reflection/beam in=0.25 cached=0.05 80%
zai-org/GLM-5.3-Flash in=0.15 cached=0.03 80%
Qwen/Qwen3.8-27B in=0.45 cached=0.1 78%
google/gemma-3-27b-it in=0.1 cached=0.04 60%
nvidia/Nemotron-3_5-Lightning in=0.06 cached=0.03 50%
openai/gpt-oss-120b in=0.15 cached=0.1 33%
Qwen/Qwen3-30B-A3B-Instruct-2507 in=0.1 cached=0.1 0%
The cached input discount is set per model. The script reads the catalog at run time, so your table may differ from this one. A model without a cached price bills every input token at the input price. Prices change, so check the model catalog before you plan a budget around them.
Design rules
- Put static content first: the system prompt, tool definitions and documents. Put the changing question last.
- Append turns to the conversation history. Do not rewrite or summarize earlier turns in place.
- Keep the prefix byte identical between calls. Render templates the same way every time.
- Send requests that share a prefix to the same model.
- Warm a prefix with one call before you fan out parallel requests.
- Budget at the uncached price and treat hits as savings.
- Caching matters most for prompts of several thousand tokens or more. On November 5, 2026 a 300-token prefix hit once in four calls while the 35k-token prefix in this cookbook hit six times out of six after the warm-up. The same day, DeepSeek V4.1 Flash 1017 missed twice before it hit on a 6k-token prefix.
Verify the result
The cache is working when:
cachedis nonzero on the calls after each model’s warm-up; rerun once if a call shows 0. A warm-up that hits means the prefix was still cached from an earlier run. SetCACHE_SALTto any new value to force a cold start: it goes in front of the system prompt, so nothing matches.- The per-call cost on the hits is well below the cost of a miss.
- Each
anti-pattern:line showscached=0.
Troubleshooting
error: NEBIUS_API_KEY is not set, exit code 2: load the.envfile into the shell you run from.error: AuthenticationError: ..., exit code 1: the API returned 401. The key is wrong or revoked. CheckNEBIUS_API_KEY.error: RateLimitError: ..., exit code 1: the API returned 429. Wait and rerun. The cached prefix may have been evicted by the time you retry, so the first call may miss again.error: CACHE_PARAGRAPHS and CACHE_QUESTIONS must be integersorerror: CACHE_QUESTIONS must be at least 1, exit code 2: fix the variable the line names.- Every call shows
cached=0: the prefix was evicted, another replica served the call, the prefix is too short, or something changed it between calls. Rerun once before you debug. warning: could not read ...followed byerror: no catalog price for <model>, exit code 2: the public catalog was unreachable. SetNEBIUS_INPUT_PRICE,NEBIUS_CACHED_PRICEandNEBIUS_OUTPUT_PRICEin USD per 1M tokens. With those set, the run completes and ends withcatalog: skipped (catalog unavailable).error: no catalog price for <model>with no warning before it, exit code 2:NEBIUS_MODELnames a model that the catalog does not list with a cached price. With several models, any one missing model stops the run before the first call, and the line names it. Fix the name or set the three price variables, which then apply to every listed model.- Usage is missing when you stream in your own code: pass
stream_options={"include_usage": True}.
Clean up
The script creates no cloud resources. Remove the virtual environment and the key file.
rm -rf .venv
rm .env
Going further
- The Kimi K3.1 Pro cookbook has a short prompt caching section for a reasoning model.
- Dedicated endpoints give each customer an isolated cache.
- Batch inference, embeddings, rerank and image generation are not cached.
- The cached tokens documentation has the current rules.