Agentic cost benchmark

A model’s price-per-token tells you almost nothing about what a task actually costs. An agent that takes five tool calls to answer a question and one that takes fifteen pay a very different bill, even on the same model — and swap in a bigger, pricier model and the gap can go either way. The only way to know what a task costs is to run it and count.

In this cookbook we run the same data-analysis task against three models hosted on Nebius Token Factory, and report measured token usage, latency, estimated cost, and whether the answer was actually right.

The ModelsThe Models

We are using the Nemotron models because there are 3 distinct models from the same family. But the benchmark is not at all tied to Nemotron — you can use any models.

Here is a visualization of the 3 Nemotron models:

• Ultra: the largest (550B parameters)
• Super: midsize (120B parameters)
• Lightning: smallest (30B parameters)

The three Nemotron tiers by intelligence index and blended list price. Source: Artificial Analysis.

The BenchmarkThe Benchmark

What

• A fixed task: analyze sales data in CSV format and produce the final output in two formats — JSON, which is easily verifiable programmatically, and Markdown, which is human readable.

• Three models: the same benchmark across multiple models.

The Process

• We use the deepagents framework, which coordinates the agents and gives them tools. It also runs the benchmark in an isolated sandbox workspace — no shell, no Python execution. The model has to read the CSVs, do the arithmetic itself, and write the answer to a file.

• Validation: Model output is compared against a ground-truth expected.json. We also collect metrics like tool calls and token counts and compute cost from the configured prices.

Code and DataCode and Data

Here is the code

And we have 2 datasets — data-1 and data-2.

PrerequisitesPrerequisites

• Python 3.12 or newer
• uv (or pip, if you’d rather manage the virtualenv yourself)
• A Nebius Token Factory account with access to the three Nemotron models listed above
• NEBIUS_API_KEY in an environment variable or a local .env file — never pasted into the script or committed

Run the cookbookRun the cookbook

1. From the agent-cost-comparison-1 directory, install dependencies:

   uv sync
   

2. Create a .env file with your API key:

   NEBIUS_API_KEY=your-token-factory-key
   

3. Run the benchmark against the default data suite:

   uv run python agent_cost_comparison_1.py
   # uses 'data-1' dir by default
   

   Each model gets its own workspace under benchmarks/data-1/<model>/, with the input files copied in and output/ created fresh.

   You can also specify a data dir:

   uv run python agent_cost_comparison_1.py --data-dir data-2
   

Verify the resultVerify the result

After each model finishes, the script prints its own validation block, its metrics, and the generated summary.md, then a final comparison table sorted by cost once all models are done.

A model “passes” when every field in result.json (model output) matches expected.json (expected output).

Here is an example expected.json:

{
  "region": "west",
  "change": -60000,
  "primary_product": "widget-b"
}

You can also open the workspace directly:

cat benchmarks/data-1/nvidia__Nemotron-3_5-Lightning/output/result.json

cat benchmarks/data-1/nvidia__Nemotron-3_5-Lightning/output/summary.md

AnalysisAnalysis

Here is a sample output from a run:

Some insights:

• All three models succeeded at the task.
• Looking at cost, Lightning wins by ~10x over Super and ~26x over Ultra.
• Lightning really lives up to its name: it was the fastest run (7.1s vs 19-29s).
• Super actually used the most tokens of the three (60,649 — more than Ultra) yet still cost 2.4x less than Ultra. Ultra’s per-token rate is 3.3x Super’s, so it costs more.
• All models use more input tokens than output (see the more detailed analysis below).

TL;DR for most tasks, smaller models can produce perfectly good results at minimal cost. Faster and cheaper!

More In-Depth AnalysisMore In-Depth Analysis

Here is the cost breakdown of running on two datasets.

Costs move run to run — the agent takes a different path each time — but the ordering has held on every run: Lightning, then Super, then Ultra.

Where the calls wentWhere the calls went

The agents work entirely through file tool calls (ls, read_file, write_file). Here is the breakdown by tool:

Input Tokens are the majority

Input tokens are the bill, everywhere. 72-87% of every model’s cost is input. That’s the agentic-loop tax: every turn re-sends the whole conversation plus accumulated tool results, so input grows quadratically-ish with turn count. You can see it in the run above: Super took the most turns (14), and that alone pushed its input to 53k tokens — the most of any model — even though its tool-call count was ordinary.

Why Nemotron Ultra needed a workaroundWhy Nemotron Ultra needed a workaround

Nemotron-3-Ultra-550b-a55b ships with a Deep Agents harness profile that includes a NemotronPolicyNudgeMiddleware. On this particular task — a single-turn “read some files, write some files” job with no multi-step handoff — that middleware can mistake the work for a task transition and nudge the model off course. The fix is a runtime override that keeps the rest of the Ultra profile intact and disables just that one middleware:

register_harness_profile(
    "nebius:nvidia/Nemotron-3-Ultra-550b-a55b",
    HarnessProfile(excluded_middleware={"NemotronPolicyNudgeMiddleware"}),
)

There is a bug open for this issue.