Search
Log in to Token FactoryLog in to AI Cloud
Data, Evals & Observability

Invoice Extraction with Confidence-Gated Human Validation

Extract invoice fields with a Nebius vision model, then escalate only the uncertain ones to a vetted human expert through Tendem by Toloka. Pay for judgment where it matters.

by AbdulMajed Raja · Developer Relations

Build it with an agent

Paste into Claude Code, Cursor, or any coding agent — it builds the recipe for you.

Build a small, production-minded Python project that extracts structured fields from invoice images with a Nebius Token Factory vision model, and escalates only the uncertain results to a human expert through Tendem by Toloka.

Manage dependencies with `uv`. Deliver `pyproject.toml`, a `.env.example`, an `agent.py` helper holding the shared Nebius client, a `use_cases/invoice_vision_human_validation.py` recipe, pytest tests under `tests/`, a `Dockerfile`, a `.dockerignore`, and a `README.md`.

Read the Token Factory credential from `NEBIUS_API_KEY`, the API base URL from `NEBIUS_BASE_URL` (default `https://api.tokenfactory.nebius.com/v1/`), the vision model ID from `NEBIUS_MODEL` (default `Qwen/Qwen2.5-VL-72B-Instruct`), and the human-expert credential from `TENDEM_API_KEY`; never print or commit either credential.

For each image in `documents/inbox/`, send it to the vision model as a base64 data URL and require a single JSON object with `invoice_number`, `issue_date`, `vendor_name`, `total_amount`, `currency`, `confidence`, and `notes`, instructing the model never to guess obscured text. Then apply a plain-Python routing rule (not a second model call) that escalates a record when confidence is below `0.95` or any required field is null. Escalated invoices become Tendem tasks that carry the image and the uncertain draft, and the human result replaces the draft.

Enforce three safety properties: a `MAX_HUMAN_PRICE_USD` cap that raises rather than approving a quote above it, a journal file mapping each image to its task ID so a restart never buys duplicate work, and a CSV written after every invoice recording a `source` column of `nebius-vision` or `tendem-human`.

The command `python use_cases/invoice_vision_human_validation.py` must process the inbox, exit zero, and print a final line prefixed exactly with `Saved invoices:`. Keep the tests deterministic by stubbing both the vision client and the Tendem client so no test makes a network call or purchases human work; assert in particular that a quote above the cap is refused. Document uv setup, both credentials, running the tests, and a single run, with clear verification, troubleshooting, and cleanup instructions.

Recipe

A vision model reading a blurred invoice still produces a number. On the sample invoice in this cookbook, Qwen2.5-VL-72B-Instruct read a total of $2,148.75 and rated its own confidence at 0.70. A human expert then looked at the same image and returned null: the total was too blurred to confirm, and the invoice number and issue date were hidden behind a PAID stamp. The model had not misread the total; it had invented one.

This cookbook builds a pipeline for that gap: it uses the model’s confidence score as a routing decision and buys human judgment only for the documents that need it.

What you will build

A batch invoice processor with three stages:

  1. Extract. Every image goes to a Nebius vision model, which returns the invoice fields plus a self-reported confidence score.
  2. Route. A plain Python rule, not another model call, decides which results are trustworthy. A record escalates if confidence falls below 0.95 or any required field is missing.
  3. Validate. Escalated invoices become paid tasks for a vetted human expert through Tendem by Toloka. The expert’s answer replaces the draft, and every row in the output CSV carries a source column recording who finalized it.

The routing rule is plain code on purpose: it is deterministic, auditable, and unit-testable, which a second model call would not be.

Prerequisites

  • Python 3.11 or newer and uv
  • A Nebius Token Factory account with access to Qwen/Qwen2.5-VL-72B-Instruct
  • A Tendem account, and an API key from Account Settings → Tendem MCP → Agent Builders
  • A funded Tendem balance. Human validation is paid work; the sample run below settled at $3.00.

Run the cookbook

  1. Clone the project and install dependencies:

    git clone https://github.com/amrrs/toloka-nebiustf-validation.git
cd toloka-nebiustf-validation
uv sync --extra dev

  2. Provide both credentials. The recipe reads them from environment files:

    export NEBIUS_API_KEY="your-token-factory-key"
export TENDEM_API_KEY="your-tendem-key"

  3. Put invoice images in documents/inbox/. A low-quality sample is included so you can run the escalation path without supplying your own documents.

  4. Run the recipe:

    uv run python use_cases/invoice_vision_human_validation.py

Results land in documents/extracted.csv, written after each invoice so an interrupted batch keeps completed work.

Cost controls

The configuration block at the top of the script holds the settings that affect spend:

MIN_CONFIDENCE = 0.95          # below this, escalate to a human
MAX_HUMAN_PRICE_USD = 10.0     # quotes above this are refused, not approved
HUMAN_WAIT_SECONDS = 6 * 60 * 60

MAX_HUMAN_PRICE_USD is a hard stop. When a quote exceeds it the run raises rather than approving, so an unexpected price becomes a person’s decision instead of a silent charge. Raising MIN_CONFIDENCE sends more work to humans and costs more; lowering it keeps more model output unchecked.

The script also keeps a journal in documents/tendem-tasks.json mapping each image to its Tendem task ID. Restart mid-batch and it resumes the existing task rather than buying the same validation twice.

Verify the result

Open documents/extracted.csv and look at the source column. Rows finalized by the model read nebius-vision; rows a human corrected read tendem-human. On the bundled sample you should see one tendem-human row where total_amount is empty and the notes explain why the expert refused to guess.

Compare that against documents/run-evidence/vision-invoice-live-run.json, which records a complete run: the model’s 2148.75 at 0.80 confidence, the escalate_to_human decision, the $3.00 quote, and the human’s corrected output.

Reproducing the first stage on its own is fast and costs a fraction of a cent. A measured extraction on the sample used 342 prompt and 82 completion tokens, about $0.000147, in 3.8 seconds.

Timing expectations

The Nebius pass returns in seconds. Human validation is an asynchronous task that a person picks up, and the script waits up to six hours for a result. Plan your first end-to-end run around that.

Troubleshooting

  • 401 from Token Factory: check NEBIUS_API_KEY, and that your account has access to the vision model.
  • The run raises on a quote: the quote exceeded MAX_HUMAN_PRICE_USD, which is the intended behavior. Review the task and raise the cap if the price is fair.
  • Nothing escalates: your invoices are clean and the model is confident. Lower MIN_CONFIDENCE or use the bundled low-quality sample to exercise the human path.
  • A task seems stuck: check documents/tendem-tasks.json for the task ID and inspect it in the Tendem console.

Clean up

Completed tasks remove themselves from the journal. To reset local state:

rm -f documents/extracted.csv documents/tendem-tasks.json
rm -rf .venv
unset NEBIUS_API_KEY TENDEM_API_KEY

Paid Tendem tasks that already settled cannot be refunded by deleting local files. If you want to stop spending, clear documents/inbox/ before the next run.

More cookbooks

Continue with another hands-on guide.

Data, Evals & Observability

Agentic cost benchmark

by Sujee Maniyam

Run a task across multiple models on Token Factory, then compare measured cost, tokens, latency, and correctness - not just list price.

~$0.11 per run< 3 min to first run