Stronger Agents with LangChain and LangGraph

Turn the hand-wired Nebius agent into a typed graph that can choose the shortest useful path to an answer.

This cookbook is part of the Agent Blueprint Recipes sequence: Foundation, Knowledge, Grounding, Orchestration, Thread Memory, User Memory, Observability, Guardrails, Actions, and Simulation. The first three recipes in that sequence prove the Nebius integration path: call the model, ground it in private data, then add fresh web context.

The next production problem is latency. A working agent can still feel broken when one answer comes back as:

Time: 40.78s | Tokens: 40 embed, 1421 in, 535 out | Cost: $0.000399

That cost is fine. The wait is not.

This cookbook introduces LangChain and LangGraph as the point where orchestration becomes explicit enough to reduce time to response: route the request, skip unnecessary model or tool work, cap generated tokens by path, and stream phase events as soon as the graph has something useful to report.

After moving the book agent into the graph, a follow-up recommendation request with recent-context needs came back like this:

Time: 6.97s | Tokens: 418 in, 266 out | Cost: $0.000161 | Routing: deliberate / curated_plus_fresh_context

The lower cost matters less than the fact that the user sees useful output in seconds, while the route and context decision stay visible for debugging.

What you will buildWhat you will build

A production-shape FastAPI service that keeps the same SSE contract as the earlier recipes, but moves latency-sensitive orchestration into a LangGraph state graph:

1. Route classifies the request as direct or deliberate without spending a model call.
2. Prepare uses a LangChain prompt template to build Nebius chat messages from graph state.
3. Budget caps output tokens differently for fast-path and deliberate-path requests.
4. Stream emits named SSE events while graph updates and Nebius tokens arrive.

Persistent context and memory primitives are saved for the next recipe in the sequence. This recipe stays focused on graph shape, typed state, and streaming events.

flowchart LR
    Request["POST /agent/run"] --> Route["route"]
    Route -->|"direct"| Direct["prepare_direct"]
    Route -->|"deliberate"| Deliberate["prepare_deliberate"]
    Direct --> Stream["Nebius stream"]
    Deliberate --> Stream
    Stream --> SSE["SSE events"]

PrerequisitesPrerequisites

• Python 3.12+
• uv
• A Nebius API key, from the Nebius console
• Docker (optional)

Run the cookbookRun the cookbook

cp .env.example .env
# Open .env and fill NEBIUS_API_KEY

uv sync
make dev

Set NEBIUS_BASE_URL=https://api.tokenfactory.nebius.com/v1/ and NEBIUS_MODEL=Qwen/Qwen3.5-397B-A17B in your .env. The repo’s .env.example and config default still carry the older api.studio.nebius.ai hostname and a chat model that Token Factory no longer serves, so override both when you copy the file.

Then in another terminal:

curl -N -X POST http://localhost:8000/agent/run \
  -H 'content-type: application/json' \
  -d '{"prompt":"Recommend recent climate fiction with enough context to explain why each book is worth reading now."}'

You should see named SSE events:

event: status
data: {"phase":"routing","targetFirstTokenMs":1200}

event: status
data: {"phase":"routed","route":"direct"}

event: status
data: {"phase":"writing","route":"direct"}

event: token
data: {"text":"Graph"}

event: done
data: {}

Walk-throughWalk-through

The FastAPI route stays intentionally boring. It validates the request, passes temperature and max-token controls into the agent, and translates typed events into SSE. The graph lives in app/core/agent.py.

request -> route -> LangGraph route node -> direct/deliberate prepare node -> Nebius stream -> SSE

The graph is small on purpose, but the production move is real. The route node makes the latency decision visible. The prepare nodes keep prompt construction out of the route handler. The token budgets make “answer directly” and “think a little more” materially different execution paths.

This is not magic acceleration. LangGraph does not make a model emit tokens faster. It helps you stop spending latency on work the current request does not need, and it gives the API layer immediate progress events instead of a silent 40-second wait.

Latency controlsLatency controls

The default .env.example includes:

DIRECT_RESPONSE_MAX_TOKENS=384
DELIBERATE_RESPONSE_MAX_TOKENS=700
FIRST_TOKEN_TARGET_MS=1200

Simple prompts use the direct route and a smaller output budget. Longer prompts, comparisons, pricing questions, or freshness-sensitive wording use the deliberate route. The client can still send temperature and max_tokens, but the graph clamps max_tokens to the selected route’s budget.

Why this is fasterWhy this is faster

The earlier knowledge and real-time recipes prove the full production path: embed, retrieve, search, synthesize, then stream. That path is valuable when the prompt needs domain knowledge or fresh context. It is wasteful when the user asks a small direct question.

This recipe separates orchestration from capability. The graph first decides which work is necessary. The direct route avoids extra model or tool calls and keeps the answer budget small. The deliberate route keeps room for a richer answer, but it is still bounded. Both routes send status events before the model finishes, so the user is never waiting on a silent connection.

MetricsMetrics

The /metrics endpoint includes the base HTTP and Nebius metrics from the earlier recipes, plus orchestration-specific signals:

agent_route_total{route="direct"}
agent_route_total{route="deliberate"}
agent_first_token_seconds_bucket{route="direct",le="1.2"}

These metrics answer the production questions this cookbook raises: which path are users actually taking, is the fast path staying fast, and did a routing change improve time to first token or just move latency around.

Memory boundaryMemory boundary

This cookbook does not introduce persistent context or memory primitives. The state graph only carries the data needed for the current request. Later recipes in the sequence introduce short-term thread memory, then durable user and application memory.

Design decisionsDesign decisions

• The route node is a deterministic keyword classifier, not a model call, so routing itself costs no latency or tokens.
• Prompt construction uses a LangChain ChatPromptTemplate so the system prompt, the routing plan, and the user prompt stay separate and testable.
• max_tokens is clamped per route (DIRECT_RESPONSE_MAX_TOKENS, DELIBERATE_RESPONSE_MAX_TOKENS) so a client-requested budget can never make the deliberate path arbitrarily expensive.
• The FastAPI handler watches for client disconnects with an asyncio.Event and stops streaming from the Nebius client rather than continuing to burn tokens on a dropped connection.

Failure modesFailure modes

• A prompt that matches no fresh-context or recommendation keyword always takes the direct route, even if it is long or ambiguous. The route node has no fallback to a model-based classifier in this recipe.
• If the Nebius stream drops mid-response, the client sees whatever token events already arrived and then a done event with partial usage, since the graph does not retry a partially streamed completion.
• Pricing shown in the trailing summary line depends on NEBIUS_ENABLE_PRICING_LOOKUP and the configured per-million-token rates; if those are left at 0, cost is reported as $0.00 even though a real request was billed.

Test itTest it

uv run pytest
uv run ruff check
uv run ruff format --check

The tests mock the Nebius streaming endpoint with respx, so they do not call the network by default.

Ship itShip it

make docker
make run-docker

The Dockerfile builds a slim multi-stage image with uv, runs as a non-root user, and exposes a /healthz healthcheck.

Going furtherGoing further

• Replace the deterministic route node with a small-model router once the routing policy needs semantics instead of keywords.
• Add a retrieval or tool node between planning and writing.
• Run independent tool calls in parallel once this graph has more than one tool edge.
• See the next recipe in the Agent Blueprint Recipes sequence for LangGraph memory for thread and user/application context.

ReferenceReference

• LangGraph quickstart