Your First Agent on Nebius

A production-shaped FastAPI agent with streaming, observability, and no hidden framework magic.

This recipe is a small FastAPI service that streams a Nebius chat completion back to the client as Server-Sent Events. The Nebius integration itself is one line: point the OpenAI SDK’s base_url at Nebius. Everything else in the service, the structured logs, the Prometheus metrics, the rate limiter, the security headers, the Dockerfile, and the test suite, is the shape of a service you could hand to an operations team.

It is not an agent framework. There is no planner, no tool loop, no graph. The goal is to get the boundaries right (config, transport, observability, lifecycle) so that a project built on top of it can add capability without re-litigating infrastructure.

What you will buildWhat you will build

A single-endpoint API:

Everything is typed, every environment variable is validated at boot, every request is tagged with a request ID, and every Nebius call is wrapped in tenacity-backed retries. The service has no database dependency.

PrerequisitesPrerequisites

• Python 3.12+
• uv, installed with curl -LsSf https://astral.sh/uv/install.sh | sh
• A Nebius API key from the Nebius console
• Docker, optionally, if you want to build the production image

Run the cookbookRun the cookbook

git clone https://github.com/nebius/nebius-partner-cookbook.git
cd nebius-partner-cookbook/cookbooks/01-first-agent-on-nebius

cp .env.example .env
# Open .env and fill NEBIUS_API_KEY

While .env is open, point the client at Token Factory. The repo’s .env.example still ships the older api.studio.nebius.ai hostname, so the first request fails without this:

NEBIUS_BASE_URL=https://api.tokenfactory.nebius.com/v1/

Then install and start:

uv sync
make dev

In another terminal:

curl -N -X POST http://localhost:8000/agent/run \
  -H 'content-type: application/json' \
  -d '{"prompt":"Explain Nebius in one paragraph."}'

You will see SSE events arriving:

event: status
data: {"phase":"thinking"}

event: token
data: {"text":"Nebius"}

event: token
data: {"text":" Token"}

…

event: status
data: {"phase":"done"}

event: done
data: {}

Multi-turn requestsMulti-turn requests

The server is stateless: it keeps no session. To hold a conversation, the client replays prior turns in the request body:

curl -N -X POST http://localhost:8000/agent/run \
  -H 'content-type: application/json' \
  -d '{
        "prompt": "And who maintains it?",
        "history": [
          {"role": "user", "content": "What is Nebius?"},
          {"role": "assistant", "content": "It is ..."}
        ]
      }'

history is optional and capped at 40 turns and 8 KB per message by the request schema. See Design decisions below for why the server stays stateless.

Walk-throughWalk-through

The directoryThe directory

app/
├── main.py              # FastAPI app, lifespan, middleware
├── config.py            # Pydantic Settings, validated at boot
├── routes/
│   ├── agent.py         # POST /agent/run with SSE
│   └── health.py        # /healthz, /readyz
├── schemas/
│   └── agent.py         # Pydantic I/O models
├── core/
│   ├── nebius_client.py # OpenAI SDK pointed at Nebius
│   └── agent.py         # The agent logic
└── observability/
    ├── logging.py       # structlog (JSON in prod)
    ├── metrics.py       # Prometheus counters/histograms
    └── middleware.py    # Request IDs, security headers, body-size limit

The split is by boundary, not by layer: core/ is the only place that knows about Nebius, routes/ is the only place that knows about HTTP and SSE, and observability/ is cross-cutting. You can unit-test core/agent.py without standing up a server, and you can change the transport from SSE to WebSocket without touching core/.

The Nebius integrationThe Nebius integration

The SDK swap is one line: base_url points at Nebius.

AsyncOpenAI(
    api_key=settings.nebius_api_key,
    base_url=str(settings.nebius_base_url),  # https://api.tokenfactory.nebius.com/v1/
    timeout=httpx.Timeout(connect=5.0, read=60.0, write=10.0, pool=5.0),
    max_retries=0,  # retries are owned by tenacity, not the SDK
)

This is why the quickstart has you set NEBIUS_BASE_URL yourself: the repo’s .env.example and config default still carry the older api.studio.nebius.ai hostname.

Nebius is OpenAI-wire-compatible, so the OpenAI SDK works unchanged: the same chat.completions.create(...) call, the same streaming protocol.

Two decisions worth calling out:

• Timeouts are split, not global. A 5 s connect timeout fails fast on a dead endpoint; a 60 s read timeout tolerates a slow generation. A single global timeout forces you to choose one or the other.
• max_retries=0 on the SDK. Retry policy lives in tenacity instead (see below), so it stays visible, testable, and swappable in one place. Two retry layers stacked silently is a production incident waiting to happen.

build_nebius_client() returns a process-wide singleton. The point is the underlying httpx connection pool: reusing it avoids a TLS handshake on every request. A fresh client per request is a common and expensive mistake.

RetriesRetries

@retry(
    retry=retry_if_exception_type((httpx.HTTPError, httpx.TimeoutException)),
    wait=wait_exponential(multiplier=0.5, min=0.5, max=8.0),
    stop=stop_after_attempt(3),
    reraise=True,
)

Retries fire on transport errors only. A 4xx (bad key, malformed request) or a content error is not retried, since replaying it just burns quota and latency for the same failure. Exponential backoff with a cap keeps a struggling upstream from being hammered.

The observabilityThe observability

Every request gets a UUID request ID, returned in the x-request-id header and bound to the structlog context so every log line for that request carries it. In production (ENV=production) logs render as JSON; in dev they render as human-readable lines, using one renderer swap and the same call sites.

A sample log line in production:

{
  "event": "agent_started",
  "level": "info",
  "request_id": "9a8f3c…",
  "path": "/agent/run",
  "model": "Qwen/Qwen3.5-397B-A17B",
  "timestamp": "2026-06-04T12:31:42.103Z"
}

Prometheus metrics include:

• http_requests_total{method,path,status}, a request counter
• http_request_duration_seconds{method,path}, a latency histogram
• nebius_tokens_total{model,type}, tokens streamed back from Nebius
• nebius_request_duration_seconds{model}, Nebius call duration

The path label is the route template (/agent/run), not the raw URL, because high-cardinality labels (per-user, per-ID paths) are the fastest way to melt a Prometheus instance.

The streamingThe streaming

The route handler returns a StreamingResponse with text/event-stream. Events are named (status, token, done, heartbeat, error) rather than raw text, so a client can switch on the event name instead of parsing prose. The agent itself is an async generator yielding typed Events; the route’s only job is to serialize each one into the SSE wire format.

Two production details that are easy to miss:

• Heartbeats. A heartbeat event is emitted every 15 s of silence. Idle SSE streams are killed by load balancers and proxies, and the heartbeat keeps the connection scored as live. The response also sets x-accel-buffering: no so nginx forwards chunks immediately instead of buffering the whole body.
• Disconnect cancellation. A background task polls request.is_disconnected() and sets a cancel_event. The agent checks it between tokens and stops pulling from Nebius the moment the client goes away, so a user closing a tab does not still cost a full generation.

The hardeningThe hardening

• The daily per-IP rate limiter protects backend routes before they call Nebius. It uses Redis when RATE_LIMIT_REDIS_URL is set and process-local memory otherwise (RATE_LIMIT_REQUESTS_PER_DAY, default 25).
• CORS is allowlist-based. CORS_ORIGINS is parsed and validated at boot, never *.
• Security headers include x-content-type-options, referrer-policy, and strict-transport-security.
• Request bodies above 64 KB are rejected with 413 before the body is read, via a content-length check in middleware.
• Lifespan handlers log startup and shutdown; on shutdown the HTTP client is closed and in-flight requests are allowed to drain.

Middleware order is load-bearing. ASGI wraps middleware last-added-runs-first, so the stack is arranged so that CORS, size checks, security headers, request IDs, metrics, and rate limiting all happen before route work. Reordering add_middleware calls silently changes behavior; there is a comment in main.py spelling out the request path.

Rate limitingRate limiting

The limiter is enabled by default and allows 25 backend requests per IP per rolling 24-hour window. Set these values in .env:

RATE_LIMIT_ENABLED=true
RATE_LIMIT_REQUESTS_PER_DAY=25
RATE_LIMIT_REDIS_URL=
RATE_LIMIT_TRUST_PROXY_HEADERS=false

Leave RATE_LIMIT_REDIS_URL empty for local in-memory counters, or point it at Redis for a shared counter across replicas:

RATE_LIMIT_REDIS_URL=redis://localhost:6379/0

When the quota is exceeded, the backend returns JSON 429 before opening an SSE stream:

{
  "detail": "daily rate limit exceeded",
  "limit": 25,
  "window": "24h",
  "retryAfterSeconds": 12345
}

Design decisionsDesign decisions

Why a stateless server? No session store means any instance can serve any request, so horizontal scaling is just adding a replica, and a crash loses nothing. The cost is bandwidth: the client resends history each turn. For a chat workload that is a few KB. When it stops being cheap (long transcripts, RAG context), the answer is a server-side store keyed by session ID rather than in-process state.

Why no agent framework? A framework would hide exactly the boundaries this recipe exists to show. core/agent.py is about 50 lines and is designed to be subclassed, so a project built on top of it can add planning, knowledge, and tools on the same shape.

Why duplicate infrastructure? The recipe is self-contained: cloning this one directory is enough to run it, with no shared base package. That costs some duplication and buys a reader the ability to study the recipe in isolation. It is a documentation choice, not an architectural recommendation; in a real monorepo, the common middleware would belong in a shared library.

Failure modesFailure modes

The except Exception in the route logs the full stack trace but emits only {"detail": "internal error"} to the client; internal details never cross the wire.

Test itTest it

make test

Tests use respx to mock every Nebius call. No network is hit, so the suite runs the same on a laptop in airplane mode as it does in CI, and it doubles as a runnable spec of the SSE contract.

Ship itShip it

Build the production image:

make docker

The Dockerfile is multi-stage, uses astral-sh/uv for the build, and ends in a slim Python 3.12 image running as a non-root user with a healthcheck wired to /healthz.

Going furtherGoing further

• Domain knowledge. The next recipe in this sequence, Domain Knowledge with Pinecone Nexus, gives the agent a grounded, compiled knowledge base instead of a frozen training cutoff.
• Tracing. Feed an inbound traceparent into the request-ID middleware and emit OpenTelemetry spans around the Nebius call; the structlog context is already the right place to bind a trace ID.
• Authentication. Drop a JWT verifier in as a FastAPI Depends; it composes cleanly with the existing rate limiter and request-ID middleware.
• Backpressure. Under heavy fan-out, cap concurrent Nebius calls with an asyncio.Semaphore in the client so one traffic spike cannot open unbounded upstream connections.