Search
Log in to Token FactoryLog in to AI Cloud
Agents & Orchestration

Sentinel: Regulatory Compliance Auditor

Agentic RAG that audits 200 enterprise SOPs against 36 regulatory frameworks and files gap findings as Jira tickets.

by Tikhon Roshchupkin

Build it with an agent

Paste into Claude Code, Cursor, or any coding agent — it builds the recipe for you.

Start in an empty directory. Clone `https://github.com/nebius/nebius-partner-cookbook` and work inside `blueprints/sentinel-compliance-auditor`. Read its `README.md`, `AGENTS.md`, and `blueprint.json` before running anything.

Install the dependencies with `uv sync`. Read `NEBIUS_API_KEY` and `PINECONE_API_KEY` from the environment, or from a `.env` file in that directory if a variable is not set there. If either is missing, stop and ask the user for it rather than searching elsewhere on the machine, and never print or commit a key. `TAVILY_API_KEY`, `OPENAI_API_KEY`, and `LANGSMITH_API_KEY` are optional; treat them the same way (environment, then `.env`, then ask) only if you plan to exercise the Grounded, Prototype, or LangSmith-traced paths, and skip them otherwise. Do not set up Jira: `JIRA_BASE_URL`, `JIRA_EMAIL`, `JIRA_API_TOKEN`, and `JIRA_PROJECT_KEY` should stay unset for this run, so `create_jira_ticket` is not reachable.

Run the offline tests first, before touching any API: `make test`. Report the pass count and confirm no network calls or API keys were needed.

With `NEBIUS_API_KEY` and `PINECONE_API_KEY` set, ingest the data so retrieval has something to find: `make ingest` then `make ingest-regulations`. Then start the LangGraph dev server with `make dev` (port 2024). In a second terminal, run a single-SOP audit against it rather than the full 200-SOP fan-out, since a full run costs tens of dollars and tens of minutes: use the LangGraph SDK or a direct API call to invoke the `sentinel_optimized` graph with a request to audit one SOP by ID (for example `SOP-AIML-001`) via `audit_single_sop`, and let it finish.

Report what you observed: whether ingestion completed and how many chunks it reported, whether the dev server started cleanly, the audit's findings for that one SOP with their compliance levels and regulations, the token counts and estimated cost the run reported, and any error or retry you saw along the way.

Recipe

Sentinel is an AI compliance auditor that assesses 200 enterprise SOPs against 36 regulation frameworks: HIPAA, SOC 2, GDPR, EU AI Act, NIST AI RMF, SR 11-7, California SB 53/SB 942/AB 853, BSA, ECOA, FCRA, PCI DSS, OWASP, FDA, the NIST SP 800 series, and EU AMLD4/ePrivacy/MDR/SCCs. A LangGraph ReAct agent fans out one sub-agent per SOP, up to 10 at a time, and each sub-agent retrieves the regulation text it needs from Pinecone, optionally checks live guidance with Tavily, and records structured findings that a batch tool files as Jira tickets. It ships with a React UI, a synthetic 200-SOP corpus for a fictional healthcare fintech, a 36-framework regulation knowledge base, and an evaluation harness that scores runs against a ground-truth compliance matrix.

What you will build

User Query (via UI or LangGraph API)
    |
    v
+-----------------------------------+
|  Sentinel Outer Agent             |  LangGraph ReAct (+ deepagents)
|  Prototype / Grounded / Optimized |  GPT-5.5 or DeepSeek-V4-Pro
|  / Production (Nemotron Ultra)    |  or Nemotron-3-Ultra-550b
+-----------------------------------+
    |
    +---> list_sops (search/discover SOPs, synonym mapping)
    |
    +---> audit_sops / audit_all_sops (ThreadPoolExecutor fan-out)
    |         |
    |         v  (per SOP, up to MAX_AUDIT_WORKERS in parallel)
    |    +----------------------------+
    |    |  Sub-Agent (sop_auditor)   |  LangGraph ReAct
    |    |  Same model as outer agent |  Retrieval capped at 30 calls
    |    +----------------------------+
    |         |
    |         +---> read_sop (full SOP text)
    |         +---> retrieve_regulation_rag (Pinecone semantic search)
    |         +---> search_web (Tavily, capped)
    |         +---> record_finding (per requirement, survives truncation)
    |         |
    |         v
    |    Returns SopAuditResult(findings, tokens, status), aggregated by
    |    value per audit call (no shared global; concurrency-safe). Transient
    |    failures retried per SOP (status-based, up to 4 attempts). Emits a
    |    token line parsed by the UI and validate_run via token_accounting.py
    |
    +---> create_jira_ticket / create_jira_tickets (batch)
    |         |
    |         v
    |    Jira Cloud REST API, tickets on a Kanban board
    |
    +---> search_web (outer agent, for ad-hoc questions)
    +---> list_regulations / retrieve_regulation_text_tool

Models: Nemotron-3-Ultra-550b (Production), DeepSeek-V4-Pro (Optimized), and GPT-5.5 (Prototype/Grounded) on Nebius and OpenAI, all built through one build_chat_model factory (chat_model.py).
Orchestration: a LangGraph ReAct agent with per-SOP sub-agents, with an optional deepagents upgrade.
Retrieval: Pinecone vector search on Qwen/Qwen3-Embedding-8B (4096 dimensions).
Grounding: Tavily live regulation search.
Observability: LangSmith tracing with cost tracking, including per-SOP sub-agent traces parented under the audit tool’s run (set SUBAGENT_TRACING=off to disable); the token-line emit and parse logic is centralized in sentinel/token_accounting.py and summed per audit call by the UI and validate_run.py.
Actuation: the Jira Cloud REST API for filing compliance gap tickets.
Deployment: LangGraph Cloud plus a FastAPI and React UI.

Prerequisites

  • Python 3.11+
  • API keys: Nebius and Pinecone are required; OpenAI, Tavily, and LangSmith are optional
Variable Required Description
NEBIUS_API_KEY Yes Nebius Token Factory API key
NEBIUS_MODEL Optional Model for the Optimized agent (deepseek-v4-pro default, nemotron, kimi-k2, glm-5); unknown values fail fast at startup
MAX_AUDIT_WORKERS Optional Sub-agent fan-out width for audit_sops / audit_all_sops (default 10)
OPENAI_API_KEY Optional OpenAI API key (for the Prototype agent)
PINECONE_API_KEY Yes Pinecone vector DB key (agentic RAG)
TAVILY_API_KEY Optional Live regulation grounding
LANGSMITH_API_KEY Optional LangSmith tracing and cloud auth
LANGCHAIN_PROJECT Optional LangSmith project name used by the eval scripts (default sentinel-agent)
JIRA_BASE_URL For Jira Atlassian site URL (e.g. https://your-org.atlassian.net)
JIRA_EMAIL For Jira Atlassian account email tied to the API token
JIRA_API_TOKEN For Jira API token from id.atlassian.com
JIRA_PROJECT_KEY For Jira Target Jira project key (e.g. SENT)
JIRA_DEFAULT_ISSUE_TYPE Optional Jira issue type for filed tickets (default Task)
LANGGRAPH_URL Optional LangGraph server the UI talks to (default http://localhost:2024)
UI_API_KEY Yes (UI) Shared secret that gates the UI’s /api/* endpoints. The UI refuses to start without it, even locally; every API call must supply a matching X-API-Key. Generate with openssl rand -hex 32

The four Jira variables are only required if you want create_jira_ticket to file tickets; the tool fails fast with a clear error if any of them are missing, so the rest of the agent runs fine without a Jira account.

Run the cookbook

make install   # creates .venv and installs the package with all extras

Copy .env.example to .env and fill in your API keys:

cp .env.example .env

Ingest data:

make ingest               # SOPs into Pinecone
make ingest-regulations   # Regulation texts into Pinecone (namespace: regulations)

Run the demo:

make dev     # LangGraph dev server on port 2024
make ui      # UI on port 8080 (connects to LangGraph)

UI_API_KEY is required; set it in .env first (e.g. openssl rand -hex 32). The UI server refuses to start without it, even locally. Then open http://localhost:8080: the UI prompts for the key on load and sends it as an X-API-Key header on every request, and the server rejects any /api/* call without it, so no agent run, Pinecone call, or Jira call happens for an unauthenticated caller.

Three screens: Audit (composer, agent picker, live stream, and a Jira findings register), Compare (a three-way agent race on one question), and Eval (a 120-question benchmark dashboard). A running audit can be stopped from the composer; Stop aborts the stream and the server cancels the LangGraph run, so the spend stops too. Agent labels and pricing come from /agents.js, generated by the server from config.PRICING, so prices change in one place.

Walk-through

Sentinel fans out by SOP using a sub-agent architecture. Each SOP is audited by a dedicated LangGraph ReAct sub-agent that:

  1. Reads the full SOP text
  2. Determines which regulations apply based on content and business unit
  3. Queries the regulation knowledge base via Pinecone vector search, with multiple keyword queries per regulation
  4. Optionally searches the web for the latest guidance
  5. Calls record_finding for each assessed requirement; findings are captured incrementally so partial progress survives truncation or errors

audit_all_sops fans out sub-agents through a ThreadPoolExecutor (configurable via MAX_AUDIT_WORKERS). Transient failures, such as provider 504s, are retried per SOP, up to 4 attempts with backoff and longer waits on rate limits, based on a structured status carried on each result, with tokens from failed attempts still counted. Retrieval is capped at 30 calls per sub-agent.

Key tools:

  • audit_all_sops: full audit across all 200 SOPs in parallel
  • audit_single_sop: audit one SOP via a dedicated sub-agent
  • list_sops: search and discover SOPs by title, ID, or business unit
  • list_regulations: list all regulations in the knowledge base
  • retrieve_regulation_text_tool: look up specific regulation requirements
  • create_jira_ticket: file a Jira ticket for a compliance gap or partial finding

Project structure

sentinel_agent/
├── sentinel/                  # Core agent package
│   ├── config.py              # API keys, model config, pricing, paths
│   ├── models.py              # Pydantic models (AuditFinding, SOPChunk, AuditMetrics)
│   ├── chat_model.py          # build_chat_model(): single ChatOpenAI factory
│   ├── token_accounting.py    # Token-line emit format + parser (shared by tools/UI/eval)
│   ├── graph/
│   │   ├── agent.py           # ReAct agent (deepagents fallback to LangGraph)
│   │   ├── naive_agent.py     # Act 0 baseline: single retrieval + single LLM call
│   │   └── tools.py           # LangChain tools: sub-agent auditing + retrieval
│   ├── retrieval/
│   │   ├── local.py           # SOP file loading and search (cached parse)
│   │   ├── regulations.py     # Pinecone regulation text retrieval
│   │   ├── ingest.py          # SOP -> Pinecone ingestion + shared section chunker
│   │   └── ingest_regulations.py  # Regulation text -> Pinecone ingestion
│   ├── eval/
│   │   └── metrics.py         # Confusion-matrix / F1 scoring (single implementation)
│   ├── actuation/
│   │   └── jira_client.py     # Jira Cloud REST client
├── ui/
│   ├── server.py              # FastAPI backend: agent registry, SSE streaming, Jira findings
│   └── static/                # React screens (Audit / Compare / Eval), served by server.py
├── scripts/
│   ├── validate_run.py        # Audit quality evaluation against compliance matrix
│   ├── compare_audit_runs.py  # Side-by-side comparison of N audit runs
│   ├── run_qa_eval.py         # 120-question Q&A eval (naive vs agentic modes)
│   ├── inspect_tool_calls.py  # LangSmith tool call inspector (args, timing, tokens)
│   ├── generate_sops.py       # SOP generation (one-time)
│   ├── extract_pdf_text.py    # PDF -> text extraction for regulations
│   └── sop_taxonomy.py        # SOP definitions + metadata
├── tests/                     # regression tests (no API keys needed)
├── data/
│   ├── sops/                  # 200 generated SOPs (10 business units)
│   ├── regulations/           # 36 regulation frameworks (txt, md, pdf, xml)
│   ├── company_profile.md     # Meridian Health Technologies background
│   ├── compliance_matrix.json # Ground truth
│   └── compliance_matrix_revised.json # Revised ground truth (16 SOC 2 corrections)
├── langgraph.json             # LangGraph deployment config
├── pyproject.toml             # Dependencies
├── Makefile                   # Build/run targets
└── .env.example                # API key template

Company profile

Meridian Health Technologies is a fictional AI-powered healthcare fintech that provides AI-driven clinical decision support and diagnostic tools, operates healthcare payment processing, lending, and fraud detection, manages patient data across EU and US jurisdictions, and deploys ML models for credit scoring and risk assessment.

Regulation coverage

Nine core regulation frameworks have full text in the Pinecone index: HIPAA Security Rule (administrative, physical, and technical safeguards), SOC 2 Trust Services Criteria (CC1 through CC9), GDPR, the EU AI Act, NIST AI RMF, SR 11-7, and the California AI laws (SB 53, SB 942, AB 853). Historical editions are included for temporal analysis, for example HIPAA 2017/2020/2024 and the EU AI Act proposal versus final text.

27 additional external standards referenced by SOPs are also available in data/regulations/: 11 NIST special publications (SP 800-53, 800-88, 800-61, CSF 2.0, 800-63B, 800-207, 800-34, 1270, Privacy Framework, 800-161, 800-218), 5 FDA/eCFR titles (21 CFR Parts 820, 11, 807, plus the AI/ML SaMD and CDS guidance), 5 EU directives (MDR, SCCs, ePrivacy, AMLD4, Funds Transfer), 2 OWASP guides (Top 10, API Security), and 4 financial laws (BSA, ECOA/Reg B, FCRA, PCI DSS). See data/regulations/README.md for the full inventory.

Data

SOPs live in data/sops/, organized by business unit subdirectory (e.g. data/sops/01_ai_ml_engineering/sop_aiml_001_*.md). Each SOP is a Markdown file with YAML frontmatter carrying sop_id, title, business_unit, and regulations fields. There are 200 SOPs across 10 business units (AI/ML Engineering, Clinical AI Products, Customer Operations, Data Governance & Privacy, Financial Services, Human Resources, IT Operations, Information Security, Legal & Compliance, Product & Engineering), 20 SOPs each. To regenerate SOPs, which requires NEBIUS_API_KEY:

python3 scripts/generate_sops.py                    # Generate all SOPs
python3 scripts/generate_sops.py --resume            # Skip already-generated files
python3 scripts/generate_sops.py --concurrency 5     # Parallel API calls
python3 scripts/generate_sops.py --sop SOP-AIML-001  # Generate a single SOP

Regulation texts live in data/regulations/ as .txt, .md, .pdf, and .xml files. PDFs are extracted to .txt via scripts/extract_pdf_text.py (pypdf) before ingestion. To ingest into Pinecone:

make ingest-regulations   # Chunks, embeds, upserts into Pinecone namespace "regulations"

data/compliance_matrix_revised.json holds 420 ground-truth (SOP, regulation) pairs across 6 regulations:

Regulation SOPs Description
SOC 2 121 Trust Services Criteria CC1-CC9
HIPAA 94 Security Rule administrative, physical, technical safeguards
GDPR 76 Data protection and privacy
EU AI Act 63 High-risk AI system requirements
NIST AI RMF 37 AI risk management framework
SR 11-7 29 Model risk management

Compliance level distribution: 170 compliant (40%), 161 partial (38%), 89 gap (21%). SOPs are varied on purpose: compliant SOPs cite regulation articles and have specific controls, partial SOPs use vague language, and gap SOPs are missing key requirements. data/compliance_matrix_revised.json includes 16 SOC 2 corrections (15 gap-to-partial, 1 partial-to-compliant) from a manual review of SOP content against the SOC 2 Trust Services Criteria.

Design decisions

  • Each audit_all_sops call returns its SopAuditResult set by value rather than through a shared global, which keeps the parallel fan-out concurrency-safe.
  • Retries are status-based rather than exception-based: a structured status travels with each result so a transient 504 can be retried (up to 4 attempts, longer backoff on rate limits) while tokens already spent on the failed attempt are still counted toward cost.
  • record_finding is called per requirement as the sub-agent works, so a truncated or errored run still keeps whatever findings it recorded rather than losing the whole SOP.
  • Token accounting is centralized in one emit and parse format (sentinel/token_accounting.py) so the UI, validate_run.py, and the eval scripts all read the same line format instead of re-deriving totals.
  • The UI’s UI_API_KEY gate is enforced with secrets.compare_digest and the process refuses to start at all without it, even for local use, since make ui binds 0.0.0.0:8080 by default.
  • Copyrighted regulation texts (AICPA SOC 2, PCI SSC PCI DSS) are kept out of version control entirely rather than shipped or stubbed; the repository documents where to obtain them instead.
  • An unrecognized NEBIUS_MODEL value raises at startup rather than silently falling back to the default, so a typo cannot make you benchmark the wrong model without noticing.

Failure modes

  • UI_API_KEY not set: make ui exits immediately with a fatal error instead of starting an unauthenticated server.
  • Unknown NEBIUS_MODEL: sentinel/config.py raises ValueError at import time rather than defaulting to DeepSeek-V4-Pro.
  • Missing Jira variables: create_jira_ticket / create_jira_tickets fail with a clear error naming the missing variable instead of silently no-opping.
  • Provider 504s and rate limits during a full audit: handled per SOP with up to 4 retries and backoff; a SOP that still fails after retries is reported as failed rather than dropped, and scripts/validate_run.py reports a “Failed %” for SOP-regulation pairs missing from the run output.
  • Truncated sub-agent runs: because record_finding persists findings as they are produced, a truncated run keeps its partial findings instead of losing them.

Troubleshooting

If make dev fails to start, confirm NEBIUS_API_KEY and PINECONE_API_KEY are set; the outer agent and its retrieval tool both need them even before you send a query. If Pinecone queries return nothing, confirm make ingest and make ingest-regulations have both been run against the index named by PINECONE_INDEX_NAME. If the UI shows no findings after an audit, check that the four JIRA_* variables are all set together; a partially configured Jira integration fails the ticket-filing tool but does not fail the audit itself.

Test it

make test    # Run all regression tests

Tests cover JSON parsing and repair, retry classification and the sub-agent fan-out, the benchmark parsing pipeline (including an emit-and-parse round trip against the real audit summary format), Jira ticket building with httpx mocked, SSE token accounting, SOP loading, metrics, and config validation. No API keys or external services are required; the suite runs entirely against mocked HTTP calls and dummy credentials set in tests/conftest.py.

scripts/validate_run.py measures audit quality by comparing a LangSmith run’s output against the compliance matrix ground truth:

# Validate a single run (uses the revised matrix by default)
python3 scripts/validate_run.py <langsmith-run-id>

# Compare two runs side by side
python3 scripts/validate_run.py <run-id-1> <run-id-2>

# Use the original (uncorrected) matrix
python3 scripts/validate_run.py --original <run-id>

# Compare 3+ runs (quality, cost, tokens, latency, tool-call counts)
python3 scripts/compare_audit_runs.py <run-id-1> <run-id-2> <run-id-3>

The script fetches run data from LangSmith (model, timing, tokens, cost, audit content), parses the audit_all_sops output, classifies each finding by regulation, aggregates to a worst compliance level per (SOP, regulation) pair, and reports matched percent, false positive and false negative percent, per-class F1 for compliant/partial/gap detection, failed percent, per-regulation accuracy, and cost/tokens/latency pulled from LangSmith run metadata.

Ship it

# Local development
make dev          # LangGraph dev server on port 2024
make ui           # UI on port 8080

# Cloud deployment
make deploy       # Deploy to LangGraph Cloud

make ui binds 0.0.0.0:8080. Before putting it on a public address, set a strong UI_API_KEY (e.g. openssl rand -hex 32) to gate the API, and front the server with a reverse proxy or load balancer that terminates TLS and applies rate limiting; those are intentionally left to the proxy layer rather than the app.

Cost

Operation Model Tokens Cost Latency
Full audit (Production) Nemotron-3-Ultra-550b ($1.00/$3.00 per M tokens) ~12M ~$12 ~23m
Full audit (Optimized) DeepSeek-V4-Pro ($1.75/$3.50 per M tokens) ~36M ~$64 ~53m
Full audit (Prototype) GPT-5.5 ($5.00/$30.00 per M tokens) ~18M ~$140 ~13m
SOP ingestion Qwen3-Embedding-8B ~2M ~$0.02 ~5m

Each SOP audit fans out a dedicated sub-agent with multiple tool calls for regulation retrieval and web search, so token counts are dominated by sub-agent usage across all 200 SOPs. Token usage and cost are shown live in the UI. Use scripts/validate_run.py to get exact cost, token, and latency breakdowns for any LangSmith run.

Jira Cloud

The create_jira_ticket tool files compliance findings as tickets on a Jira Kanban board via the Jira Cloud REST API v3. Tickets are created for gap or partial findings at medium severity or higher.

  • Client: sentinel/actuation/jira_client.py, a sync REST client with basic auth (email and API token)
  • Description format: Atlassian Document Format (ADF)
  • Labels: sentinel, compliance-finding, severity, regulation slug, SOP slug
  • Priority mapping: critical to Highest, high to High, medium to Medium, low to Low

Setup: create an API token at id.atlassian.com, then set in .env:

JIRA_BASE_URL=https://your-org.atlassian.net
JIRA_EMAIL=your-email@example.com
JIRA_API_TOKEN=your-api-token
JIRA_PROJECT_KEY=SENT

Going further

More cookbooks

Continue with another hands-on guide.

Agents & Orchestration

Your First Agent on Nebius

by Clément Sauvage

A FastAPI service that streams one Nebius chat completion as SSE, with structured logs, Prometheus metrics, rate limiting, and a production Dockerfile.

< 5 min to first run
Agents & Orchestration

Building a Deep Research Agent powered by Tavily and Token Factory

by AbdulMajed Raja

Wire Tavily web search into a Deep Agent running on a Nebius Token Factory model, then ask it a research question and get back a sourced, multi-step report.

~$0.07 per run< 5 min to first run
Agents & Orchestration

Build an enterprise knowledge base with OpenWiki and Kimi K2.7 Code

by AbdulMajed Raja

Point LangChain's OpenWiki agent at a repository through Nebius Token Factory and get a reviewable, evidence-grounded Markdown knowledge base your teams can own in Git.

From $0.79 per run~15 min to first run
Agents & Orchestration

Stronger Agents with LangChain and LangGraph

by Clément Sauvage

Route requests through a typed LangGraph, cap token budgets by path, and stream phase events so users see progress before the model finishes.

< 6 min to first run
Agents & Orchestration

Short-Term Memory with LangChain

by Clément Sauvage

Add thread-scoped short-term memory to a Nebius-backed LangChain agent so follow-up questions in one conversation resolve correctly.

< 6 min to first run
Agents & Orchestration

Long-Term Memory with LangChain and Postgres

by Clément Sauvage

Add durable, Postgres-backed user memory to a LangGraph agent so it recalls preferences across threads and restarts, with list, summary, and delete endpoints.

< 8 min to first run
Agents & Orchestration

Adding Guardrails with LangChain

by Clément Sauvage

Add input and output guardrails, including PII redaction, to a Nebius-backed LangChain agent with fail-closed behavior.

< 8 min to first run
Agents & Orchestration

Actions with MCP and Stripe

by Clément Sauvage

Let a guarded book agent request a Stripe test-mode checkout link through MCP, with a human approval step standing between the model and the payment call.

< 10 min to first run
Agents & Orchestration

Build a Jev-Style Typed Decision Model With GLM-5.3 Flash

by AbdulMajed Raja

Turn GLM-5.3 Flash on Token Factory into a typed decision model: declared answers in, probabilities out, read from logprobs instead of generated text. Measured on 100 tickets at under a cent per 100 decisions.

~$0.02 per run< 5 min to first run