Verification & Testing Guide

Overview

The platform incorporates an automated verification test suite containing 148 comprehensive tests across 13 test files. These tests validate compliance with 21 CFR Part 11, FastMCP protocol handling, EU Annex 11, GAMP 5, ISO 13485:2016, and multi-agent squad workflows.

Executing the Test Suite

Run tests inside the running api container:

docker compose exec api pytest tests/ -v

Or run tests with coverage reporting:

docker compose exec api pytest tests/ -v --cov=src --cov-report=term-missing

Test Suites Breakdown

Test File	Number of Tests	Scope of Verification
test_backup_restore_sop.py	7	PostgreSQL backup ledgers, SOP-IT-018/019 policies, Part 11 append-only trigger immutability, Memgraph lineage traversal, and Mission 8 execution.
test_pydantic_evals_suite.py	15	Pydantic Evals framework, GxP evaluators (tone, FMEA math, citations, 5-Whys, CSA rigor), OpenTelemetry span verification, and CI/CD quality gate.
test_rustfs_watcher_triage.py	8	S3QueueWatcher file drop detection, autonomous AI document triage, PostgreSQL workflow queue state transitions, and REST dispatch endpoints.
test_rustfs_integration.py	8	RustFS S3 bucket operations, SHA-256 integrity verification, object manifest registration, and multipart file upload endpoints.
test_21cfr_part11_compliance.py	9	21 CFR Part 11 § 11.10(e) audit trail immutability triggers, Subpart C e-signatures, EvidenceGate supersession rejection, and Medical Device File (MDF) graph models.
test_specialized_agents.py	28	Pydantic AI agent runners, structured output schemas, ISO 13485:7.4 foundation model governance, and Clause 8.5.2 vs 8.5.3 remediation.
test_all_agents_e2e.py	19	Live REST API endpoint testing (/api/agents/run, /api/agents/vsr, /api/query, /api/audit-trail, /api/health) across all specialist agents.
test_mcp_server.py	12	FastMCP server initialization, SSE tool registration, parameter binding, and agent delegation across all 12 exposed tools.
test_dspy_optimization.py	12	Declarative signatures, regulatory metrics, composite evaluators, synthetic dataset generation, and prompt registry versioning.
test_optimization_ui_api.py	9	Optimization REST endpoints for benchmark evaluations, prompt version promotion, and sandbox executions.
test_team_missions.py	8	Cross-functional squad workflows for Missions 1 through 8, and REST execution endpoints (POST /api/demos/team-mission).
test_hybrid_rag.py	6	Document ingestion, Qdrant hybrid vector search, Memgraph multi-hop graph traversal, and EvidenceGate verification.
test_observability.py	6	Local Langfuse connectivity, OpenTelemetry agent/tool trace dispatch, session tracking, and pre-flight context logging.
test_helpdesk_tools.py	3	Pydantic AI tool integration for GxP and Non-GxP service desk ticket management.

Running Interactive CLI Demos

1. Collaborative Multi-Agent Teams Demo

Runs all 8 cross-functional squad missions in sequence:

docker compose exec api python demo_multi_agent_teams.py --mission all

2. Specialized 23-Agent Showcase

Executes each specialist agent against its primary regulatory scenario:

docker compose exec api python demo_pydantic_ai.py