Skip to content

Troubleshooting & Runbook

Diagnostic Triage Checklist

When troubleshooting issues across the platform, follow this sequential diagnostic protocol:

Step 1: Verify Container Health

Check the operational status of all Docker services:

Terminal window
docker compose ps

If any container displays unhealthy or restarting, inspect its logs:

Terminal window
docker compose logs --tail=100 <container_name>

Step 2: Test Core Service Health Endpoints

Execute curl commands against internal APIs:

Terminal window
# FastAPI Backend
curl -f http://localhost:8090/api/health
# DOCX Service
curl -f http://localhost:5055/api/health
# Langfuse Observability
curl -f http://localhost:3001/api/public/health
# Wiki Documentation
curl -f http://localhost:4321/health

Common Failure Scenarios and Resolutions

1. Database Connection Timeouts or Refused Connections

  • Symptom: api container fails to start, reporting Cannot connect to postgres:5432 or Connection refused on memgraph:7687.
  • Cause: Database startup takes longer on machines with slower disk I/O, causing API container healthcheck to timeout before databases accept connections.
  • Resolution:
    • Verify PostgreSQL health directly:
      Terminal window
      docker compose exec postgres pg_isready -U gxp_admin -d lifescience_qms_rag
    • Restart the API container after the databases are confirmed healthy:
      Terminal window
      docker compose restart api

2. Port Binding Conflicts on Host

  • Symptom: Error response from daemon: driver failed programming external connectivity on endpoint ...: Bind for 0.0.0.0:XXXX failed: port is already allocated.
  • Cause: Existing processes on the host machine are occupying default ports.
  • Resolution:
    • Identify the process occupying the port:
      Terminal window
      ss -tuln | grep :<port>
    • Override the port in .env using the corresponding port variable:
      • POSTGRES_HOST_PORT (Default: 5434)
      • QDRANT_HOST_PORT (Default: 6335)
      • MEMGRAPH_HOST_PORT (Default: 7688)
      • WIKI_HOST_PORT (Default: 4321)
      • CHATBOT_HOST_PORT (Default: 3010 / 3002)

3. Ollama Embedding Generation Timeout

  • Symptom: Agent queries or seed scripts hang during vector embedding generation.
  • Cause: Host Ollama instance is either not running, has not downloaded the required embedding model, or is refusing Docker bridge network connections.
  • Resolution:
    • Ensure the model is downloaded on the host:
      Terminal window
      ollama pull qwen3-embedding:8b
    • Restart Ollama with origins open to allow container network bridge access:
      Terminal window
      OLLAMA_ORIGINS="*" ollama serve
    • Confirm connectivity from inside the api container:
      Terminal window
      docker compose exec api curl -f http://host.docker.internal:11434/api/tags

4. DeepSeek or OpenAI API Authorization Errors

  • Symptom: Agent calls fail with AuthenticationError: Invalid API key.
  • Cause: Missing or malformed DEEPSEEK_API_KEY or OPENAI_API_KEY in .env.
  • Resolution:
    • Verify that the key in .env contains no surrounding quotes, spaces, or newline characters.
    • Test the key directly from the host or container using curl.

5. Re-Seeding the Complete Database Architecture

  • Symptom: Missing requirements, unlinked assets, or empty vector collections.
  • Resolution:
    • Re-run the comprehensive seeding script:
      Terminal window
      docker compose exec api python -m src.seed.seed_all
    • If a complete clean wipe is necessary, remove database volumes and re-initialize:
      Terminal window
      docker compose down -v
      docker compose up -d --build
      docker compose exec api python -m src.seed.seed_all