Troubleshooting & Runbook
Diagnostic Triage Checklist
When troubleshooting issues across the platform, follow this sequential diagnostic protocol:
Step 1: Verify Container Health
Check the operational status of all Docker services:
docker compose psIf any container displays unhealthy or restarting, inspect its logs:
docker compose logs --tail=100 <container_name>Step 2: Test Core Service Health Endpoints
Execute curl commands against internal APIs:
# FastAPI Backendcurl -f http://localhost:8090/api/health
# DOCX Servicecurl -f http://localhost:5055/api/health
# Langfuse Observabilitycurl -f http://localhost:3001/api/public/health
# Wiki Documentationcurl -f http://localhost:4321/healthCommon Failure Scenarios and Resolutions
1. Database Connection Timeouts or Refused Connections
- Symptom:
apicontainer fails to start, reportingCannot connect to postgres:5432orConnection refused on memgraph:7687. - Cause: Database startup takes longer on machines with slower disk I/O, causing API container healthcheck to timeout before databases accept connections.
- Resolution:
- Verify PostgreSQL health directly:
Terminal window docker compose exec postgres pg_isready -U gxp_admin -d lifescience_qms_rag - Restart the API container after the databases are confirmed healthy:
Terminal window docker compose restart api
- Verify PostgreSQL health directly:
2. Port Binding Conflicts on Host
- Symptom:
Error response from daemon: driver failed programming external connectivity on endpoint ...: Bind for 0.0.0.0:XXXX failed: port is already allocated. - Cause: Existing processes on the host machine are occupying default ports.
- Resolution:
- Identify the process occupying the port:
Terminal window ss -tuln | grep :<port> - Override the port in
.envusing the corresponding port variable:POSTGRES_HOST_PORT(Default: 5434)QDRANT_HOST_PORT(Default: 6335)MEMGRAPH_HOST_PORT(Default: 7688)WIKI_HOST_PORT(Default: 4321)CHATBOT_HOST_PORT(Default: 3010 / 3002)
- Identify the process occupying the port:
3. Ollama Embedding Generation Timeout
- Symptom: Agent queries or seed scripts hang during vector embedding generation.
- Cause: Host Ollama instance is either not running, has not downloaded the required embedding model, or is refusing Docker bridge network connections.
- Resolution:
- Ensure the model is downloaded on the host:
Terminal window ollama pull qwen3-embedding:8b - Restart Ollama with origins open to allow container network bridge access:
Terminal window OLLAMA_ORIGINS="*" ollama serve - Confirm connectivity from inside the
apicontainer:Terminal window docker compose exec api curl -f http://host.docker.internal:11434/api/tags
- Ensure the model is downloaded on the host:
4. DeepSeek or OpenAI API Authorization Errors
- Symptom: Agent calls fail with
AuthenticationError: Invalid API key. - Cause: Missing or malformed
DEEPSEEK_API_KEYorOPENAI_API_KEYin.env. - Resolution:
- Verify that the key in
.envcontains no surrounding quotes, spaces, or newline characters. - Test the key directly from the host or container using curl.
- Verify that the key in
5. Re-Seeding the Complete Database Architecture
- Symptom: Missing requirements, unlinked assets, or empty vector collections.
- Resolution:
- Re-run the comprehensive seeding script:
Terminal window docker compose exec api python -m src.seed.seed_all - If a complete clean wipe is necessary, remove database volumes and re-initialize:
Terminal window docker compose down -vdocker compose up -d --builddocker compose exec api python -m src.seed.seed_all
- Re-run the comprehensive seeding script: