Skip to content
Hybrid GraphRAG Wiki & Architecture

RustFS S3 Storage & Autonomous Queue Watcher

Overview

The platform deploys RustFS as an S3-compatible high-performance object storage engine running in container hybrid-rag-rustfs on host ports 9120 (API) and 9121 (Console). It manages raw binary documents, vendor manuals, execution attachments, and database archive files.

RustFS S3 Web Console

Cryptographic Object Storage Manifest

Under 21 CFR § 11.10(b) and § 11.10(c), electronic records stored in object stores must guarantee integrity and protection against unauthorized deletion or alteration.

Every object uploaded to RustFS is registered in the PostgreSQL object_storage_manifests table:

  • manifest_id: Unique UUID identifier.
  • bucket_name: S3 storage bucket (gxp-instrument-manuals, gxp-migration-staging).
  • object_key: S3 path key.
  • file_size_bytes: Byte length.
  • sha256_checksum: Cryptographic hash calculated before upload.
  • mime_type: Content type.
  • uploaded_by: User or service account identity.
  • created_at: Contemporaneous timestamp.

Presigned download URLs generated via /api/storage/presigned-url are time-bounded and authenticated, enforcing access control without exposing long-lived S3 credentials.

Autonomous S3 Queue Watcher (src/storage/watcher.py)

In commercial life sciences facilities, files are frequently dropped into external storage from SFTP drops, automated lab instruments, or legacy QMS exports. The S3QueueWatcher background engine automates the intake and triage of these files:

[ External Drop ] (SFTP / Lab Instrument / Legacy Export)
        |
        v
[ RustFS S3 Bucket: gxp-instrument-manuals ]
        |
        v
[ S3QueueWatcher Engine ] ---> Detects new untriaged binary
        |
        v
[ AI Document Triage Agent ]
Analyzes file snippet, determines GxP impact, and classifies category
        |
        v
[ PostgreSQL workflow_queue ]
Status: QUEUED -> PROCESSING -> COMPLETED
        |
        +-----------------------+-----------------------+
        |                       |                       |
        v                       v                       v
[ IQ/OQ Qualification ]  [ Migration ETL ]     [ Asset Calibration ]
Extracts parameters      100% Data             Registers NIST Cert
Generates protocols      Reconciliation        and Asset Schedule

Workflow Queue State Machine

The intake process is tracked in PostgreSQL workflow_queue:

  1. QUEUED: File detected and hash registered in manifest.
  2. PROCESSING: DocumentTriageAgent inspects content and dispatches downstream job.
  3. COMPLETED: Pipeline executed successfully and audit trail logged.
  4. FAILED: Failure recorded with full stack trace for QA investigation.