Document Ingestion & Chunking Pipelines

Overview

The document ingestion pipeline (src/ingestion/) converts unstructured and semi-structured Life Sciences documents into synchronized entries across PostgreSQL, Qdrant, and Memgraph.

Ingestion Architecture

[ Input Document (Markdown / Text / PDF / DOCX) ]
                       |
                       v
            [ Document Structure Parser ]
            Extracts Header, Section Hierarchy, Tables, Metadata
                       |
                       v
         [ GxP Context-Preserving Chunker ]
         Chunks by Section Boundaries (500-1000 tokens)
         Prepends Document Breadcrumbs & Section Title
                       |
                       v
         [ Cryptographic Hashing Engine ]
         Computes SHA-256 for Document and each Chunk
                       |
       +---------------+---------------+
       |               |               |
       v               v               v
 [ PostgreSQL ]   [ Qdrant ]      [ Memgraph ]
 Canonical Store  Dense/Sparse    Knowledge Graph
 Row & Hashes     Embedding Sync  Entity Nodes & Edges

Chunking Strategy

Standard character-count chunking splits sentences arbitrarily, destroying regulatory meaning and breaking numerical criteria away from their requirements. The platform uses Context-Preserving Structural Chunking:

1. Section-Boundary Splitting: Splits documents along markdown headings (##, ###, ####), XML section tags, or paragraph breaks.
2. Contextual Breadcrumb Injection: Every chunk is prepended with its hierarchical origin:

   Document: SOP-QA-012 (Rev 3.0) | Section 4.2: Audit Trail Review Procedures
   ---------------------------------------------------------------------------
   [Chunk Content...]

   This ensures that even when a chunk is retrieved in isolation, the embedding vector and LLM context retain document identity and section context.
3. Table & Formula Protection: Markdown tables, acceptance criteria ranges, and mathematical equations are preserved intact within a single chunk rather than fractured.

Synchronized Multi-Store Persistence

When a document is ingested:

1. PostgreSQL: Stores the raw document in controlled_documents and inserts chunk records into document_chunks with chunk_hash = SHA256(content).
2. Embeddings Engine: Generates dense embeddings (via Ollama qwen3-embedding:8b or OpenAI) and sparse token frequency vectors.
3. Qdrant: Upserts vector points with full payload attributes.
4. Memgraph: Creates the :Document node, creates :Chunk nodes, and creates :CONTAINS_CHUNK relationships.