Skip to main content

Knowledge Base Indexing

Overview​

A knowledge base is the set of documents a customer uploads for the AI to answer questions from (RAG). PDF, DOCX, and plain text files are all supported.

This job runs under its own worker profile (SVC=knowledge-worker). It converts documents into chunks, generates embeddings, and stores the results in both PostgreSQL (the knowledge_chunk table) and Meilisearch so they can be searched.

It gets a dedicated profile because the work is resource-hungry — downloading files, parsing them, calling the embedding API — and must not compete for resources with the consumers that answer users in real time.

Business Flow​

The handler branches on payload.action.

index_document​

  1. Load the knowledge_document row by documentId, skipping if it is missing or already deleted.
  2. parseFile fetches the file from its URL and extracts text according to file type:
    • PDF via extractPDFText (ledongthuc/pdf)
    • DOCX via extractDOCXText, which reads the zip and parses the XML
    • text and other types are used as-is
  3. ChunkingService splits the text into chunks according to the configured size and overlap.
  4. Call the embedding API to convert each chunk into a vector.
  5. Write to knowledge_chunk (content, vector, and metadata) and index into Meilisearch.
  6. updateStats refreshes the document and chunk counts on the knowledge_base table.

delete_document​

Removes that document's chunks from both PostgreSQL and Meilisearch, then updates the statistics.

reindex_all​

Walks every document in the knowledge base and reindexes the whole set — used after a change to the chunking or embedding configuration.

Other cases​

  • An unrecognised action logs a warning and returns nil (acknowledged).
  • A payload that fails to decode returns mq.Permanent and goes to the DLQ.

Key Files & Functions​

  • internal/knowledgeindex/service.go
    • Service.IndexDocument(), DeleteDocument(), ReindexAll()
    • indexDocumentBody(), parseFile(), classifyFileType(), fetchFile()
    • extractPDFText(), extractDOCXText(), updateStats()
    • the Payload struct with fields Action, DocumentID, KnowledgeBaseID, LineOaID, OrganizationID
  • internal/knowledgeindex/chunking.go — ChunkingService, which splits text into chunks
  • internal/knowledgeindex/consumer.go — Consumer.HandleIndexJob, dispatching on action
  • internal/meilix/meilix.go — the Meilisearch client for index, delete, and search
  • internal/embedx/embedx.go — the embedding API client
  • cmd/worker/main.go — runKnowledgeWorker(), the entry point for the knowledge-worker profile
  • Queue: knowledge_index (profile knowledge-worker)

Connections to Other Services​

  • Receives jobs from: the cms-api-go knowledge base domain, covering uploads, document deletions, and reindex commands
  • Tables: knowledge_document (metadata and file URL), knowledge_chunk (content and vectors), and knowledge_base (statistics)
  • Meilisearch (MEILISEARCH_HOST, MEILISEARCH_API_KEY) — the index backing full-text and hybrid search
  • Embedding API (EMBEDDING_API_URL) — converts text into vectors
  • External HTTP — downloads document files from their URL, either public or presigned S3
  • Consumer of the output: AI Message Intent Classification in ai_knowledge mode, which retrieves chunks for the LLM to answer from
  • This job makes no LINE API calls