Building an Enterprise Knowledge Base with Vector Databases and LLMs

Large Language Models (LLMs) are remarkable at understanding and generating text, but they have blind spots:

• They can’t reliably answer questions about private, internal, or real-time data.
• Their training data may be outdated or lack your specific domain knowledge.

To overcome this, we use Retrieval-Augmented Generation (RAG) — a method that connects an LLM to a vector database containing your knowledge base. This lets you “talk” to your data in plain language, with results grounded in real documents.

1. Understanding RAG: The Big Picture

RAG solves three key LLM limitations:

1. Freshness – Using the most recent internal data.
2. Relevance – Focusing on domain-specific knowledge.
3. Accuracy – Reducing hallucinations by grounding answers in retrieved content.

Example: A pharmaceutical company could ask:

“What safety concerns were raised in the last three clinical trial reports for Drug A?”

Instead of relying on memory from training, the RAG pipeline:

1. Retrieves the relevant parts of those reports from a vector database.
2. Passes them to the LLM.
3. Generates an answer citing the actual findings.

Diagram 1: RAG Workflow

(User → LLM → Vector DB → Knowledge Base → Relevant Chunks → LLM → Answer)

2. Why Vector Databases Matter

Unlike keyword search, vector databases store embeddings — numerical representations of meaning. This enables semantic search that understands intent, not just exact words.

Example: The query “cost of adopting a cat” could mean:

• A pet adoption scenario, or
• Caterpillar Inc. (stock ticker CAT).

Without context, keyword search might mix the two. Vector search, combined with contextual embeddings, distinguishes between them.

Popular Vector Stores:

• Milvus – Scalable, distributed search.
• PgVector – PostgreSQL extension.
• Pinecone, Weaviate, Qdrant – Cloud-native options with various feature sets.

2.1. Challenges in Vector Database Setup

From the whitepaper’s details:

• Partitioning Trade-offs
• Embedding Model Choice
• Distance Metrics

Diagram 2: How Vector Search Works

1. Embed – Convert documents into vectors.
2. Index – Store vectors for fast retrieval.
3. Retrieve – Use similarity search to find the closest matches.

3. Indexing for Speed and Scale

Two common indexing methods from the whitepaper:

1. IVFFlat (Inverted File) – Fast to build, smaller storage, but slower queries.
2. HNSW (Hierarchically Navigable Small Worlds) – Builds a layered graph of vector connections.

4. The Generation Stage: Choosing the Right LLM

Bigger models are not always better. WizardLM-13B and Mistral-7B often outperform larger models for certain RAG use cases, especially when cost and latency matter.

Whitepaper insights:

• Different models with the same parameter count can behave very differently.
• Fine-tuning on your documents often yields more gains than jumping to a larger base model.

5. Prompt Engineering in Practice

Even with the right LLM, prompts matter.

• Add explicit grounding instructions: “Answer only based on the provided document.”
• Use chain-of-thought prompting: “Let’s think step by step.”
• Include input-output examples for better pattern adherence.

Example from the whitepaper: Formatting prompts in Markdown or JSON helps separate user queries from retrieved content, reducing hallucinations.

6. Guardrail Engineering

Prompt engineering can’t eliminate hallucinations entirely, so we use guardrails:

• Stop Words – Halt generation when encountering patterns like "User:" to prevent fake dialogue.
• Categorical Validation – Ensure the answer is one of the allowed values for constrained questions.

7. Scaling to Production

Scaling RAG isn’t just about retrieval quality — it’s about cost, latency, and uptime.

Challenges & Solutions from the whitepaper:

• GPU Costs – Usage spikes demand elastic scaling.
• Data Transfer Overhead – Minimize CPU↔GPU transfer time.
• Load Variability – Handle peak loads without overpaying during off-hours.

Key Best Practices for Building RAG Systems

• Embeddings – Start with small, fast models like sentence-BERT and optimize later. Avoid picking the largest model without testing.
• Indexing – Use HNSW for large-scale search and IVFFlat for smaller workloads. Don’t rely on brute-force search.
• Retrieval – Match the embedding model to your specific domain data. Avoid using generic embeddings for all data types.
• Generation – Choose cost-effective LLMs and fine-tune if necessary. Bigger is not always better.
• Prompt Engineering – Add clear grounding instructions and format inputs consistently. Avoid vague or unstructured prompts.
• Guardrails – Implement stop words and validation checks to reduce hallucinations. Don’t depend only on prompts for accuracy.
• Scaling – Use elastic GPU scaling and optimize data transfers. Avoid overprovisioning during low-usage periods.