What are the most important considerations when building a production-ready RAG application?

[aliahsan9]

🏷️ Discussion Type

Question

Body

I'm currently building AI-powered business applications using ASP.NET Core, Angular, SQL Server, and RAG. While creating a BusinessOS project, I realized that retrieval quality, chunking strategy, embeddings, and prompt design can significantly affect results.

For developers who have deployed RAG systems in production:

What challenges did you encounter?
How do you evaluate retrieval quality?
Which vector databases and embedding models do you recommend?
What lessons would you share with someone building their first production RAG application?

I'd love to learn from the community's experiences.

Guidelines

• I have read and understood this category's guidelines before making this post.

[roohan-514]

Deployed a few RAG systems in production. Here's what I learned the hard way:
Chunking strategy matters more than you think

• Fixed-size chunking with overlap is a good start but document-aware chunking (by sections/paragraphs) works much better in practice
• Test different chunk sizes — for Q&A, 500-1000 tokens with 100-150 overlap is usually sweet spot
• Metadata preservation is critical — always store source doc, page/section number, chunk index so you can cite back
  Embedding models
• OpenAI text-embedding-3-small — best bang for buck, 1536-dim, cheap
• Cohere embed-v3 — excellent for retrieval, supports multimodal
• BGE-large-en-v1.5 — best open-source if you want local/hybrid
• E5-mistral-7b-instruct — overkill unless you have specific accuracy requirements
• Don't overthink this — start with text-embedding-3-small, swap later if needed
  Vector databases — pick based on your scale
• Up to 1M vectors: pgvector on existing PostgreSQL — no infra overhead, good enough
• 1M-10M: Qdrant or Weaviate (both excellent, easy to self-host)
• 10M+: Pinecone (expensive but zero ops) or Milvus (complex but powerful)
• Don't start with a dedicated vector DB — use pgvector first, migrate later when you actually need it
  Biggest production challenges

1. Hallucination from retrieved context — Users blame AI when it ignores a relevant doc and makes stuff up. Mitigation: reranking stage + prompt that says "if context doesn't answer, say so"
2. Latency — naive flow is embedding query -> vector search -> LLM call. That's 3 API calls. Cache frequent queries, use streaming, consider hybrid search to reduce rerank load
3. Chunk ambiguity — "it" in one chunk refers to something in previous chunk you didn't retrieve. Mitigation: add context window (retrieve N chunks around the match), or use LLM to rewrite queries with context
4. Evaluation is the real problem — You'll have no idea if improvements actually help without good eval
   How to evaluate retrieval quality

• Build a test set of 50-100 Q&A pairs where you know the right document(s)
• Measure hit rate (was the right doc in top-k) and MRR (how high was it)
• For end-to-end: correctness (does the answer use the right context) and faithfulness (does it stay true to retrieved docs)
• Tools: Ragas (open-source eval framework), LangSmith for tracing
  Quick lessons
• Start simple — naive RAG with proper chunking beats over-engineered RAG 80% of the time
• Reranking (Cross-Encoder) adds 10-15% recall but doubles latency — use it only when needed
• Track every query + retrieved chunks + response in a DB — invaluable for debugging
• Hybrid search (vector + keyword) beats pure vector search for most enterprise docs
• Your eval set is your most important asset — invest in it early

[aliahsan9]

@roohan-514 Really appreciate the detailed response. The recommendation to start simple with pgvector and focus on evaluation makes a lot of sense.