ExperimentsRAG chunk size 512 vs 1024 vs 2048 on retrieval quality

RAG chunk size 512 vs 1024 vs 2048 on retrieval quality

Conclusion: 1024 wins on F1; 2048 wins on long-context QA. Recommend hybrid retrieval.

Complete · published Mar 28AblationOwner: Vikram ShettyProject: customer-support-rag-hindi

Hypothesis

1024 chunks optimize precision/recall tradeoff.

Methodology

  • Datasets: TBD
  • Evaluators: TBD
  • Sample size: 2,847 cases per arm — powered to detect a 2pp difference at α=0.05, β=0.2
  • Judge model: Bilingual Hindi+English judge (κ=0.81 inter-rater)
  • Statistical tests: Two-sided t-test on per-case scores; bootstrap CIs at 95%

Pre-registered analyses

  • Per-dimension quality comparison (5 dimensions)
  • Cost ($/1K cases), latency (p50/p95/p99)
  • Severity-weighted red-team finding count

Decision criteria

Adopt Sarvam-1 as primary if quality non-inferior (≥–1pp on each dimension) AND cost < 50% AND latency < 75%. Otherwise retain claude-3-5-sonnet primary.