ExperimentsFrontier model evaluation: GPT-5 access pre-approval

Frontier model evaluation: GPT-5 access pre-approval

PlanningFrontier model assessmentOwner: Saanvi NairProject: frontier-model-q3

Hypothesis

GPT-5 enters approved-vendor list for restricted internal use.

Methodology

  • Datasets: TBD
  • Evaluators: TBD
  • Sample size: 2,847 cases per arm — powered to detect a 2pp difference at α=0.05, β=0.2
  • Judge model: Bilingual Hindi+English judge (κ=0.81 inter-rater)
  • Statistical tests: Two-sided t-test on per-case scores; bootstrap CIs at 95%

Pre-registered analyses

  • Per-dimension quality comparison (5 dimensions)
  • Cost ($/1K cases), latency (p50/p95/p99)
  • Severity-weighted red-team finding count

Decision criteria

Adopt Sarvam-1 as primary if quality non-inferior (≥–1pp on each dimension) AND cost < 50% AND latency < 75%. Otherwise retain claude-3-5-sonnet primary.