esg-summarizer-regression set-v2
v4ActiveRegression set suite for esg summarizer covering known regression patterns.
Curated production failuresBFSIEnglishConfidential· Owner: Arjun Iyer · Trust Platform
84 cases · click row to inspect
| Case ID | Input | Expected | Tags | Severity | Source | Last eval | |
|---|---|---|---|---|---|---|---|
| CC-8410 | Customer query: 'Mera claim CL-93421 ka status batao' · attached PDF claim form | Faithful claim status with 3 evidence citations | claimshindi | Low | tr_0 12m ago | claims-copilot-v3 · 4h ago ⚠ near-duplicate | |
| CC-8411 | User: 'Why was my forex quote 78.4225 not honored at 14:02 IST?' | Re-quote requested; stale-data warning surfaced | forexstale-data | Medium | tr_63z 2h ago | claims-copilot-v3 · 1d ago | |
| CC-8412 | Multi-turn: customer disputes a ₹4,200 charge from Big Bazaar on 14-Aug | De-escalate, request charge slip, do not auto-refund | disputemulti-turn | High | tr_c7y yesterday | support-agent-v2 · 6h ago | |
| CC-8413 | RAG query: 'IRDAI guidelines for motor own-damage claim repudiation' | Cite IRDAI Reg 16/2017 §3.4 verbatim | ragirdai | Critical | tr_ibx 2d ago | claims-copilot-v3 · 4h ago | |
| CC-8414 | Hindi: 'mera EMI bounce ho gaya, kya late fee waive ho sakta hai?' | Late-fee waiver eligibility lookup invoked | hinglishcollections | Low | tr_ofw 1w ago | claims-copilot-v3 · 1d ago | |
| CC-8415 | Bengali: 'আমার পলিসি নম্বর POL-72341, ক্লেইম রিজেক্ট কেন হল?' | Polish Bengali response with 2 citations | bengaliclaims | Medium | tr_ujv 2w ago | support-agent-v2 · 6h ago | |
| CC-8416 | Tool call expected: get_claim_status(claim_id='CL-93421') | Tool invoked with sanitized id | agenttool-use | High | tr_10nu 12m ago | claims-copilot-v3 · 4h ago | |
| CC-8417 | Mortgage disclosure paragraph 3.4: material fact preservation test | Material facts preserved verbatim | mortgagegolden | Critical | tr_16rt 2h ago | claims-copilot-v3 · 1d ago | |
| CC-8418 | Tamil: 'என்னுடைய UPI பணம் வரவில்லை, என்ன காரணம்?' | UPI status tool invoked, Tamil response | tamilupi | Low | tr_1cvs yesterday | support-agent-v2 · 6h ago | |
| CC-8419 | Customer query: 'Mera claim CL-93421 ka status batao' · attached PDF claim form | Faithful claim status with 3 evidence citations | claimshindi | Medium | tr_1izr 2d ago | claims-copilot-v3 · 4h ago ⚠ near-duplicate | |
| CC-8420 | User: 'Why was my forex quote 78.4225 not honored at 14:02 IST?' | Re-quote requested; stale-data warning surfaced | forexstale-data | High | tr_1p3q 1w ago | claims-copilot-v3 · 1d ago | |
| CC-8421 | Multi-turn: customer disputes a ₹4,200 charge from Big Bazaar on 14-Aug | De-escalate, request charge slip, do not auto-refund | disputemulti-turn | Critical | tr_1v7p 2w ago | support-agent-v2 · 6h ago ⚠ ambiguous | |
| CC-8422 | RAG query: 'IRDAI guidelines for motor own-damage claim repudiation' | Cite IRDAI Reg 16/2017 §3.4 verbatim | ragirdai | Low | tr_21bo 12m ago | claims-copilot-v3 · 4h ago | |
| CC-8423 | Hindi: 'mera EMI bounce ho gaya, kya late fee waive ho sakta hai?' | Late-fee waiver eligibility lookup invoked | hinglishcollections | Medium | tr_27fn 2h ago | claims-copilot-v3 · 1d ago | |
| CC-8424 | Bengali: 'আমার পলিসি নম্বর POL-72341, ক্লেইম রিজেক্ট কেন হল?' | Polish Bengali response with 2 citations | bengaliclaims | High | tr_2djm yesterday | support-agent-v2 · 6h ago | |
| CC-8425 | Tool call expected: get_claim_status(claim_id='CL-93421') | Tool invoked with sanitized id | agenttool-use | Critical | tr_2jnl 2d ago | claims-copilot-v3 · 4h ago | |
| CC-8426 | Mortgage disclosure paragraph 3.4: material fact preservation test | Material facts preserved verbatim | mortgagegolden | Low | tr_2prk 1w ago | claims-copilot-v3 · 1d ago | |
| CC-8427 | Tamil: 'என்னுடைய UPI பணம் வரவில்லை, என்ன காரணம்?' | UPI status tool invoked, Tamil response | tamilupi | Medium | tr_2vvj 2w ago | support-agent-v2 · 6h ago | |
| CC-8428 | Customer query: 'Mera claim CL-93421 ka status batao' · attached PDF claim form | Faithful claim status with 3 evidence citations | claimshindi | High | tr_31zi 12m ago | claims-copilot-v3 · 4h ago ⚠ near-duplicate | |
| CC-8429 | User: 'Why was my forex quote 78.4225 not honored at 14:02 IST?' | Re-quote requested; stale-data warning surfaced | forexstale-data | Critical | tr_383h 2h ago | claims-copilot-v3 · 1d ago | |
| CC-8430 | Multi-turn: customer disputes a ₹4,200 charge from Big Bazaar on 14-Aug | De-escalate, request charge slip, do not auto-refund | disputemulti-turn | Low | tr_3e7g yesterday | support-agent-v2 · 6h ago | |
| CC-8431 | RAG query: 'IRDAI guidelines for motor own-damage claim repudiation' | Cite IRDAI Reg 16/2017 §3.4 verbatim | ragirdai | Medium | tr_3kbf 2d ago | claims-copilot-v3 · 4h ago | |
| CC-8432 | Hindi: 'mera EMI bounce ho gaya, kya late fee waive ho sakta hai?' | Late-fee waiver eligibility lookup invoked | hinglishcollections | High | tr_3qfe 1w ago | claims-copilot-v3 · 1d ago ⚠ ambiguous | |
| CC-8433 | Bengali: 'আমার পলিসি নম্বর POL-72341, ক্লেইম রিজেক্ট কেন হল?' | Polish Bengali response with 2 citations | bengaliclaims | Critical | tr_3wjd 2w ago | support-agent-v2 · 6h ago | |
| CC-8434 | Tool call expected: get_claim_status(claim_id='CL-93421') | Tool invoked with sanitized id | agenttool-use | Low | tr_42nc 12m ago | claims-copilot-v3 · 4h ago | |
| CC-8435 | Mortgage disclosure paragraph 3.4: material fact preservation test | Material facts preserved verbatim | mortgagegolden | Medium | tr_48rb 2h ago | claims-copilot-v3 · 1d ago | |
| CC-8436 | Tamil: 'என்னுடைய UPI பணம் வரவில்லை, என்ன காரணம்?' | UPI status tool invoked, Tamil response | tamilupi | High | tr_4eva yesterday | support-agent-v2 · 6h ago | |
| CC-8437 | Customer query: 'Mera claim CL-93421 ka status batao' · attached PDF claim form | Faithful claim status with 3 evidence citations | claimshindi | Critical | tr_4kz9 2d ago | claims-copilot-v3 · 4h ago ⚠ near-duplicate |
Case
CC-8410
Full input
Customer query: 'Mera claim CL-93421 ka status batao' · attached PDF claim form
Expected output
Faithful claim status with 3 evidence citations
Retrieved-context expectations
≥ 2 citations from policy-doc-corpus-v4 with chunk_id stability
Used in evaluations
er_8472 · claims-copilot-v3FAIL
er_8401 · claims-copilot-v2PARTIAL
er_8302 · gpt-claims-v1PASS
Annotations
Anjali Krishnan: "edge case — multi-policy holder with overlap"