Valid references.
An unsupported call time.

Define the expected answer first

Three fictional cases cover a clear discharge plan, conflicting records, and missing information. Records, expected facts, discrepancies, gaps, and prohibited inferences were frozen before generation. Answer keys stayed outside the model input.

Two repeats per case and prompt version produced 18 outputs. All remain in the evaluation, including four validation failures. Substantive items were split into claims; pure questions without factual premises were excluded. Coverage, reference integrity, and claim support were assessed separately.

Three iterations

• V1: gpt-4.1-mini-2025-04-14, temperature 0.2; model-written quotations and document scopes.
• V2: the same model, revised instructions, and exact source passages resolved by code.
• V3: gpt-5.4-mini-2026-03-17, medium reasoning; stronger attribution instructions, simpler schema, and scopes supplied by code.

“Usable” means structurally accepted. V1 failed twice on nonexact quotations; V2 failed on incomplete gap scope and a one-source conflict. V3 had no mechanical failures.

Support ratings use attached evidence, including declared document scopes for bounded absence claims. Source-wide flags identify claims unsupported anywhere in the records; they overlap the support ratings. Repeated claims count again.

No controlled winner: model, prompt, schema, and evidence rendering changed together. Reviewer judgment and claim granularity also affect comparisons.

Final coverage

Discrepancies apply only to Theo; other cases have none expected. Two final briefs exceeded the 350-word target. Lena run 1 reached 436 words and 15 items while missing the address, arrival-at-home, and current-situation gap.

A timestamp became a call time

“The discharge summary lists an October 8 discharge, and the outreach log records an unanswered outreach attempt on October 9 at 9:15 AM.”

Theo’s first V3 response adds 9:15 AM. Cited passage O3 gives only a date; the time belongs to the log timestamp. Valid references and exact quotations did not support that inference. The recorded walkthrough retains the error for correction.

Lena run 2 also called a checkbox’s state undocumented although the source says it is blank. That ambiguous wording received partial support. Earlier V2 outputs overstated missing documentation and labeled unconfirmed plans as conflicts.

Limits and next steps

This is AI-assisted assessment, with no human or clinical validation. The results establish neither staff time savings nor production readiness. All 18 passed the assessed care-action boundary check.

Separate workflow checks exercised evidence opening, editing, approval invalidation, and saving. One additional live generation supported UI verification and is outside the 18 scored outputs.

Next: interview case managers, observe preparation, assess comprehension and editing effort, and add independently authored holdouts with human review. Keep the timestamp failure as a regression case while reducing repetition without losing useful uncertainty.

Inspect the evidence

• Source cases · Answer keys · Rubric · Freeze manifest
• V1 claim review · V1 run index
• V2 claim review · V2 run index
• V3 claim review · V3 run index