How can we evaluate model safety under distribution shift, and what metrics help?

• A. Out-of-distribution and stress tests; metrics include robustness scores, worst-case risk, and calibrated confidence across shifts.
• B. Only in-distribution accuracy is needed.
• C. Deterministic checks with no variability.
• D. Evaluations rely solely on user feedback.

Answer: (A)

Evaluating model safety under distribution shift hinges on understanding how a model behaves when inputs come from a different distribution than the one it was trained on. The best approach mixes out-of-distribution tests with stress tests to reveal vulnerabilities across a range of shifts, rather than relying on a single scenario.

Out-of-distribution tests expose the model to data from new domains or altered input statistics, showing how performance, decisions, or failures translate outside the training distribution. Stress tests push inputs to challenging edge cases, perturbations, or adversarial-like conditions to probe robustness under pressure. The metrics illuminate different safety dimensions: robustness scores summarize how performance degrades as shifts intensify; worst-case risk focuses on the largest potential harm across a defined set of shift scenarios; calibrated confidence across shifts gauges whether the model’s probability estimates stay well-calibrated and trustworthy even when data drift occurs.

In practice, you’d build a diverse suite of shifted data and perturbations, run the model across them, and compute these metrics to characterize safety margins beyond the original distribution. This approach helps ensure that safety guarantees hold in real-world, uncertain environments where distribution shifts are common.

Relying solely on in-distribution accuracy misses how the model might behave under new or perturbed inputs. Deterministic checks with no variability fail to reveal vulnerabilities that only appear under drift or diverse edge cases. Evaluations based only on user feedback lack systematic, objective measurement and can miss widespread safety issues.