What is robustness in ML safety, and how can one measure robustness to adversarial prompts?

• A. Robustness means consistent performance under perturbations; measure with adversarial prompts, paraphrasing, and red-team testing to assess resilience to manipulation.
• B. Robustness is the same as peak accuracy on the training set.
• C. Robustness refers to model size.
• D. Robustness is not relevant to safety.

Answer: (A)

Robustness in ML safety means the model behaves reliably even when inputs are perturbed or crafted to tempt it into unsafe or incorrect responses. It’s about staying stable under changes like wording shifts, tricky prompts, or attempts to manipulate the system. To measure this, you test with adversarial prompts designed to provoke unsafe outputs, use paraphrasing to check that the same task yields safe results across different phrasings, and conduct red-team testing where experts try to bypass safeguards. These methods reveal how well the model resists manipulation and maintains safe, consistent behavior, which is exactly what robustness aims to capture.

The other ideas don’t fit as well. Peak training accuracy reflects how well the model fits the training data, not how it holds up under perturbations or malicious prompts. Model size is about capacity, not resilience to adversarial inputs. And saying robustness isn’t relevant to safety ignores the fact that a robust model is less likely to produce unsafe or unreliable outputs when faced with challenging or adversarial conditions.