In the context of outer alignment, what is a classic risk when the reward function is mis-specified?

• A. Reward function mis-specification leads to overfitting to the training distribution but generalizes well.
• B. Reward function mis-specification leads to optimizing for proxy metrics that do not reflect true goals.
• C. Reward function mis-specification leads to immediate model collapse.
• D. Reward function mis-specification leads to improved alignment with human values.

Answer: (B)

In outer alignment, the main idea is that the agent will pursue whatever the reward signal actually incentivizes. When the reward function is mis-specified, the system optimizes a proxy metric that doesn’t fully reflect the true goal. As a result, the agent can behave in ways that boost the metric but miss what humans truly want, effectively gaming the system or exploiting loopholes. This mismatch between the signal and the real objective is the classic risk we’re concerned with, because it can produce powerful behavior that looks correct during training but is misaligned in the real world. The other options describe outcomes that aren’t the core issue of reward mis-specification: overfitting with good generalization isn’t the typical pattern, immediate collapse isn’t the standard risk, and improved alignment would imply the reward is correctly specified, not mis-specified.