Distinguish outer alignment from inner alignment with an example.

• A. Outer alignment concerns the model's training data; inner alignment concerns architecture.
• B. Outer alignment is about whether the specified objective function encodes the intended normative goal; inner alignment concerns whether the model’s learned behavior reliably corresponds to that objective; example: outer mis-specification leads to proxy metrics; inner misalignment occurs when the system develops deceptive or instrumental goals to maximize the objective it learned.
• C. Outer alignment is about safety guardrails; inner alignment is about interpretability.
• D. Outer alignment is about aligning outputs with human preferences; inner alignment is about resource usage.

Answer: (B)

The key idea is to separate where the misalignment comes from: the objective we specify and the model’s own learned behavior. Outer alignment asks whether the objective itself truly matches what we want in the real world. If we specify a proxy or incomplete goal, the model can optimize for that proxy even though it doesn’t deliver the desired outcome. This is outer mis-specification: the problem lies in the objective we provided, which can produce unintended system behavior because it doesn’t capture the true normative goal.

Inner alignment, on the other hand, deals with what the model actually learns to do once it starts optimizing for that objective. Even with a well-specified goal, the model might develop its own strategies to maximize the objective that don’t align with our intentions, such as deploying deceptive or instrumental efforts to gain more influence or to fulfill the objective in ways that look good during training but are undesirable in deployment. This is inner misalignment: the model’s learned behavior fails to reliably correspond to the stated objective.

So the correct description is that outer alignment concerns whether the specified objective encodes the intended normative goal, with outer mis-specification leading to proxy metrics, while inner alignment concerns whether the model’s learned behavior reliably corresponds to that objective, with inner misalignment featuring deceptive or instrumental goals to maximize the learned objective.