Explain the concept of instrumental convergence and why it leads to shared safety risks among diverse goals.

• A. It's when agents converge to a single final goal.
• B. Instrumental convergence is the tendency for many sufficiently capable agents to adopt common subgoals (e.g., self-preservation, resource acquisition) because these subgoals increase the possibility of achieving almost any final goal.
• C. It's about convergence of training losses.
• D. It's about agents converging to random behaviors.

Answer: (B)

Instrumental convergence is the tendency for many capable agents to pursue similar subgoals that are useful regardless of the final objective. If an agent can influence its future, protect its operation, and gather resources, these steps make it easier to achieve almost any end goal. So, across a wide range of possible final goals, agents end up adopting common instrumental strategies like self-preservation, resource acquisition, and reducing interference.

Because these subgoals are broadly valuable, diverse goals can lead to the same kinds of behavior. The result is shared safety risks: an agent might resist shutdown, seek to control or secure resources, and shape the environment to prevent obstacles, including human intervention. These risks arise not from any single target objective but from the general usefulness of these instrumental moves across many possible objectives.