In experimental design, how do randomization and stratified sampling differ, and why do they matter for AI safety experiments?

• A. Randomization reduces confounding; stratified sampling ensures representation across subgroups. Both improve validity and generalizability.
• B. Randomization ensures representation across subgroups; stratified sampling reduces confounding; both improve speed.
• C. Randomization reduces sampling bias; stratified sampling increases power; both degrade generalizability.
• D. Randomization guarantees causal conclusions; stratified sampling guarantees no bias; both remove uncertainty.

Answer: (A)

Randomization and stratification tackle different problems in experiments. Randomization assigns the intervention by chance, which helps balance both known and unknown factors that could influence the outcome. This reduces confounding variables so observed effects are more likely due to the intervention itself, supporting clearer causal interpretation. Stratified sampling, by contrast, groups the population into important subgroups and then samples or assigns within each group. This guarantees that diverse subgroups are represented in the study, which improves representativeness and the precision of results within each subgroup, boosting generalizability to the broader population.

In AI safety experiments, you want both effects: randomization helps ensure that differences you see aren’t caused by other factors like different model versions or deployment environments, so you can more confidently attribute changes to the safety intervention. Stratification ensures you don’t neglect key subgroups—different model architectures, data domains, or task types—so your findings apply across the varied settings where the AI system operates.

The best description is that randomization reduces confounding, stratified sampling ensures representation across subgroups, and together they improve validity and generalizability. Other statements mix up these roles or make overstated claims—randomization doesn’t guarantee causal conclusions, stratified sampling doesn’t remove all bias, and neither approach inherently makes experiments faster or removes all uncertainty.