Compare RLHF (reinforcement learning from human feedback) and RLAIF (reinforcement learning from AI feedback), including potential safety implications.

• A. RLHF uses human feedback to shape policy; RLAIF uses AI-generated feedback; RLHF can ensure alignment with human preferences but is limited by feedback quality and scalability; RLAIF can scale but risks biased or untrusted signals and can cause feedback loops or model exploitation. Calibration and oversight are needed.
• B. RLHF uses AI feedback to shape policy; RLAIF uses human feedback; AI feedback is always more reliable.
• C. RLHF uses human feedback to shape policy; RLAIF uses AI-generated feedback; RLHF always scales better than RLAIF.
• D. RLAIF uses human feedback; RLHF uses AI feedback; both are equally scalable.

Answer: (A)

Relying on human feedback (RLHF) means the learning signal comes from people rating or ranking model outputs, guiding the policy toward what humans judge as desirable or safe. This alignment with human values tends to produce behavior that matches nuanced preferences and safety concerns that are hard for a machine to infer on its own. But collecting high-quality human feedback is costly and slow, which makes scaling difficult. That limitation often leaves gaps in coverage and can introduce inconsistencies if different raters have varying standards or if feedback quality declines under heavy workloads.

Relying on AI feedback (RLAIF) replaces human judges with AI evaluators to generate the training signal. The big advantage is scalability: you can produce vast amounts of feedback quickly and continuously. The downside is that the feedback signal inherits the biases, blind spots, and potential manipulations of the AI evaluator. If the AI signals are biased or untrusted, the model can learn to optimize for those signals rather than for true human values, and feedback loops can develop where the model repeatedly reinforces its own proxy objectives. Calibration, diverse evaluation signals, and careful oversight are needed to mitigate these risks, often combining AI feedback with human review for critical judgments.

Because of these trade-offs, the statement that RLHF best ensures alignment with human preferences but is limited by feedback quality and scalability, while RLAIF can scale but brings biased signals and potential feedback loops (requiring calibration and oversight) is the strongest and most accurate overall. The other options misstate who provides the feedback, overclaim reliability or scalability, or reverse the roles of RLHF and RLAIF.