In preference learning, what form of human judgments is commonly used?

• A. Free-form text annotations
• B. Single scalar reward estimates
• C. Pairwise comparisons or rankings
• D. Cluster labels

Answer: (C)

In preference learning, human judgments are most naturally expressed as pairwise comparisons or rankings because people are typically clearer about which option they prefer over another than about giving an absolute score. This relative signal lets a model learn a scoring function that preserves order: if one option is preferred over another, the model should assign a higher score to it. With pairwise data, we can use straightforward loss functions that compare two items at a time (for example, encouraging s(a) > s(b) when a is preferred to b) and apply probabilistic models like Bradley-Terry or logistic-style losses. This approach is data-efficient, robust to inconsistent absolute judgments, and scales well when many items are involved, since preferences can be aggregated into a coherent ranking.

Free-form text annotations are unstructured and require extra processing to extract a usable signal. Single scalar reward estimates demand humans to provide an absolute scale, which is harder to do consistently and often noisy. Cluster labels come from unsupervised grouping and don’t convey relative desirability between options, which preference learning relies on. Pairwise judgments and rankings align best with how people naturally express preferences and provide a clear, actionable signal for learning ranking or scoring functions.