DRIFT: Learning from Abundant User Dissatisfaction in Real-World Preference Learning

This paper introduces DRIFT (Dissatisfaction-Refined Iterative preFerence Training), a novel approach for preference learning in real-world large language model deployments. It leverages abundant implicit user dissatisfaction signals, which are more common than explicit satisfaction feedback, to improve model alignment.

Standard Reinforcement Learning from Human Feedback (RLHF) relies on explicitly labeled preference data (e.g., 'Response A is better than B'), which is static, expensive, and hard to scale. However, real-world deployments generate massive amounts of unlabelled interaction data where users implicitly signal dissatisfaction by refining their prompts. Current methods fail to effectively utilize this 'trial-and-error' data because of the noise and the lack of ground-truth labels.

The authors propose a probabilistic framework that interprets a sequence of (prompt, response, refinement) as a signal of relative preference. They define a latent reward model that estimates the probability of user dissatisfaction. The method involves: 1) Collecting traces where users refine prompts. 2) Generating a distribution of 'hindsight' responses (what the model *should* have said) using the refined prompts. 3) Training the model using an off-policy reinforcement learning objective that maximizes the likelihood of the better (hindsight) response while minimizing the likelihood of the original unsatisfactory response, adjusting for the distributional shift.

Experimental results demonstrate that DRIFT significantly outperforms baselines (such as DPO trained on heuristically inferred pairs) across multiple benchmarks. The method successfully recovers valid preference signals from noisy refinement traces. In controlled environments, DRIFT agents achieved higher win-rates against standard RLHF models, proving that implicit dissatisfaction signals in conversation histories are a rich, untapped resource for model alignment.

The paper addresses a significant bottleneck in the LLM lifecycle: the high cost and scarcity of high-quality preference data. By formalizing the intuition that user refinements imply dissatisfaction, DRIFT offers a theoretically grounded method to utilize vast amounts of existing log data. The distributional approach to modeling 'hindsight' responses is a clever solution to the off-policy mismatch problem. However, the method relies on the assumption that refinements are primarily driven by dissatisfaction rather than exploration or user error, which may not always hold. Additionally, the computational cost of generating and sampling hindsight distributions during training could be higher than standard DPO.