Reinforcement Learning with Human Feedback

PPO

read more

• The N Implementation Details of RLHF with PPO

DPO

DPO starts with preference data pairs(prompt, accepted and rejected) and a SFT model without the requirement of reward model. One thing worth mentioning is the preference data should be in distribution, i.e., it should be generated by the SFT model. Otherwise, consider fine-tuning the SFT model with some of the preference data to alleviate distribution shift before the actual DPO training,

read more

• DPO (PPO代替)解读与实践【大模型论文系列】
• Is DPO Always the Better Choice for Preference Tuning LLMs?
• The N Implementation Details of RLHF with PPO