Add PAPOTrainer for preference-based optimization

[SolarWindRider]

What does this PR do?

This PR introduces a new trainer named PAPOTrainer, which extends GRPOTrainer to support the PAPO (Preference Alignment via Policy Optimization) algorithm.

Motivation

PAPO is a variant of GRPO that incorporates a contrastive preference optimization mechanism to improve stability when positive samples are sparse. But the official code use verl. To make it convenient for everyone to use, I implemented the TRL version of the code based on the PAPO formula, and it runs successfully.

Implementation Details

• Added trl/trainer/papo_trainer.py
• Added trl/trainer/papo_config.py
• Updated __init__.py to include PAPOTrainer
• All tests pass locally with pytest tests/trainer/test_papo_trainer.py -v

🧪 Example Usage

https://github.com/SolarWindRider/avr/blob/main/train_papo.py

I have tested my trainer[with PEFT and FSDP] on Ascend910C and H20[single node with 8 cards].

Checklist

• I have tested this code locally
• I have run all tests with pytest
• I have followed the code style guidelines
• I have added docstrings and comments

[qgallouedec]

Thanks for your contribution. Can you mode this new trainer to trl.experimental instead? Ideally, we would also have a small mention in the paper index section of the documentation

[SolarWindRider]

Thanks for your contribution. Can you mode this new trainer to trl.experimental instead? Ideally, we would also have a small mention in the paper index section of the documentation

Thank you for your advice. I have moved this new trainer to trl.experimental and also added PAPO info in paper index.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[qgallouedec]

LGTM, thanks!

[SolarWindRider]

LGTM, thanks!

Thank you for your help!
{\__/}
( • - •)
/つThanks