The implementation of adaptive reward appears inconsistent with the paper definition of final reward composition.
Paper formula:
$$
r_i = s_{L,i} \cdot (r_{\text{format},i} + r_{\text{div},i})
$$
Current code behavior effectively does:
$$
r_i = (r_{\text{format},i} \cdot s_{L,i}) + r_{\text{div},i}
$$
So the diversity term is added after length scaling, instead of being scaled together with the format term.
Questions
- Is this intentional? (i.e., should diversity bonuses be independent of response length?)
- If not, should both components be scaled together:
reward = (base_rewards_list[i] + diversity_bonuses[i]) * length_scalar?
- Does the paper's formula in the submitted version match the actual intended implementation?
The implementation of adaptive reward appears inconsistent with the paper definition of final reward composition.
Paper formula:
Current code behavior effectively does:
So the diversity term is added after length scaling, instead of being scaled together with the format term.
Questions
reward = (base_rewards_list[i] + diversity_bonuses[i]) * length_scalar?