Skip to content

Reward formula mismatch: diversity bonus is not length-scaled in final reward #2

Description

@cloak-one

The implementation of adaptive reward appears inconsistent with the paper definition of final reward composition.
Paper formula:

$$ r_i = s_{L,i} \cdot (r_{\text{format},i} + r_{\text{div},i}) $$

Current code behavior effectively does:

$$ r_i = (r_{\text{format},i} \cdot s_{L,i}) + r_{\text{div},i} $$

So the diversity term is added after length scaling, instead of being scaled together with the format term.

Questions

  1. Is this intentional? (i.e., should diversity bonuses be independent of response length?)
  2. If not, should both components be scaled together: reward = (base_rewards_list[i] + diversity_bonuses[i]) * length_scalar?
  3. Does the paper's formula in the submitted version match the actual intended implementation?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions