RARE: Reinforcement Learning
with Adaptive Rubric Evolution
for Open-Ended Generation
Verifiable rewards have advanced reinforcement learning in mathematics and code. Open-ended generation, however, has no automatic verifier of overall quality: a response must satisfy multiple requirements, and many different responses can be valid. Rubric-based reinforcement learning makes these requirements explicit, but learning progresses unevenly. Some criteria become fully satisfied and lose their ability to distinguish outputs, while others remain informative. We introduce Reinforcement Learning with Adaptive Rubric Evolution (RARE), which uses current-policy feedback to guide which criteria change and what their replacements assess. At the sample level, RARE replaces criteria that are fully satisfied across a query’s current outputs with checks of remaining quality differences. It retains all other criteria and scores, preserving feedback that still distinguishes outputs. Within this update rule, we compare rewarding desired qualities with penalizing observed flaws. We study video script generation, where a model must produce coherent, executable narratives under user constraints. Across 2D animation, 3D / stop-motion, and live action, targeting observed flaws performs best, with a mean gain of 3.82 points over matched static rubrics. Model and human preferences support these gains, and separate experiments show improvements on writing and general generation benchmarks. We release the training code and 5,989 query–rubric pairs.
For each query, RARE samples eight outputs and scores them against the current rubric. A criterion is saturated when all eight valid judgments assign its maximum score. RARE replaces up to three saturated criteria, rescores the same outputs under the replacements, and uses the updated rewards in the current policy update.
Targets observed flaws in the current outputs.
Describes desired qualities the outputs should satisfy.
Combines flaw-focused and desired-quality criteria.
These three sample-level variants use the same replacement rule. Higher reward scores always indicate better quality.
We evaluate on 596 held-out queries using 12 fixed metrics covering query fulfillment, narrative coherence, cinematic executability, and dramatic effectiveness. Scores are normalized to 0–100; RL results average four training seeds. RARE with Negative replacements improves all three tasks and achieves the largest mean gain over its matched static baseline.
GPT-5.4 and Claude Sonnet 5 blindly compare RARE (Negative) with Sample-Static on all 596 test queries, in both script orders. Both judges favor RARE in every video type. More than 80% of preferences agree after reversing script order.
Five computer science PhD students blindly judge 60 script pairs, with three independent ratings per pair. Majority votes favor RARE in 38 pairs and Sample-Static in 13, leaving nine ties or unresolved cases.
70.8% preference score when ties and unresolved cases count as half a win (95% bootstrap CI: 60.0–80.8%).
Separate policies trained for writing and general generation also benefit from rubric evolution.
| Model | BiGGen 1–5 |
WritingBench 1–10 |
|---|---|---|
| Base | 4.035 | 8.22 |
| Static | 4.152 | 8.26 |
| RARE (Negative) | 4.259 | 8.34 |
Nine selected prompts, with outputs from the initial model and RARE.
English translations of the Chinese outputs. Scores evaluate the original scripts.
Loading examples…
| Criterion | Base | RARE |
|---|
Qwen3-4B-Instruct-2507 · initial model
Sample-level · Negative replacement
These examples compare the initial model with RARE. They are selected individual outputs; aggregate comparisons with static-rubric training appear in the Results section.
% BibTeX entry coming soon.