Self-evolving reasoning models learn from their own generated questions, yet repeated self-training can lead to performance collapse. R-Quest uses question validity and novelty feedback to sustain self-evolution.
vs. R-Zero at round ten
Sustained self-evolution
Math, general reasoning & code
Qwen and Llama
01 / THE PROBLEM
Self-evolution depends on the questions used to train the solver. Our preliminary study identifies two recurring problems.
1 / INVALID QUESTIONS
The invalid-question share rises from 23.5% to 60.5% between rounds one and five. Answer-consistency filtering makes the training pool worse: 72.36% of retained questions are invalid in round five.
Answer agreement alone does not ensure that a question is well posed.
In a controlled experiment, we minimally edit invalid training questions to make them valid. We preserve their domain, minimize changes in difficulty, and keep the other training conditions matched. Comparing the original and repaired data reveals the effect of question validity.
Minimal repairs reverse the late-stage decline under otherwise matched training conditions.
2 / REPEATED TASKS
R-Zero discourages repetition by clustering questions with BLEU and assigning a penalty based on each cluster’s size. However, changes in wording, notation, or numerical parameters can make the same mathematical task look lexically different.
We group R-Zero’s filtered fifth-round questions by their underlying mathematical construction and requested task, rather than by wording.
The largest type alone accounts for 31.5% of the sample, but BLEU treats it as 22 separate clusters. Because the repetition penalty uses individual cluster size, this fragmentation weakens the penalty on a task type that is actually very frequent.
Repetition should be assessed by the underlying mathematical task.
02 / THE METHOD
R-Quest changes what counts as a useful training question. It equips the solver to reject invalid problems and assesses novelty through the underlying mathematical task, addressing the failure modes of answer-only training and lexical similarity.
01 / VALIDITY FEEDBACK
An answer-only solver produces inconsistent answers to a flawed question. High uncertainty can be mistaken for difficulty, allowing invalid questions to accumulate in training.
Validity-aware initialization teaches the solver to identify the flaw and return INVALID. Invalid questions are rejected instead of becoming training supervision.
02 / NOVELTY FEEDBACK
BLEU separates reworded versions of the same mathematical task into different lexical clusters. Fragmented clusters weaken the repetition penalty on a frequently recurring exercise.
We decompose novelty supervision into atomic pairwise judgments that the frozen base model can perform, without introducing external models or tools. Comparing the mathematical setup and requested task lets it recognize equivalent exercises despite changes in wording, notation, or numerical values.
With GRPO, we equip the solver to distinguish a flawed question from a difficult but valid one, producing an answer for the latter and INVALID for the former. A penalty on rejecting valid questions discourages excessive refusal.
A frozen base model compares each candidate with sampled references, identifying repetition from the mathematical setup and requested task. A single same-type match rejects the candidate, preventing changes in wording or values from masquerading as new tasks.
Validity and novelty feedback shape questioner rewards as the questioner and solver alternate GRPO updates. The solver learns from validity- and answer-consistency-filtered questions, with a small replay of initialization data to preserve its ability to reject invalid ones.
SAMPLED PAIRWISE JUDGMENTS
For each generated question, the frozen base model checks whether it repeats the underlying task of any of K randomly sampled questions. A single match triggers rejection.
The more often a task occurs in the generated pool, the more likely it is to be rejected. With the default K = 8, a task appearing in 20% of questions is rejected about 83% of the time.
Hover or tap to compare all three curves.
03 / THE RESULTS
R-Quest sustains its gains over ten rounds of self-evolution, reaching its best mathematical performance in the final round on Qwen3-4B-Base.
R-Zero peaks at 49.10 in round three, then declines to 34.60 by round ten. R-Quest retains its gains despite minor fluctuations between rounds.
+5.75 points above the base model after ten rounds, compared with R-Zero’s decline of 11.57 points.
Domain averages and individual benchmark scores (%).
Scroll horizontally to view all benchmark columns.
| Method | Average | AMC | Minerva | MATH-500 | GSM8K | OlympiadBench | AIME 2025 | AIME 2024 |
|---|---|---|---|---|---|---|---|---|
| Base model | 46.17 | 47.34 | 51.84 | 74.20 | 88.63 | 39.11 | 11.35 | 10.73 |
| Base + validity init. | 47.07 | 49.14 | 53.68 | 74.40 | 89.76 | 41.04 | 10.73 | 10.73 |
| R-Zero | 49.10 | 54.84 | 54.78 | 76.20 | 92.42 | 42.37 | 12.29 | 10.83 |
| OCNR | 49.57 | 55.94 | 58.09 | 77.60 | 92.12 | 43.11 | 7.92 | 12.19 |
| R-Diverse | 49.82 | 55.31 | 57.35 | 77.20 | 92.19 | 43.70 | 9.48 | 13.54 |
| R-Quest Ours | 51.64 | 57.89 | 58.82 | 77.60 | 92.57 | 46.67 | 13.44 | 14.48 |
Explore the paper’s analysis figures.
Either validity or novelty feedback alone achieves a higher peak score than R-Zero. However, validity feedback is more critical for sustained gains: its removal leads to late-stage collapse, whereas the variant without novelty feedback shows a milder decline and remains above the base model.
04 / CITE THIS WORK
Questioning the Questions: Sustaining Self-Evolution in Reasoning Models
Preprint · 2026
arXiv:2610.04299
@misc{li2026rquest,
title={Questioning the Questions: Sustaining
Self-Evolution in Reasoning Models},
author={Jinyuan Li and Chengsong Huang and
Langlin Huang and Donghong Cai and
Shiping Gao and Yuyi Yang and
Jiaxin Huang},
year={2026},
eprint={2610.04299},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2610.04299}
}