deepseek_v4_pro_l1_002_02_05_w3_proposal.md

Counterfactual Step Advantage: Dense Credit Assignment for Multi-Turn LLM Agent RL Without a Separate Critic

Introduction

Training large language model (LLM) agents to act effectively across multi-turn interactive environments has become a central challenge in AI research. The dominant paradigm uses reinforcement learning (RL) with group-relative policy optimization algorithms such as GRPO (Shao et al., 2024) and DAPO (Yu et al., 2025), which sample a group of rollout trajectories for each training task, normalize outcome rewards within the group into normalized advantages, and update the policy accordingly. These algorithms have proven successful for mathematical reasoning tasks where each rollout is a single-turn chain of thought with a binary verifiable outcome. However, when applied to multi-turn agent tasks—web navigation, tool use, task-oriented dialogue, embodied interaction—the signal they provide is fundamentally sparse: every action in a rollout shares the identical trajectory-level advantage, regardless of its individual contribution to the outcome.

This sparsity creates two well-documented problems. First, good actions in failed trajectories are penalized, and bad actions in successful trajectories are rewarded, diluting the learning signal (Choudhury et al., 2025; Yu et al., 2025). Second, in long-horizon tasks, the probability that a trajectory-level reward accurately reflects the quality of any specific intermediate action approaches zero, and agents trained under trajectory-level RL may fail to develop genuine reasoning strategies, instead resorting to shallow heuristics or hallucinated thoughts (Wang et al., 2025).

Existing solutions to this credit assignment problem fall into three categories. Process reward models (PRMs) train a separate critic to score individual steps, either explicitly via supervised learning on expert labels (Choudhury et al., 2025) or implicitly via preference optimization on trajectory pairs (Liu et al., 2025b). Step-level progress attribution methods train auxiliary models to estimate the contribution of each action toward the goal (Deng et al., 2025). Turn-wise advantage models exploit training-time privileged information such as reference solutions to evaluate action quality (Jain et al., 2025). All of these approaches share a common burden: they require training and maintaining a separate model—a critic, a PRM, or a progress estimator—alongside the policy, adding architectural complexity, data requirements, and domain-specific tuning overhead.

We propose an alternative that avoids training any separate model while still delivering step-level credit assignment. Our observation is that the group-sampling mechanism already present in GRPO and DAPO—sampling G rollouts for the same task from the same policy—naturally generates within-group counterfactuals at each step position. Different rollouts take different actions at corresponding step indices under different (but related) observation histories. We exploit this by decomposing the trajectory-level advantage into a sequence of step-level counterfactual advantages, computed directly from the per-step action-likelihood ratios across the group, without any auxiliary model.

Concretely, for each step t across G rollouts, we compute a step-specific advantage as the deviation of each rollout's action probability from the group's average action probability at that step position, weighted by the rollout's normalized outcome. This produces per-step advantages that, when aggregated across steps, recover the trajectory-level advantage while enabling the policy update to reinforce specific useful actions in otherwise-failed rollouts and penalize specific harmful actions in otherwise-successful rollouts. We call this Counterfactual Step Advantage (CSA). The method requires no additional model parameters, no privileged training-time information, and minimal computation beyond what GRPO already performs. It can be applied as a drop-in credit assignment module within any group-relative RL framework. We design experiments across five agent benchmarks spanning web navigation, embodied interaction, tool-augmented dialogue, and software engineering to test whether CSA improves sample efficiency and final performance over GRPO/DAPO baselines and whether it closes the gap with PRM-based approaches without their added complexity.

Related Work

We organize prior work into five families: group-relative policy optimization for LLMs, process reward and credit assignment methods, search-based and self-training approaches, multi-turn preference and curriculum methods, and synthetic data generation for agent training.

Group-relative policy optimization. GRPO (Shao et al., 2024), introduced in DeepSeek-Math and adopted in DeepSeek-R1, replaces the learned critic of PPO with a group-relative baseline: for each task, G rollouts are sampled, their outcome rewards are normalized into advantages (reward minus group mean divided by group standard deviation), and all tokens in a rollout share the same advantage. DAPO (Yu et al., 2025) improved GRPO with four key techniques—Clip-Higher, Dynamic Sampling, Token-Level Policy Gradient Loss, and Overlong Reward Shaping—and demonstrated state-of-the-art mathematical reasoning results. Both algorithms treat every token in a rollout identically with respect to the outcome. DyPO (Feng et al., 2025) identified this as a limitation for multi-turn interactive reasoning environments where state transitions provide task-critical information, and proposed step-wise and rollout-level optimization variants of GRPO. However, DyPO's step-wise variant requires annotated per-step action labels, which limits its applicability to settings without expert demonstrations. Our method operates directly on the group rollout data that GRPO already collects and requires no additional annotations.

Process reward and credit assignment methods. Two main subfamilies exist: those that train explicit process reward models (PRMs) and those that learn step-level advantages implicitly. AgentPRM (Choudhury et al., 2025) trains a separate PRM using Monte Carlo rollouts to estimate step-level correctness probabilities, then uses these as dense rewards for policy optimization. SWEET-RL (Jain et al., 2025) proposes learning a turn-wise advantage function from trajectory preference pairs using a Bradley-Terry objective, parametrized via the LLM's language modeling head with a length-normalized log-probability ratio. A key requirement of SWEET-RL is access to training-time information (reference solutions, ground-truth web pages), which is unavailable in most real agent settings. iStar (Liu et al., 2025b) trains an implicit PRM jointly with the policy via trajectory-based DPO, using the log-probability ratio between the PRM and an old policy snapshot as step-level rewards. SPA-RL (Deng et al., 2025) trains a progress estimator that predicts task completion from the current state, then uses predicted progress deltas as intermediate rewards. All four methods share a common characteristic: they require training an auxiliary model separate from the policy. Our method differs by constructing step-level advantages directly from within-group action-probability comparisons without any auxiliary model.

Search-based and self-training approaches. Agent Q (Putta et al., 2024) uses Monte Carlo Tree Search (MCTS) over web pages to construct step-level preference pairs, then applies DPO at the action level. Agent-R (Xi et al., 2025) uses MCTS-guided reflection to construct revision trajectories where errors are identified and corrected mid-trajectory, then applies iterative SFT on these revised trajectories. Both approaches are effective but incur substantial computational cost from tree search and require repeated environment interaction during training. Our method does not require tree search; it uses the rollouts that GRPO-style algorithms already sample.

Multi-turn preference and curriculum methods. ETO (Song et al., 2024) proposes exploration-based trajectory optimization, where failed trajectories are refined through interaction and used as preference pairs. WebRL (Qi et al., 2025) presents a self-evolving online curriculum reinforcement learning framework with a separately trained outcome reward model and an advantage estimator, demonstrating strong results on web agent benchmarks. Multi-Turn DPO (Xiong et al., 2024) extends DPO to trajectory-level preference pairs but inherits DPO's credit assignment coarseness. HOPE (Abdulhai et al., 2025) introduces hindsight-guided exploration by splicing counterfactual actions from a learned generator into past trajectories. Our approach complements these methods: it can serve as the credit assignment module within any of these frameworks, replacing or augmenting their advantage estimation.

Synthetic data generation for agent training. APIGen-MT (Liu et al., 2025a) generates verifiable multi-turn agent training data through an agentic pipeline. PC Agent-E (Anonymous, 2025) proposes efficient computer-use agent training through trajectory collection, thought completion, and trajectory boosting without large-scale human demonstrations. AgentBank (Song et al., 2024b) curates 50,000+ interaction trajectories for SFT-based agent training. These works focus on constructing high-quality datasets for supervised fine-tuning rather than on improving the RL training signal itself.

Where our method stands. Closest to our work are iStar (Liu et al., 2025b), which produces step-level advantages but trains a separate implicit PRM, and the GRPO advantage formulation used in DAPO (Yu et al., 2025), which shares the group-sampling structure but applies the same advantage uniformly across all steps in a trajectory. Our method occupies a new point in the design space: step-level credit assignment from group counterfactuals without any auxiliary model. We do not claim to outperform PRM-based methods in absolute performance; our hypothesis is that CSA provides denser credit assignment than trajectory-level GRPO at negligible additional cost, approaching PRM-level sample efficiency with GRPO-level simplicity.

Methods

Background: GRPO advantage formulation

In GRPO (Shao et al., 2024), for each task prompt x, the current policy π_θ samples G rollouts {τ₁, ..., τ_G}. An outcome reward verifier produces a scalar reward r(τ) for each complete trajectory. Each rollout's advantage is computed as:

A^E(τᵢ) = (r(τᵢ) − mean({r(τⱼ)}_{j=1}^{G})) / std({r(τⱼ)}_{j=1}^{G})

This episode-level advantage is then applied uniformly to every token (or action step) in the rollout. The policy gradient at each step t in rollout i uses the same A^E(τᵢ), multiplied by the importance sampling ratio and clipped:

J_GRPO(θ) = E [1/(G·T) Σᵢ Σₜ min(ρ_{i,t}·A^E(τᵢ), clip(ρ_{i,t}, 1±ε)·A^E(τᵢ))]

where ρ_{i,t} = π_θ(a_{i,t}|o_{i,t},x) / π_old(a_{i,t}|o_{i,t},x).

Core observation: within-group counterfactuals

When GRPO samples G rollouts for the same task, at each step index t, the G rollouts produce G actions (a_{1,t}, a_{2,t}, ..., a_{G,t}) taken under G observation histories (o_{1,<t}, o_{2,<t}, ..., o_{G,<t}). Because observation histories diverge across rollouts, the actions at step t are not strictly comparable as counterfactuals in a causal sense. However, we argue they are informative as associative counterfactuals: if rollout i succeeded and rollout j failed, and at step t rollout i took an informative action while rollout j took a vacuous one, this difference carries a signal about action quality at that step position, even if the observation histories differ.

This is the same principle that makes GRPO work at the trajectory level: GRPO already compares trajectories with different observation trajectories under the same initial prompt. We extend this comparison to the step level.

Counterfactual Step Advantage (CSA) decomposition

We define the step-level counterfactual advantage for action a_{i,t} as:

A^S(a_{i,t}) = (ℓ_{i,t} − mean({ℓ_{j,t}}_{j=1}^{G})) / std({ℓ_{j,t}}_{j=1}^{G})

where ℓ_{i,t} is a step-level signal that captures the alignment between the action taken in rollout i at step t and the outcome of rollout i. We propose constructing ℓ as:

ℓ_{i,t} = A^E(τᵢ) · [log π_θ(a_{i,t}|o_{i,<t},x) − mean({log π_θ(a_{j,t}|o_{j,<t},x)}_{j=1}^{G})]

The intuition: a high-probability action under the current policy in a successful rollout is a strong candidate for a good action; a high-probability action in a failed rollout is a candidate for an action to discourage. The deviation of the action's log-probability from the group mean at that step captures how unusual or decisive the action was relative to alternatives the policy considered.

We then define the combined advantage:

A(a_{i,t}) = A^E(τᵢ) + λ · A^S(a_{i,t})

where λ is a mixing coefficient (set to 0.5 by default). This combined advantage preserves the trajectory-level signal while adding step-level differentiation. When λ = 0, the method reduces to standard GRPO. When λ > 0, favorable steps in failed trajectories receive less negative (or potentially positive) advantages, and unfavorable steps in successful trajectories receive less positive (or potentially negative) advantages.

A key property: the step-level decomposition preserves the sum of advantages across steps up to the group mean. For any rollout i, the sum over t of the combined advantages approximately recovers the episode-level advantage scaled by trajectory length, ensuring that the policy update respects the overall trajectory quality while differentiating actions within it.

Practical implementation

CSA is implemented as a thin wrapper around the GRPO advantage computation in the training loop. The steps are:

  1. Sample G rollouts {τᵢ} from the current policy π_θ for each task prompt.
  2. Collect outcome rewards r(τᵢ) from the environment.
  3. Compute episode-level advantages A^E(τᵢ) as in standard GRPO (Equation 1).
  4. For each step t, collect the G log-probabilities log π_θ(a_{j,t}|o_{j,<t},x) across all rollouts.
  5. For each (i,t), compute ℓ_{i,t} and A^S(a_{i,t}) as defined above.
  6. Form the combined advantage A(a_{i,t}) = A^E(τᵢ) + λ · A^S(a_{i,t}).
  7. Apply the standard GRPO clipped policy gradient using A(a_{i,t}) instead of A^E(τᵢ) at each step.

Steps 4–6 add negligible computation relative to the rollout sampling (step 1) which dominates training time. No separate model training or extra forward passes are required.

Why this should help

In long-horizon tasks where only one or two critical actions determine success, standard GRPO spreads the sparse outcome signal uniformly across 20–50+ steps, effectively diluting the signal-to-noise ratio. CSA concentrates the signal on steps where the policy made a decisive choice—an action whose probability deviates from what the policy would typically do at that stage. The mixing coefficient λ controls the trade-off: with λ = 0, we recover GRPO; with λ = 1, we rely entirely on the step-level signal. We hypothesize that intermediate values provide the best of both worlds.

Where it is most likely to fail

CSA relies on the assumption that the policy's log-probability at each step is informative about action quality, even when observation histories differ across rollouts. In environments where observation histories diverge radically after the first few steps—for instance, in highly stochastic environments where identical actions lead to different states—the cross-trajectory comparison at step t becomes less meaningful. Additionally, in tasks where every step is equally critical and the policy's action probabilities are uniformly distributed (high entropy), the log-probability deviation carries little information. In such settings, CSA may provide no benefit over standard GRPO and could potentially add noise. We expect CSA to be most beneficial in tasks with a mix of routine and critical steps—the exact regime where trajectory-level credit assignment is most problematic.

Experiments Design

Tasks and environments

We evaluate on five benchmarks selected to span diverse multi-turn agent settings:

  1. WebShop (Yao et al., 2022): Web navigation and product search requiring multi-step interaction with search, click, and purchase actions. Outcome reward: whether the purchased product matches the specification.
  2. ALFWorld (Shridhar et al., 2021): Embodied household task completion in a text-based simulator. Tasks require 10–40 actions. Outcome reward: binary task completion.
  3. τ-bench (Yao et al., 2024): Tool-augmented task-oriented dialogue requiring API calls, database queries, and multi-turn customer-agent interaction. Outcome reward: task completion score.
  4. SWE-bench Lite (Jimenez et al., 2024): Software engineering tasks requiring multi-step code editing, file navigation, and test execution. Outcome reward: whether the produced patch passes all tests.
  5. VirtualHome (Puig et al., 2018): Household activity simulation with long-horizon action sequences (20–60 steps). Outcome reward: binary task completion.

These benchmarks cover web navigation (WebShop), embodied interaction (ALFWorld, VirtualHome), tool-augmented language tasks (τ-bench), and code-based agent tasks (SWE-bench Lite), with trajectory lengths ranging from 5 to 60+ steps.

Baselines

  1. Standard GRPO (Shao et al., 2024): The trajectory-level group-relative algorithm as used in DeepSeek-R1, adapted for multi-turn agent tasks.
  2. Standard DAPO (Yu et al., 2025): GRPO with Clip-Higher, Dynamic Sampling, Token-Level Policy Gradient Loss, and Overlong Reward Shaping.
  3. iStar (Liu et al., 2025b): Implicit PRM trained via trajectory-based DPO, alternating with policy optimization. Represents the state of the art in model-based step-level credit.
  4. SPA-RL (Deng et al., 2025): Stepwise progress attribution with a trained progress estimator for intermediate rewards.
  5. Multi-Turn DPO (Xiong et al., 2024): Trajectory-level preference optimization without a critic.
  6. CSA-GRPO (our method): GRPO with Counterfactual Step Advantage replacing the uniform episode-level advantage.
  7. CSA-DAPO (our method): DAPO with CSA replacing the uniform episode-level advantage.

We test two ablations of our method: (a) CSA-GRPO (λ = 0.0): equivalent to standard GRPO—verifies that any improvement is due to the CSA mechanism; (b) CSA-GRPO (λ = 1.0): pure step-level signal without trajectory-level weighting—tests the contribution of each component.

Training protocol

All methods use the same base model (Llama-3.1-8B-Instruct) and the same training budget per benchmark: 10,000 online rollout steps. For GRPO, DAPO, and CSA variants, we use G = 8 rollouts per task prompt, following the DAPO dynamic sampling recommendations. For iStar, we follow the authors' setup with alternating PRM and policy optimization. For SPA-RL, we train the progress estimator on exploration data as described in the original paper. All methods start from the same SFT checkpoint (behavior cloning on 100 expert demonstrations per benchmark) to ensure a fair comparison.

Metrics

  • Task success rate: The primary metric—percentage of evaluation tasks completed successfully.
  • Sample efficiency: Success rate as a function of the number of environment interactions during training (learning curves).
  • Step-level credit accuracy: For a subset of tasks with annotated step-level importance labels (collected for analysis only, not used in training), we measure whether the algorithm's implicit per-step advantage correlates with human judgments of step quality. This is a diagnostic metric to verify the mechanism.
  • Trajectory length at convergence: Average number of steps taken by the trained agent—measures whether step-level credit leads to more efficient action sequences.
  • Generalization to unseen tasks: For ALFWorld, we evaluate on held-out unseen task instances. For τ-bench, we evaluate on held-out dialogue domains.

What result would support the idea

A successful outcome would show CSA-GRPO substantially outperforming standard GRPO on success rate and sample efficiency, particularly on the longest-horizon benchmarks (VirtualHome, ALFWorld), while approaching the performance of iStar and SPA-RL at a fraction of the training cost (since no auxiliary model is trained). CSA-DAPO should similarly outperform standard DAPO, demonstrating that CSA is composable with other GRPO improvements. The ablation with λ = 0 should recover GRPO performance, and λ = 1 should underperform intermediate λ values, confirming that the combined advantage is necessary.

What result would weaken the idea

If CSA-GRPO performs equivalently to standard GRPO across all benchmarks and trajectory lengths, the step-level signal is not carrying useful information beyond the trajectory-level signal. If CSA underperforms GRPO, the cross-trajectory comparisons are introducing noise rather than signal. If CSA improves over GRPO but fails to approach iStar or SPA-RL even on long-horizon tasks, then auxiliary model-based credit assignment may be fundamentally necessary for substantial gains.

What result would falsify the idea

Performance degradation relative to GRPO on the majority of benchmarks, combined with negative step-level credit accuracy (anti-correlation between CSA advantages and human step-quality annotations), would indicate that within-group action-probability comparisons are actively misleading, falsifying the core mechanism. Similarly, if the method shows benefit only on short-horizon tasks (under 10 steps) but degrades on longer horizons, the core claim that CSA helps specifically with the long-horizon credit assignment problem is false.

Impact Statement

This work targets a practical bottleneck in training capable LLM agents: the sparsity of feedback in multi-turn RL. If successful, CSA would make dense, step-level credit assignment available as a drop-in component of existing RL pipelines without the engineering complexity of training and maintaining separate critic or PRM models. This would lower the barrier to training LLM agents for real-world interactive tasks—customer support, software development assistance, autonomous research, data analysis—where defining per-step rewards is infeasible but outcome rewards are readily available.

The method's simplicity means it can be adopted by any group training open-source LLM agents with GRPO or DAPO, which is the most widely used open-source RL recipe for LLM training today. By improving sample efficiency, it could reduce the compute cost of agent training, making it more accessible to academic and smaller-industry labs. We do not foresee specific negative societal impacts beyond those inherent to more capable AI agents; the method is a training signal improvement, not a capability unlock.

Success Criteria

Supports the idea: CSA-GRPO achieves at least a 5 percentage point improvement in task success rate over standard GRPO on benchmarks with average trajectory length ≥ 15 steps (ALFWorld, VirtualHome, SWE-bench Lite), with the gap widening as trajectory length increases. Step-level credit accuracy (correlation between CSA advantages and human step-importance judgments) is positive and statistically significant (p < 0.05). CSA-GRPO achieves at least 70% of the performance gain of iStar (the strongest PRM baseline) relative to GRPO, while requiring no additional model training beyond the policy.

Weakens the idea: CSA-GRPO shows small improvements (< 3 percentage points) over GRPO that do not increase with trajectory length, or step-level credit accuracy is positive but not statistically distinguishable from zero. Performance gains are concentrated on short-horizon tasks, contradicting the mechanism's claimed benefit for long-horizon credit assignment.

Falsifies the idea: CSA-GRPO performs worse than or equal to standard GRPO on the majority of benchmarks, step-level credit accuracy is near zero or negative, performance degrades on long-horizon tasks, or the method is dominated by both GRPO and simpler baselines like Multi-Turn DPO.

References

Abdulhai, M., White, I., Snell, C., Sun, C., Hong, J., Zhai, Y., Xu, K., and Levine, S. (2025). The road not taken: Hindsight exploration for LLMs in multi-turn RL. In EXAIT Workshop at ICML 2025. https://openreview.net/forum?id=1jqYCS1LxR

Anonymous (2025). PC Agent-E: Efficient agent training for computer use. arXiv preprint arXiv:2505.13909. https://arxiv.org/abs/2505.13909

Choudhury, S., Jain, A.G., Gonzalez-Pumariega, G., Chen, W., and Zhao, W. (2025). Process reward models for LLM agents: Practical framework and directions. arXiv preprint arXiv:2502.10325. https://arxiv.org/abs/2502.10325

Deng, Y., Zhang, Y., Yuan, S., Xiao, Y., and Chen, J. (2025). SPA-RL: Reinforcing LLM agents via stepwise progress attribution. arXiv preprint arXiv:2505.20732. https://arxiv.org/abs/2505.20732

Feng, X., Shi, W., Yuan, M., Wu, J., Wang, Q., and Feng, F. (2025). DyPO: Dynamic policy optimization for multi-turn interactive reasoning. In ICLR 2025 Workshop on Foundation Models in the Wild. https://openreview.net/forum?id=OWDBiMKYdo

Jain, A.G., Gonzalez-Pumariega, G., Chen, W., Zhao, W., and Choudhury, S. (2025). SWEET-RL: Training multi-turn LLM agents on collaborative reasoning tasks. arXiv preprint arXiv:2503.15478. https://arxiv.org/abs/2503.15478

Jimenez, C.E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. (2024). SWE-bench: Can language models resolve real-world GitHub issues? In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=VTF8yNQM66

Liu, Z., He, Z., Liang, T., Yu, D., Shi, Y., and Song, L. (2025a). APIGen-MT: Agentic pipeline for multi-turn data generation. arXiv preprint arXiv:2504.03601. https://arxiv.org/abs/2504.03601

Liu, X., Gou, Z., Liang, T., Luo, R., Liu, H., and Yang, Y. (2025b). Agentic reinforcement learning with implicit step rewards. arXiv preprint arXiv:2509.19199. https://arxiv.org/abs/2509.19199

Puig, X., Ra, K., Boben, M., Li, J., Wang, T., Fidler, S., and Torralba, A. (2018). VirtualHome: Simulating household activities via programs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

Putta, P., Mills, E., Garg, N., Motwani, S., Finn, C., Garg, D., and Rafailov, R. (2024). Agent Q: Advanced reasoning and learning for autonomous AI agents. arXiv preprint arXiv:2408.07199. https://arxiv.org/abs/2408.07199

Qi, Z., Liu, X., Iong, I.L., Lai, H., Sun, X., Sun, J., Yang, X., Yang, Y., Yao, S., Xu, W., Tang, J., and Dong, Y. (2025). WebRL: Training LLM web agents via self-evolving online curriculum reinforcement learning. In The Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=oVKEAFjEqv

Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y.K., Wu, Y., and Guo, D. (2024). DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. https://arxiv.org/abs/2402.03300

Shridhar, M., Yuan, X., Côté, M.-A., Bisk, Y., Trischler, A., and Hausknecht, M. (2021). ALFWorld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations. https://openreview.net/forum?id=0IOX0YcCdTn

Song, Y., Yin, D., Yue, X., Huang, J., Li, S., and Lin, B.Y. (2024). Trial and error: Exploration-based trajectory optimization of LLM agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024). https://aclanthology.org/2024.acl-long.409

Song, Y., Xiong, W., Zhao, X., Zhu, D., Wu, W., Wang, K., Li, C., Peng, W., and Li, S. (2024b). AgentBank: Towards generalized LLM agents via fine-tuning on 50000+ interaction trajectories. In Findings of EMNLP 2024. https://aclanthology.org/2024.findings-emnlp.116

Wang, Z., Wang, K., Wang, Q., Zhang, P., Li, L., Yang, Z., Jin, X., Yu, K., Nguyen, M.N., Liu, L., Gottlieb, E., Lu, Y., Cho, K., Wu, J., Fei-Fei, L., Wang, L., Choi, Y., and Li, M. (2025). RAGEN: Understanding self-evolution in LLM agents via multi-turn reinforcement learning. arXiv preprint arXiv:2504.20073. https://arxiv.org/abs/2504.20073

Xi, Z., Ding, Y., Chen, W., Hong, B., Guo, H., Wang, J., Yang, D., Liao, C., Guo, X., He, W., et al. (2025). Agent-R: Training language model agents to reflect via iterative self-training. arXiv preprint arXiv:2501.11425. https://arxiv.org/abs/2501.11425

Xiong, W., Song, Y., Zhao, X., Wu, W., Wang, X., Wang, K., Li, C., Peng, W., and Li, S. (2024). Watch every step! LLM agent learning via iterative step-level process refinement. In Proceedings of EMNLP 2024. https://aclanthology.org/2024.emnlp-main.93

Yao, S., Chen, H., Yang, J., and Narasimhan, K. (2022). WebShop: Towards scalable real-world web interaction with grounded language agents. In Advances in Neural Information Processing Systems. https://openreview.net/forum?id=R9KnuFlvnU

Yao, S., Shafran, I., Cao, Y., and Narasimhan, K. (2024). τ-bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045. https://arxiv.org/abs/2406.12045

Yu, Q., Zheng, H., Xi, Z., Li, Z., Hu, J., Gao, K., Huang, K., Yu, L., Wang, L., Zhang, M., Li, M., Wang, N., Tian, P., Huang, P., Chen, Q., Wang, Q., Chen, R., Ge, R., Zhang, R., Pan, R., Wang, R., Jin, R., Chen, S., Lu, S., Zhou, S., Chen, S., Ye, S., Wang, S., Zhou, S., Pan, S., Li, S., Shao, S., Wu, T., Ye, T., Yun, T., Pei, T., Sun, T., Zeng, W., Zhao, W., Liu, W., Liang, W., Gao, W., Yu, W., Zhang, W., Xiao, W., An, X., Liu, X., Wang, X., Chen, X., Nie, X., Cheng, X., Liu, X., Xie, X., Liu, X., Yang, X., Li, X., Su, X., Lin, X., Li, X., Jin, X., Shen, X., Chen, X., Sun, X., Wang, X., Song, X., Zhou, X., Wang, X., Shan, Y., Li, Y., Wang, Y., Wei, Y., Zhang, Y., Xu, Y., Li, Y., Zhao, Y., Sun, Y., Wang, Y., Yu, Y., Zhang, Y., Shi, Y., Xiong, Y., He, Y., Piao, Y., Wang, Y., Tan, Y., and Ma, Y. (2025). DAPO: An open-source LLM reinforcement learning system at scale. arXiv preprint arXiv:2503.14476. https://arxiv.org/abs/2503.14476