CoPE: Teaching Language Agents to Forecast Their Own Future Actions

TL;DR. Agentic RL usually learns from one scalar reward at the very end of a long rollout. CoPE adds a cheap extra signal: on successful rollouts, the agent is also trained to forecast the next few actions from each point in the trajectory. Acting and forecasting share one LLM backbone and improve each other during training. At inference nothing changes: the agent still acts one step at a time, with zero extra cost.

Paper · Code · Project page

One reward, many steps

Training LLM agents with reinforcement learning is expensive. A single rollout in WebShop, ScienceWorld or AppWorld can involve dozens of interaction steps and tool calls, and yet the standard recipe (e.g., GRPO) reduces all of that to one number: did the episode succeed or not? Every action in the trajectory receives the same advantage. The learning signal tells the agent whether it succeeded, not how the trajectory unfolded.

So the natural question is: given a history \(h_t\) somewhere in the middle of a rollout, what else could we ask the model to predict? Prior work has explored two kinds of targets, and both have problems.

• Value targets estimate how likely success is from \(h_t\). But a trajectory only records its final outcome, so every step is labeled with the same \(Y\). Telling steps apart requires step-level annotation, extra rollouts from intermediate states, or exact state recurrence.
• Observation targets ask the model to predict the next observation \(o_{t+1}\). In realistic environments that observation is an HTML DOM tree or a wall of terminal output, and most of its tokens have nothing to do with the decision. The loss ends up dominated by reproducing noise, and it only reaches one step ahead.

The example below makes the contrast concrete. The terminal log after running the tests is long and messy; the action that actually fixes the bug is a single short line.

The idea: forecast your own future actions

CoPE introduces a third kind of target: multi-step action targets. From history \(h_t\) in a successful rollout, the model is trained to produce the actions that actually followed, \(a_{t:t+K} = (a_t, a_{t+1}, \dots, a_{t+K})\), skipping the observations in between.

Why is this the right target? Consider the hindsight action chunk, the distribution over the next \(K+1\) actions among rollouts that end in success:

Two observations make this useful:

1. GRPO already fits the \(K = 0\) slice of it. Under binary outcome rewards, the expected (idealized, on-policy) GRPO update follows the gradient of a cross-entropy against \(q(a_t \mid h_t)\), the single next action of successful rollouts. It never asks the representation at \(h_t\) to anticipate what comes after.
2. Samples of the full chunk are free. Any action sequence that followed \(h_t\) in a successful rollout is already an exact sample from \(q(a_{t:t+K} \mid h_t)\). No world model, no simulated futures: we just read the targets off the rollouts we already collected.

So success enters as a condition that selects trajectories, not as a label stamped on every step. Each step gets its own continuation as its target; those targets are concise and decision-relevant; and they span several steps of the horizon.

How CoPE trains: two losses, one backbone

Concretely, CoPE uses a single LLM for two roles:

• the policy \(\pi_\theta(a_t \mid h_t)\), the model’s usual next-action distribution, and
• the forecaster \(p_\theta(a_{t:t+K} \mid h_t)\), the same model with a forecasting instruction appended to the history.

The training objective is simply

where the action forecasting loss \(\mathcal{L}_{\text{AF}}\) is the negative log-likelihood of the realized chunk \(a_{t:t+K}\) at every step of every successful rollout. Three details matter:

• Success filtering. Fitting failed trajectories too would learn the unconditional rollout distribution and push the shared backbone toward exactly the behaviors GRPO is trying to suppress. Without success conditioning, training collapses to zero reward within the first few steps.
• Group normalization. As the policy improves, more rollouts in each group succeed. An unnormalized sum would make the forecasting gradient grow with the success count and fight the RL objective, so CoPE averages over the successful rollouts in each group.
• No inference change. The forecasting instruction is only used in training. At test time the agent acts with \(\pi_\theta\) alone, one action per step, so a CoPE policy costs exactly what a GRPO policy costs.

Why it works: acting and forecasting co-evolve

The name CoPE comes from a two-way loop between the two roles.

Acting helps forecasting, via trajectory collection. The forecasting targets come from the policy itself. For a fixed rollout policy, the optimal forecaster is, by Bayes’ rule,

the policy’s own action chunks reweighted by their chance of success. Success filtering implements this reweighting by rejection sampling. The target is therefore always a step ahead of the policy that generated it, and a better policy supplies more (and better) successful rollouts to learn from.

Forecasting helps acting, via implicit temporal ensembling. This is the less obvious direction, and the animation below is the best way to see it.

Reading the grid by column, each action of a successful rollout is supervised not only when it is taken, but also when it is forecast from the \(K\) preceding histories. These forecasts don’t conflict with acting: they are all conditionals of the same hindsight distribution, so the best forecast of \(a_t\) made at \(h_{t-k}\) is the GRPO target marginalized over the observations in between. Because the forecaster and policy share every weight, this ensemble over time is folded into the policy itself, at no inference cost.

If this sounds familiar from robotics, it should. Action chunking policies such as ACT predict sequences of future actions, and recent analysis attributes much of chunking’s benefit to exactly this kind of ensemble of delayed predictors. CoPE borrows the training signal of chunking but still acts one step at a time.

Does the ensemble really end up inside the policy? To test this, we trained a separate LoRA forecaster on trajectories from a GRPO policy (so forecasting gradients could not touch the policy), and used it at test time to rerank 8 candidate actions with forecasts from earlier histories. On ScienceWorld, compared with reranking the same candidates without forecasts, this explicit ensembling lifts GRPO by +7.5 and +3.0 points at 150 and 250 training steps, narrowing its gap to CoPE, but it adds nothing when applied to CoPE (−0.5). What GRPO only gets from a second model and extra forward passes at test time, CoPE already delivers in a single forward pass.

Results

We evaluate on five long-horizon, sparse-reward benchmarks: three simulated environments (ALFWorld, ScienceWorld, WebShop) and two realistic applications (τ²-bench, customer service under policy constraints with a user simulator; and AppWorld, composing API calls across apps with code as actions). We use Qwen2.5-7B-Instruct everywhere except AppWorld, which uses Qwen2.5-14B-Instruct.

Compared with GRPO in the same setup, CoPE gains +5.0 points on ALFWorld, +12.5 on WebShop, +8.0 on ScienceWorld, +1.2 on τ²-bench and +13.7 on AppWorld’s Normal split. On the harder AppWorld Challenge split it also improves, from 18.7% to 22.5%.

On the simulated tasks CoPE also beats the two observation-prediction baselines, ECHO and RWML, on every aggregate metric, which supports the core claim: forecasting actions over several steps is a better auxiliary target than predicting the next observation. The gains are not uniform at the subtask level, though; ECHO or RWML do better on ALFWorld’s Cool and Pick2 families.

Faster learning

The improvements show up early in training, not just at the end.

Little overhead

The forecasting pass runs only over successful rollouts, so its cost scales with the batch success rate. On ScienceWorld it adds 11% time per training step on average, but the better policy produces shorter rollouts, so the measured end-to-end step time actually drops by 9% compared with GRPO. There is no separate model, no pretraining stage, and no inference overhead.

What matters: a few ablations

Multiple delays beat a single one. One way to read the forecasting loss: the \(k\)-th term predicts \(a_t\) without the last \(k\) observations, a “delay-\(k\)” prediction. Variants that train different sets of delays line up neatly (success rate change vs. GRPO on ScienceWorld):

Plain imitation of successful actions (delay 0) even hurts: success filtering keeps detours, suboptimal or invalid actions that the agent later recovers from, and the recovery that decides success only shows up at later delays. Delay 0, a single delay, and the full ensemble rank in the same order as they do for robot policies.

A few more findings: group normalization suppresses gradient-norm spikes and stabilizes late training, and the gains carry over to a different backbone. With Qwen3-8B, CoPE improves both score and success rate on WebShop and ScienceWorld, e.g. ScienceWorld success 61.5 → 71.5.

Takeaways

• Agentic RL wastes most of each rollout; successful rollouts already contain step-specific supervision in the form of the actions that followed each history.
• Forecasting those multi-step action chunks is a cleaner auxiliary target than values (shared across steps) or observations (long, noisy, one step ahead).
• Sharing one backbone makes acting and forecasting co-evolve: the policy supplies better targets, and forecasting folds a temporal ensemble back into the policy.
• It’s cheap: one extra loss, no extra model, no inference cost, and up to 2.0× better sample efficiency than GRPO.

Check out the paper, the code, and the project page for the full details. This is joint work with Haofei Yu (equal contribution), Yining Zhao, Xuyan Ye, Yuxiang Liu, Zhigang Hua, Yan Xie, Shuang Yang, and Jiaxuan You.

@article{yu2026cope,
  title   = {Co-Evolving Acting and Forecasting for Language Agents},
  author  = {Yu, Haofei and Chen, Yixing and Zhao, Yining and Ye, Xuyan and
             Liu, Yuxiang and Hua, Zhigang and Xie, Yan and Yang, Shuang and
             You, Jiaxuan},
  journal = {arXiv preprint arXiv:2610.20597},
  year    = {2026}
}