Learning Reasoning Rewards from Expert Demonstrations with Inverse Reinforcement Learning

Authors: Claudio Fanconi, Nicolas Astorga, Mihaela van der Schaar

Affiliations: University of Cambridge

Venue: ICLR 2026 Workshop on LLM Reasoning & ICML 2026 Workshop on Decision-Making from Offline Datasets to Online Adaptation

Links: Paper PDF Code

Figure 1 overview for Learning Reasoning Rewards from Expert Demonstrations with Inverse Reinforcement Learning — Figure 1: R-AIRL learns reasoning rewards from expert demonstrations and reuses the reward for training, reranking, and diagnostics.

Abstract

Teaching large language models to reason during post-training often relies on reinforcement learning with explicit outcome- or process-based rewards, but such reward functions can be difficult to obtain or define for complex tasks. This paper proposes Reasoning Adversarial Inverse Reinforcement Learning (R-AIRL), which learns process-level reasoning rewards directly from expert chain-of-thought demonstrations instead of imitating expert traces through supervised fine-tuning. The learned reward can be reused across the reasoning pipeline: as a post-training signal, for inference-time reranking, and for process-level diagnostics that localise reasoning failures. Across GSM8K, MMLU-Pro, and MedReason, R-AIRL improves over supervised fine-tuning in most settings, increases pass@1 through reranking by up to 17.4 points, and localises reasoning failures with up to 86.1% accuracy.