SIRI: Self-Internalizing Reinforcement Learning with Intrinsic Skills for LLM Agent Training

EMNLP 2026 Main Conference

Proceedings of EMNLP 2026 · Budapest, Hungary · pp. 4402–4416

Zhongyu He1,2,*, Yuanfan Li2,*, Fei Huang2, Tianyu Chen2, Siyuan Chen2,†, Xingyang Li2,†, Meng Hsuan Yu2, Xiangrong Liu1, Leyi Wei3, Lu Pan2, Ke Zeng2, Xunliang Cai2
1Xiamen University, 2Meituan, 3Macao Polytechnic University
*Equal Contribution   †Corresponding Authors

Motivation

Motivation: comparison between skill-augmentation methods and SIRI
Conceptual comparison between (a) traditional skill-augmentation frameworks and (b) our SIRI.

Abstract

Long-horizon LLM agents can benefit from reusable skills, yet existing skill-based methods often rely on external skill generators during training or persistent skill retrieval at inference, increasing engineering complexity, context length, and deployment latency. We propose Self-Internalizing Reinforcement learning with Intrinsic skills (SIRI), a three-phase framework that enables agents to discover, validate, and internalize skills without external skill generators or inference-time skill banks. SIRI first warms up the policy with GiGPO to acquire basic interaction ability and collect successful skill-free trajectories. It then performs self-skill mining, where the current policy summarizes compact skills from its own successful plain rollouts and validates them through paired skill-augmented and skill-free rollouts. Finally, SIRI distills only beneficial skill-guided action tokens into the plain policy using trajectory-level utility and action-level advantage. At inference, the agent runs with the original prompt only. On ALFWorld and WebShop with Qwen2.5-7B-Instruct, SIRI improves GiGPO from 0.908 to 0.930 on ALFWorld and from 0.728 to 0.813 on WebShop, outperforming prompt-based, RL-based, and memory-augmented baselines. Further analysis shows that our self-mining strategy can achieve performance comparable to distillation with closed-source large model.

Framework

Overview of the SIRI framework

Overview of SIRI. (a) Phase 0: The agent interacts with the environment to collect initial trajectory data for cold-start policy initialization. (b) Phase 1: Candidate skills are extracted into a structured SkillBank, followed by parallel rollouts with and without skills to compute the hindsight macro utility Δg for dynamic skill evaluation. (c) Phase 2: The shared policy parameterizes validated skills via cross-context self-distillation under asymmetric inputs, optimized by an advantage-weighted loss ℒint gated by the macro performance indicator 1[Δg(i) > 0].

Phase 0: Policy Warmup

The agent is warmed up with GiGPO to acquire basic interaction ability and collect successful skill-free trajectories as high-quality behavioral evidence for skill mining.

Phase 1: Self-Skill Mining & Utilization

The current policy summarizes compact skills from its own successful plain rollouts. Candidate skills are validated through paired skill-augmented and skill-free rollouts via the hindsight utility Δg.

Phase 2: Skill Internalization

Useful skill-guided action tokens are distilled into the plain policy through cross-context self-distillation, weighted by action-level advantage and gated by trajectory-level utility.

Quantitative Results

Performance on ALFWorld and WebShop. For ALFWorld, we report the success rate on each subtask and the overall average. For WebShop, we report both the average score and the success rate. The best and second-best results among all listed methods are highlighted in bold and underline, respectively.

Method ALFWorld WebShop
PickLookCleanHeatCoolPick2All ScoreSucc.
API-based LLMs
Gemini-3-Flash 0.9640.8570.5710.7220.9620.9530.852 0.1410.165
GLM-5.1 0.9130.8180.8520.9380.6670.6670.797 0.1560.125
Base Model: Qwen2.5-7B-Instruct
Origin 0.1790.6430.0480.0000.0380.0530.125 0.1660.039
Prompt-based Agentic or Memory-based Methods
ReAct 0.4850.3540.3430.1320.1820.1760.312 0.4620.195
Reflexion 0.6200.4160.4490.3090.3630.2380.427 0.5810.288
RL-based Methods
PPO 0.9230.6400.9250.8950.8030.6880.804 0.8140.687
RLOO 0.8760.7820.8730.8130.7190.4890.755 0.8030.657
GRPO 0.9080.6610.8930.7470.7250.6470.776 0.7930.661
GiGPO 0.9770.8270.9880.8370.8930.7920.908 0.8440.728
Memory-Augmented RL-based Methods
MemRL 0.6280.3850.2220.1250.0800.0000.214 0.2950.092
EvolveR 0.6490.3330.4640.1330.3330.3330.438 0.4250.176
Mem0+GRPO 0.7810.5480.5610.3100.6500.2690.547 0.5810.375
SimpleMem+GRPO 0.8950.3630.6000.5000.6490.2630.625 0.6780.469
SkillRL 0.9790.7140.9000.9000.9550.8750.899 0.8520.727
SIRI (Ours) 0.9750.8751.0000.8570.8500.8950.930 0.8990.813

Training Dynamics & Analysis

You can slide to view more results.

Ablation Study on WebShop

Ablation study of SIRI on WebShop. We report the average score and success rate with Qwen2.5-7B-Instruct backbone.

MethodScoreSuccess
w/o Phase 00.8500.711
w/o Phase 2 (w/ Skill)0.8980.805
w/o Phase 2 (w/o Skill)0.8520.719
w/o Phase 1 and 2 (GiGPO)0.8440.728
SIRI (Full)0.8990.813

Impact of Base Optimization Algorithm

Impact of different base optimization algorithms (GRPO vs. GiGPO) on the performance of SIRI on WebShop. We report the average score and success rate with Qwen2.5-1.5B-Instruct backbone.

MethodScoreSuccess
GRPO0.7580.568
SIRI (GRPO)0.779↑0.0210.648↑0.080
GiGPO0.8310.650
SIRI (GiGPO)0.853↑0.0220.727↑0.077

Robustness on a Smaller Backbone (Qwen2.5-1.5B-Instruct)

Performance on ALFWorld and WebShop with Qwen2.5-1.5B-Instruct. For ALFWorld, we report the success rate on each subtask and the overall average. For WebShop, we report both the average score and the success rate. The best and second-best results are highlighted in bold and underline, respectively.

Method ALFWorld WebShop
PickCleanCoolLookHeatPick2All ScoreSuccess
Origin0.0590.0330.0420.0550.0970.0000.0410.2310.052
ReAct0.1740.1570.0770.2050.0620.0200.1280.4010.113
Reflexion0.3530.2170.1940.2220.1360.0370.2180.5580.219
PPO0.6480.5710.4640.4050.6060.4740.5440.7380.515
RLOO0.8830.7100.6640.5280.6280.5690.6970.7390.521
GRPO0.8530.8450.5970.5370.7820.5350.7280.7580.568
GiGPO0.9440.9480.7980.6750.9440.7640.8670.8310.650
SIRI (Ours)0.8971.0000.8700.8330.9470.6000.8750.8530.727

Examples of Self-Mined Skills

Each skill abstracts a successful trajectory into a reusable heuristic with applicability conditions, a high-level strategy, and actionable key steps.

Skill Example — ALFWorld

When to apply: When the required object is not found at the initial search location and must be located to proceed with the task.

Strategy: Evaluate the current observation to confirm the object's absence, then systematically navigate to the next most logical storage receptacle based on the object's typical context.

Key steps: Observe the current location | Verify the target object is absent | Identify the next probable storage receptacle | Navigate to the new location

Skill Example — WebShop

When to apply: When a product page is open and specific attribute variations must be chosen to match task requirements.

Strategy: Configure the product by selecting all required options, confirm that constraints like price remain satisfied, and finalize the transaction.

Key steps: Select required item attributes | Verify constraints are met | Execute the purchase action

Cite SIRI

Citation formats from the ACL Anthology paper page.

VenueEMNLP 2026 · Main Conference
LocationBudapest, Hungary
Pages4402–4416
PublisherAssociation for Computational Linguistics
Copy as

BibTeX

@inproceedings{he-etal-2026-siri,
    title = "{SIRI}: Self-Internalizing Reinforcement Learning with Intrinsic Skills for {LLM} Agent Training",
    author = "He, Zhongyu  and
      Li, Yuanfan  and
      Huang, Fei  and
      Chen, Tianyu  and
      Chen, Siyuan  and
      Li, Xingyang  and
      Yu, Meng Hsuan  and
      Liu, Xiangrong  and
      Wei, Leyi  and
      Pan, Lu  and
      Zeng, Ke  and
      Cai, Xunliang",
    editor = "Lee, Hung-Yi  and
      Dev, Sunipa  and
      Li, Jessy  and
      Elliott, Desmond",
    booktitle = "Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing",
    month = oct,
    year = "2026",
    address = "Budapest, Hungary",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.emnlp-main.192/",
    pages = "4402--4416",
    ISBN = "979-8-89176-459-0",
    abstract = "Long-horizon LLM agents can benefit from reusable skills, yet existing skill-based methods often rely on external skill generators during training or persistent skill retrieval at inference, increasing engineering complexity, context length, and deployment latency. We propose Self-Internalizing Reinforcement learning with Intrinsic skills (SIRI), a three-phase framework that enables agents to discover, validate, and internalize skills without external skill generators or inference-time skill banks. SIRI first warms up the policy with GiGPO to acquire basic interaction ability and collect successful skill-free trajectories. It then performs self-skill mining, where the current policy summarizes compact skills from its own successful plain rollouts and validates them through paired skill-augmented and skill-free rollouts. Finally, SIRI distills only beneficial skill-guided action tokens into the plain policy using trajectory-level utility and action-level advantage. At inference, the agent runs with the original prompt only. On ALFWorld and WebShop with Qwen2.5-7B-Instruct, SIRI improves GiGPO from 0.908 to 0.930 on ALFWorld and from 0.728 to 0.813 on WebShop, outperforming prompt-based, RL-based, and memory-augmented baselines. Further analysis shows that our self-mining strategy can achieve performance comparable to distillation with closed-source large model."
}
Download