Teaching Multi-Agent LLM Systems to Improve Themselves

Credit assignment and policy-gradient optimization, redefined in language space.

Data Mining and Reinforcement Learning (DaRL) Lab
Arizona State University  ·  Cisco Research  ·  UNC Chapel Hill

Both papers accepted to EMNLP 2026

LangMARL MASkills DaRL-GenAI

One question, two answers

LLM agents coordinate well when a human writes the right prompts. They stop getting better the moment that human stops. The reason is not model capacity — it is that a multi-agent system only ever observes a global outcome: the team succeeded, or it did not. That signal cannot say which agent should change, so self-improvement degenerates into vague reflections and monolithic prompt rewrites.

Classical multi-agent reinforcement learning has a name for this — the credit assignment problem — and forty years of machinery for solving it. Our work brings that machinery into language space, where an agent's "parameters" are text rather than tensors. Two papers ask the same question at two levels of abstraction:

Who?
LangMARL attributes team outcomes to individual agents and updates their policies.
Which?
MASkills attributes them to individual skill invocations and evolves the skill library.
Text
Both replace numeric gradients with natural-language credits and edit directions.
EMNLP 2026 Main Credit Assignment Toolkit
LangMARL
Natural Language Multi-Agent Reinforcement Learning via Semantic Credit Assignment
LangMARL compared with Auto Prompt Engineer, Reflexion, and TextGrad/Symbolic paradigms.

LangMARL rebuilds the cooperative MARL stack in natural language: a language policy actor per agent, a centralized language critic that reads whole trajectories and emits per-agent causal credits, a language policy gradient estimator, and a language policy optimizer — all under centralized training with decentralized execution, and packaged with TorchRL-style abstractions.

It leads on HotPotQA, MATH, HumanEval, all five Overcooked-AI layouts, and Pistonball up to 20 agents. The critic's attributions agree with human annotators on 89.9% of trajectories, and identically-prompted agents spontaneously differentiate into complementary roles.

Project page Code Tutorial
EMNLP 2026 Findings Agent Skills Transfer
MASkills
Continual Skills Optimization for Multi-Agent LLM Systems
MASkills pipeline: skill execution, skill credit assignment, stabilized language gradient descent, skill evolution.

Experience memories record what happened; skills record what to do again. MASkills makes the skill library — skill.yaml, SKILL.md, resources — the thing being optimized. A critic assigns credit to individual skill invocations by counterfactual comparison; edit directions are aggregated across trajectories, skills, agents, and topologies and smoothed with momentum; four operators refine, induce, consolidate, and prune skills, with every edit gated by held-out validation and rollback.

It reaches 76.3 F1 on HotpotQA and 23.3 average on GAIA, and skills learned in one environment transfer to unseen tasks with no further optimization.

Project page Code

The shared recipe

Both frameworks run the same four-stage loop. Only the object being optimized changes — a policy in LangMARL, a skill library in MASkills.

  • Roll out under decentralized execution. Each agent acts from its own text-parameterized policy and local observation, with no access to global state.
  • Assign credit centrally, in language. A critic reads the full trajectory and explains, in words, how each agent — and in MASkills, each invoked skill — causally helped or hurt the outcome. This replaces the scalar value function.
  • Estimate a language gradient. Credits become structured natural-language edit directions: what to reinforce, what to change, what to drop. No numeric gradient is ever computed.
  • Optimize, then verify. Gradients are aggregated across a batch and applied as semantic updates. MASkills adds momentum across cycles for variance reduction and a held-out validation rule that rolls back any regression — a trust region for text.
Mapping between TorchRL abstractions and LangMARL abstractions.
Familiar abstractions, new representation. TensorDict → langmarl.State, ProbabilisticActor → langmarl.LLMActor, ValueOperator → langmarl.LLMCritic, Adam → langmarl.LanguagePolicyOptimizer. Building an LLM multi-agent optimizer looks like writing a standard deep-RL pipeline.

BibTeX

@inproceedings{yao2026langmarl,
    title={LangMARL: Natural Language Multi-Agent Reinforcement Learning via Semantic Credit Assignment},
    author={Yao, Huaiyuan and Da, Longchao and Liu, Xiaoou and Fleming, Charles and Chen, Tianlong and Wei, Hua},
    booktitle={Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
    year={2026},
    url={https://github.com/DaRL-GenAI/LangMARL},
}

@inproceedings{yao2026maskills,
    title={MASkills: Continual Skills Optimization for Multi-Agent LLM Systems},
    author={Yao, Huaiyuan and Liu, Xiaoou and Fleming, Charles and Chen, Tianlong and Wei, Hua},
    booktitle={Findings of the Association for Computational Linguistics: EMNLP 2026},
    year={2026},
    url={https://github.com/DaRL-GenAI/MASkills},
}