MASkills: Continual Skills Optimization for Multi-Agent LLM Systems

The standard recipe for a self-improving agent is self-reflection into an experience memory. Memories do preserve useful experience, but they are a weak basis for continual improvement: they record what happened in past trajectories rather than which action policy to reuse, they lack reliable invocation conditions, and as the store grows, useful lessons get buried under noisy, redundant, and stale free-form records.

A skill is a better unit. It is a structured package of procedural knowledge — a skill.yaml descriptor, a SKILL.md body, and auxiliary resources — that tells an agent when to invoke it, how to act, and which tools to use. Agents discover skills from lightweight descriptions, load the details on demand, and refine them as experience accumulates. Recent work develops such reusable abstractions, but stays predominantly single-agent, ignoring the coordination dynamics that determine team-level utility.

MASkills performs policy improvement directly in the agents' skill space rather than their parameter space, following the language-space policy-gradient structure of TextGrad and LangMARL. Each of its three stages answers one of the challenges above.

Skills are exposed as callable tools, so every invocation is visible in the execution log as a skill trace ξ_i. For each skill k that agent i actually invoked along a trajectory, a centralized language critic produces a structured credit C_i^text(τ, k) by contrasting the timesteps where k was active against the counterfactual where it was not — explaining whether the skill helped coordination, was redundant, caused a failure, or should be generalized. The critic also emits an agent-level residual credit for effects no single skill explains; that residual is what signals when a genuinely new skill is needed.

Raw critiques are noisy and often contradictory. MASkills first converts each credit into a trajectory-level edit direction g_i(τ, k) — a structured natural-language patch proposing refinements, adjustments, generalizations, or removals. An LLM aggregator then merges these recursively across four axes at once — trajectories → skills → agents → topologies — performing pattern merging, conflict resolution, and prevalence weighting. Because aggregation is topological, centralized, decentralized-peer, and hierarchical systems are just three ways of rooting the same step, and everything downstream is unchanged.

Finally, updates carry momentum: each edit combines the current aggregated gradient with the previous cycle's, so transient or contradictory critiques are suppressed and persistent improvements survive. This is the language-space analogue of momentum gradient descent.

Every candidate edit is evaluated on a held-out validation set before it is committed, with all other agents' skills held fixed. An update is accepted only if validation return does not drop by more than a tolerance δ that absorbs rollout noise; otherwise it is rolled back. Momentum plays the role of variance reduction; validation and rollback play the role of a trust region.

All tasks are instantiated as cooperative Dec-POMDPs: a team of role-specialized LLM agents (retrieval, verification, planning, memory tracking, tool use, decision making) interacts under decentralized execution toward a shared team objective. GPT-5.1 serves as the optimizer backbone; GPT-4o-mini (HotpotQA, LoCoMo) and Qwen2.5-7B (GAIA) act as the execution backbones.

Two checks. First, are evolved skills better than skills an LLM just writes when asked? Prompt-generated skills turn out to give only marginal gains over prompting the agents directly, whereas continually optimized skills produce substantial improvements on both LoCoMo and GAIA — iterative, trajectory-driven refinement yields much higher-quality procedural abstractions than one-shot generation.

Second, do they transfer? We move skill libraries learned in one environment into unseen target tasks with no additional optimization. Transferred skills improve downstream performance across every source–target pair tested: skills optimized on GAIA push HotpotQA past strong prompting baselines like CoT and MultiPersona, and LoCoMo-derived skills improve long-horizon memory reasoning. MASkills is capturing reusable behavioral patterns, not task-specific prompting heuristics.

We run MASkills under centralized, hierarchical, and decentralized-peer structures with the optimization pipeline untouched. It stays competitive throughout — the framework is not coupled to any one orchestration structure. Interestingly, the best topology is strongly task-dependent. HotpotQA and GAIA favor decentralized peers, which benefit from diverse exploration and independent retrieval or tool trajectories. LoCoMo favors centralization, especially for multi-hop memory: a shared global state reduces memory fragmentation across sessions and keeps long-horizon reasoning coherent. Hierarchical coordination lands in between.

Our experiments focus on cooperative settings with relatively fixed roles and communication topologies; dynamic, adversarial, and open-world environments remain underexplored. Extending MASkills to adaptive organizational structures, competitive multi-agent games, and large-scale decentralized coordination is natural future work.

As the skill library keeps expanding, scalability challenges emerge in retrieval, consolidation, and coordination efficiency. Hierarchical skill organization, retrieval compression, and lifelong-learning mechanisms are promising directions for efficient long-term skill evolution at scale.

Ethics. Automatically evolved skills can amplify undesirable behaviors inherited from the underlying LLMs — factual errors, social biases, unsafe tool use, overconfident reasoning. Held-out validation, rollback, consolidation, and pruning mitigate this, but they are not a substitute for human oversight in high-stakes applications. Deployments that learn from user interaction should follow strict privacy practices and maintain transparent audit logs of skill modifications.