MAS-Orchestra: Training AI Agents to Compose Multi-Agent Systems with Reinforcement Learning
Hook
What if the best multi-agent architecture for your task isn't the one you manually designed, but one learned through thousands of trial-and-error rollouts? MAS-Orchestra trains a 7B model to compose agent systems the same way RLHF trains models to follow instructions.
Context
Every developer who's built with AutoGen or LangGraph knows the pain: you spend days wiring together agent graphs, tweaking which agent calls which, debugging circular dependencies, only to discover your carefully-crafted architecture performs worse than a single GPT-4 call. The dirty secret of multi-agent systems is that most of them are expensive theater—adding latency and API costs without meaningful accuracy gains.
The root problem is that agent orchestration has been a human problem. You decide upfront whether to use chain-of-thought, self-consistency, debate between agents, or parallel search. But optimal architecture varies wildly by task: a math competition problem (AIME) benefits from a single Reflexion agent iterating on its reasoning, while a multi-hop web research task needs three parallel SearchAgents scraping different sources. Salesforce AI Research's MAS-Orchestra reframes this as a machine learning problem: train a small orchestrator model with reinforcement learning to generate entire agent architectures in one shot, learning from reward signals which topologies actually improve answers versus which just burn tokens.
Technical Insight
MAS-Orchestra's core architectural insight is treating orchestration as holistic composition rather than incremental graph construction. The system maintains a library of agent primitives—CoT (chain-of-thought), Self-Consistency (sample multiple reasoning paths), Reflexion (self-critique and retry), Debate (multiple agents argue), and WebSearch (tool-augmented retrieval). The orchestrator is a fine-tuned Qwen2.5-7B-Instruct model that takes a user query and outputs a complete MAS architecture as structured JSON, specifying which agents to instantiate, how they connect, and what their individual prompts are.
Here's what an orchestrator output looks like for a complex research question:
{
"agents": [
{
"id": "search_1",
"type": "WebSearchAgent",
"query": "quantum entanglement experimental verification 2024",
"tools": ["tavily", "duckduckgo"]
},
{
"id": "search_2",
"type": "WebSearchAgent",
"query": "Bell inequality violations latest research",
"tools": ["brightdata"]
},
{
"id": "synthesizer",
"type": "ReflexionAgent",
"dependencies": ["search_1", "search_2"],
"prompt": "Synthesize findings on quantum entanglement experiments"
}
],
"execution_mode": "parallel",
"max_iterations": 2
}
This JSON gets executed by Ray workers that spawn actual agent instances powered by frontier models (GPT-4, Claude-3.5-Sonnet). The orchestrator itself never does the heavy reasoning—it's purely a routing and composition layer. This capability separation is economically clever: the 7B model learns which architectures work for which task types, while expensive frontier models handle actual problem-solving.
Training happens through GRPO (Group Relative Policy Optimization) on VERL, Volcano Engine's distributed RL framework. The pipeline collects 32 rollout samples per batch, where each rollout means the orchestrator proposes an architecture, Ray executes it end-to-end, and an LLM judge scores the final answer against ground truth. The reward function balances correctness (did we get the right answer?) against efficiency (how many agent calls did we make?). Over thousands of iterations, the orchestrator learns patterns: simple questions get routed to single CoT agents, medium complexity triggers Reflexion for self-correction, and high complexity spawns parallel search agents.
The breakthrough insight from their experiments on MASBench—a new benchmark with 400 tasks taxonomized across five complexity axes—is that the trained orchestrator learns dynamic delegation. On AIME24 math problems, it routes 100% of queries to single-agent Reflexion or Debate, recognizing that parallel search adds zero value for formal reasoning. But on BrowseComp+, a multi-hop web research benchmark, it consistently spawns 3-4 parallel WebSearchAgents because the task structure demands distributed information gathering. This wasn't hand-coded—it emerged from the reward signal.
The infrastructure stack reveals why this hasn't been done before. VERL handles rollout parallelization across distributed GPUs, managing model sharding for the 7B orchestrator while coordinating with Ray clusters executing heterogeneous agent workflows. Each training batch requires hundreds of frontier model API calls (every agent in every rollout architecture calls GPT-4/Claude), plus LLM-as-judge evaluations. The repo's installation instructions requiring VERL forks and version pinning (Ray==2.49.2, specific protobuf patches) show this is bleeding-edge research infrastructure, not production-ready tooling.
One underappreciated detail: the agent library's function-calling abstraction makes the action space tractable. Instead of an exponential graph-building problem (should agent A call agent B with these parameters at this timestep?), the orchestrator outputs a declarative config in one forward pass. This is why RL training converges—the credit assignment problem is cleaner when you're scoring complete architectures rather than chaining incremental decisions.
Gotcha
The agent primitive library is hand-curated and frozen at training time. You get CoT, Self-Consistency, Reflexion, Debate, and WebSearch—that's it. If you want to add a new agent type (say, a CodeExecutionAgent or ImageAnalysisAgent), you can't just plug it in at inference. The orchestrator was trained on a fixed set of primitives, and extending it requires retraining the entire GRPO pipeline with your new agent included in rollouts. This isn't a framework that dynamically discovers novel agent architectures; it optimizes composition within a predefined set.
The evaluation story exposes a deeper limitation: all benchmarks have ground-truth answers (AIME math problems, HotpotQA factual questions, BrowseComp+ research tasks with verified answers). This makes reward engineering trivial—just check if the final answer matches. But real-world multi-agent applications often target open-ended tasks: writing a research report, planning a vacation, debugging legacy code. How do you train an orchestrator when there's no single correct answer? The LLM-as-judge approach becomes your bottleneck, and the orchestrator could learn to exploit judge biases rather than actually improve reasoning quality. The repo provides zero guidance on reward engineering for subjective tasks, suggesting this hasn't been explored.
Infrastructure costs are prohibitive for anyone outside a research lab. Training requires massive API budgets—32 rollout samples per batch, each rollout potentially spawning multiple agents making frontier model calls, across thousands of training iterations. The paper doesn't report total training costs, but back-of-napkin math suggests tens of thousands of dollars in API fees for a single GRPO run. And that's before you factor in VERL's GPU requirements for the orchestrator training itself. Reproducibility is effectively gated behind institutional compute budgets.
Verdict
Use if: You're a research team building adaptive agent systems where optimal architecture genuinely varies by task complexity (not just by domain), you have the infrastructure budget for distributed RL training with VERL and Ray, and you're working on structured benchmarks with ground-truth answers where reward signals are unambiguous. MAS-Orchestra's real contribution is proving that orchestration can be learned rather than hand-designed, and MASBench alone is worth exploring as the first principled taxonomy of when multi-agent systems actually help versus add overhead. Skip if: You're deploying a fixed multi-agent workflow for production (just hardcode it in LangGraph—you'll get better reliability and debuggability), you lack institutional compute resources (the VERL dependency and API costs make this research-only infrastructure), you need to work with modalities beyond text (no vision/audio agent primitives exist), or your tasks are open-ended without clear correctness metrics (the reward engineering problem becomes intractable). Most teams should study the architectural ideas here—holistic composition, learned routing, dynamic agent selection—but implement them in lighter-weight frameworks rather than adopting this exact stack.