Plano: The Envoy-Based AI Proxy That Routes Agents With a 4B-Parameter Model

Hook

Most API gateways use simple regex and path matching to route traffic. Plano uses a 4-billion parameter language model to read natural language descriptions of your agents and decide where requests should go.

Context

Building multi-agent AI systems creates an orchestration nightmare. You need to decide which agent handles each request, route between multiple LLM providers for cost or latency optimization, add guardrails for safety, and capture telemetry for debugging and improvement. The typical approach embeds all this logic directly in your application code—hard-coded if-else chains checking intent, manual API client instantiation for different models, custom middleware for logging, and per-agent instrumentation for observability.

This creates tight coupling between your agents and infrastructure concerns. Every new agent requires code changes across your routing layer. Switching LLM providers means refactoring API clients. Adding safety filters requires touching every service. Testing becomes complex because you can't easily mock the orchestration layer. And because this logic lives in your application, it's language-specific—your Python agents can't share infrastructure code with your Node.js agents. Plano takes a different approach: move orchestration, routing, safety, and observability into a proxy layer that sits between your clients and agents, turning infrastructure concerns into declarative configuration.

Technical Insight

Plano is built on Envoy by members of the original Envoy core team, which means it inherits Envoy's production-hardened architecture: event-driven I/O, filter chains, dynamic configuration, and battle-tested reliability. But instead of using Envoy's traditional route matching (checking paths, headers, and query parameters), Plano adds an intelligent routing layer powered by Plano-Orchestrator, a custom 4B-parameter model that understands semantic intent.

Here's how you configure agents in Plano's YAML:

agents:
  - id: weather-agent
    description: "Provides current weather conditions and forecasts for any location worldwide"
    endpoint:
      http:
        uri: "http://weather-service:8080"
        timeout: 5s
  
  - id: calendar-agent
    description: "Manages calendar events, schedules meetings, and checks availability"
    endpoint:
      http:
        uri: "http://calendar-service:8080"
        timeout: 3s

listeners:
  - name: main-listener
    address: 0.0.0.0:9090
    routes:
      - match:
          prefix: "/v1/chat/completions"
        route:
          orchestrator:
            model: "plano-orchestrator-v1"
            agents:
              - weather-agent
              - calendar-agent

When a request arrives at /v1/chat/completions with a user message like "What's the weather in Seattle?", Plano sends the prompt and agent descriptions to the orchestration model. The model semantically matches the request intent to the agent descriptions and routes to the weather-agent—no regex patterns, no keyword matching, no hardcoded logic. If you add a new agent, you just add a YAML block with a natural language description; the routing model figures out when to use it.

The agent endpoints themselves are simple HTTP servers implementing OpenAI-compatible interfaces. This means any language works:

# A minimal Plano-compatible agent in Python
from flask import Flask, request, jsonify

app = Flask(__name__)

@app.route("/v1/chat/completions", methods=["POST"])
def chat_completions():
    data = request.json
    messages = data.get("messages", [])
    
    # Your agent logic here
    response_content = process_weather_request(messages)
    
    return jsonify({
        "id": "chatcmpl-123",
        "object": "chat.completion",
        "choices": [{
            "message": {
                "role": "assistant",
                "content": response_content
            },
            "finish_reason": "stop"
        }]
    })

Because Plano uses standard OpenAI response formats, your agents don't need custom SDKs or framework dependencies. The proxy handles all the infrastructure complexity through Envoy's filter chain architecture. You can add moderation filters, memory hooks, rate limiting, and retry logic entirely through configuration:

filters:
  - name: content-moderation
    type: plano.filters.moderation
    config:
      provider: openai
      block_threshold: 0.8
      categories:
        - violence
        - sexual
        - hate
  
  - name: observability
    type: plano.filters.telemetry
    config:
      export:
        otlp:
          endpoint: "otel-collector:4317"
      capture:
        - prompts
        - completions
        - token_usage
        - latency

These filters apply to all agents automatically. Add a new agent, and it immediately gets the same safety guardrails and telemetry without any code changes. The observability filter emits OpenTelemetry traces with what Plano calls 'Agentic Signals'—structured data about agent interactions, token usage, model performance, and routing decisions. This gives you a unified view across your entire agent fleet.

Plano also handles LLM routing, letting you define fallback chains or cost-optimized routing strategies:

llm_providers:
  - id: primary-llm
    provider: openai
    model: gpt-4
    api_key: ${OPENAI_API_KEY}
  
  - id: fallback-llm
    provider: anthropic
    model: claude-3-sonnet
    api_key: ${ANTHROPIC_API_KEY}

routing_strategy:
  type: fallback
  order:
    - primary-llm
    - fallback-llm
  retry_policy:
    max_attempts: 2
    backoff: exponential

If your primary LLM is rate-limited or times out, Plano automatically fails over to the backup. You can switch models, add providers, or change routing strategies without touching application code—the agents just call the Plano endpoint and don't care about which LLM ultimately services the request.

Gotcha

The biggest limitation is the dependency on Plano's hosted orchestrator model and LLM providers, which are currently only available in the US-central region. The free tier exists for testing, but production deployments require either hosting the orchestration model yourself (which requires significant ML infrastructure) or contacting the Katanemo team for API keys. This creates potential vendor lock-in: your routing logic depends on their proprietary model, and while they claim the model will eventually be open-sourced, there's no timeline.

The Envoy foundation is both a strength and a weakness. If you're already running Envoy in your infrastructure, Plano integrates naturally. But if you're not, you're inheriting Envoy's operational complexity: dynamic configuration management, filter chain ordering, TLS certificate management, and the learning curve of xDS APIs. For teams building simple single-agent applications, this is massive overkill—you're deploying a sophisticated proxy infrastructure to route requests that could be handled by a few lines of application code. The project is also relatively young despite the star count, meaning edge cases and production gotchas are still being discovered. Early adopters should expect to contribute back bug reports and feature requests.

Verdict

Use if: You're building multi-agent systems with complex orchestration requirements, need to standardize infrastructure across polyglot services, want to decouple agent logic from cross-cutting concerns like safety and observability, or already operate Envoy-based infrastructure. Plano excels when you have multiple agents, frequently change routing logic, or need unified telemetry across diverse agents. The natural language routing is genuinely novel and reduces the maintenance burden of traditional route tables. Skip if: You're building a single-agent application where simple HTTP routing suffices, can't accept dependency on a young project with hosted model requirements, don't have the operational capacity to run Envoy in production, or need complete control over your orchestration logic without external model dependencies. Teams without existing Envoy expertise should carefully weigh the learning curve against the benefits.