> your AI agent picks dependencies from memory; give it dated facts — try starlog.dev ↗ vet your agent's deps ↗ vibe-coding is fine. vibe-importing isn’t. — try starlog.dev ↗ vibe-importing isn’t fine ↗ your agent has never seen your private packages — try starlog.dev ↗ facts for private packages ↗ a linter for the dependencies your AI agent picks — try starlog.dev ↗ a linter for agent deps ↗ whois is redacted, cdns mask the rest — get the real operator — try whoisgeni.us ↗ who really runs that domain ↗ domain attribution that shows its work — full evidence chain — try whoisgeni.us ↗ domain intel w/ evidence ↗

← Back to Articles

Transect: Building Auditable Truth from Agent Chaos with Judge Cohorts

[ View on GitHub ]

Transect: Building Auditable Truth from Agent Chaos with Judge Cohorts

Hook

When your AI agent runs for 300 turns across 48 hours, spawning sub-agents and evading sandboxes, saying "it tried to escape" in your security report isn't enough—you need timestamps, vote counts, and verifier attestations that survive legal review.

Context

Long-horizon agent evaluations—the kind where GPT-4 writes code for days or Claude attempts red-team scenarios across hundreds of tool calls—produce transcripts that humans can't reliably analyze. A 200-turn conversation with tool outputs, reasoning traces, and nested sub-agent spawns generates tens of thousands of tokens. Reading it sequentially destroys your ability to spot patterns: when did the agent shift from exploration to exploitation? Which sub-agent actually triggered the sandbox escape?

Existing solutions fail at scale. Langsmith and AgentOps give you raw event streams but no semantic classification. Scoring functions in eval frameworks like Inspect AI operate per-sample, missing intra-trajectory phase transitions. Rolling your own LLM judge works until you realize a single model's classification has no reliability guarantee—was that "exploitation phase" label accurate, or did the judge hallucinate based on a truncated digest? For AI safety researchers auditing adversarial agent behavior, shipping a report that says "the agent was deceptive in turns 45-67" requires proof: which judge models agreed, what was the vote distribution, did a verifier double-check low-confidence spans?

Technical Insight

Judged Path - LLM Costs

Free Path

Yes

No

Scout Transcript JSONL

Structural Scanner

Metadata Extraction

Token Usage, Compactions, Interventions

Digest Generator

DigestSettings Cap per Turn

Reasoning/Text/Tool/Subtask Summaries

Judge Cohort

Multiple Models Vote

Low Confidence?

Verifier Re-examines

Provenance DataFrame

Structural DataFrame

HTML Report / Custom Analysis

System architecture — auto-generated

Transect solves this with a two-tier architecture: structural scanners extract metadata for free, while judged scanners use LLM cohorts with voting to classify semantic phases and sub-agents, tracking provenance at every step.

The digest generation system is the hidden MVP. When your transcript hits 200 turns with tool outputs that individually exceed 10K tokens, you can't feed everything to a judge model. Naive truncation loses critical context. Transect's DigestSettings let you cap each message component independently:

from transect import DigestSettings, run_judged_scan

settings = DigestSettings(
    reasoning_tokens=500,  # Cap agent reasoning per turn
    text_tokens=300,       # Limit tool output text
    tool_calls_tokens=200, # Summarize tool call JSON
    subtask_tokens=400     # Cap sub-agent task descriptions
)

timeline = run_judged_scan(
    transcript_path="agent_run.jsonl",
    phase_spec="phase_vocabulary.yaml",
    digest_settings=settings,
    regime="cohort_with_verifier"
)

The genius is structural: each turn's digest preserves interpretability (you see what the agent did) while fitting hundred-turn sequences into a 128K context window. The judge receives turn summaries with explicit markers showing where content was capped, maintaining awareness of information boundaries.

Judge regimes implement production-grade reliability tracking. The cohort_with_verifier mode runs multiple models (GPT-4, Claude-3.5, Gemini) on each turn, records individual votes, computes agreement scores, then dispatches a verifier model to re-examine turns where vote entropy exceeds a threshold:

# Resulting DataFrame includes provenance columns
phase_df = timeline.phase_annotations
print(phase_df[['turn', 'phase_label', 'judge_votes', 
                'agreement_score', 'verified']].head())

#    turn phase_label              judge_votes  agreement_score verified
# 0     5  exploration  [explore, explore, explore]            1.00    False
# 1    23  exploitation [exploit, explore, exploit]            0.67     True
# 2    67  deception    [deception, deception, ...]           1.00    False

That verified=True on turn 23 means a fourth model re-judged the low-agreement classification. This isn't academic paranoia—when you're documenting that an AI attempted privilege escalation, you need evidence that survives adversarial scrutiny. The DataFrame schema treats every classification as a probabilistic signal with an audit trail, not ground truth.

The scanner pipeline separates concerns elegantly. Structural scanners always run first, extracting token usage, context compactions, human interventions, and sub-agent spawns through deterministic parsing—zero API cost. This gives you the skeleton:

from transect import run_structural_scan

# Fast, free metadata extraction
structural_timeline = run_structural_scan(
    transcript_path="agent_run.jsonl"
)

# Inspect compactions and sub-agent spawns before paying for judges
print(f"Compactions: {len(structural_timeline.compactions)}")
print(f"Sub-agents: {len(structural_timeline.sub_agent_spans)}")

Only after validating the structural parse do you spend API budget on judged scans with your phase vocabulary. This iteration model (structural → refine spec → judged) prevents wasting $50 re-running judges because you forgot to include "reconnaissance" in your phase taxonomy.

For adversarial evaluations, Transect supports Petri audit timelines where you're analyzing both a target agent and an auditor agent simultaneously. The digest system includes bidirectional context—when judging the auditor's turn, the digest includes what the target agent just did, and vice versa. This catches interaction patterns like "auditor probes, target evades" that uni-directional analysis misses.

All outputs are pandas DataFrames with explicit join keys (transcript_id, turn, agent_span_id, phase_index), not opaque JSON blobs. The HTML report is a rendering layer; real analysis happens in Jupyter:

import pandas as pd

# Merge phase annotations with token usage
analysis = phase_df.merge(
    structural_timeline.token_usage,
    on=['transcript_id', 'turn']
)

# Which phases burn the most tokens?
phase_costs = analysis.groupby('phase_label')['total_tokens'].sum()
print(phase_costs.sort_values(ascending=False))

This dataframe-first design means you can plug Transect into existing ML pipelines, train classifiers on judge-labeled phases, or build custom dashboards without fighting the framework.

Gotcha

The Inspect ecosystem lock-in is real. Transect only ingests Inspect eval logs and OpenClaw exports—if your team uses Langsmith traces, AgentOps data, or generic JSONL conversation logs, you're writing adapter code or switching eval frameworks. For teams outside the Inspect AI world (national labs, model providers running structured evals), this is acceptable. For startups using LangChain with Langsmith observability, it's a non-starter.

Judge costs scale brutally with transcript length and voting rounds. A 300-turn transcript with 3-model cohort voting and verifier re-checks on 20% of turns burns thousands of API tokens per run. There's no incremental update mechanism—refining your phase vocabulary means re-judging the entire transcript, not delta-updating low-confidence spans. If you're iterating rapidly on agent architectures and re-running evals hourly, judge costs become prohibitive fast. The structural-only mode helps, but you're still blind to semantic phases until you pay.

Digest capping trades recall for tractability. Judges never see full tool outputs—if your agent's sandbox escape attempt lives in characters 8,000-8,500 of a tool response but your text_tokens cap is 300, the judge classifies based on a truncated digest that omits the smoking gun. You can raise caps, but then you hit context limits on shorter models or blow your latency budget. There's no automatic importance-based summarization; you're guessing at caps based on domain knowledge.

Verdict

Use if: You're running adversarial agent evaluations where defensibility matters more than speed—security audits, capability elicitation for model labs, red-team assessments where you need evidence trails showing which models agreed the agent exhibited deceptive behavior. Use if your transcripts exceed 100 turns and involve sub-agent spawns or multi-day sessions where humans lose the plot. Use if you're already in the Inspect AI ecosystem or willing to migrate for structured eval workflows. The provenance tracking and judge cohort voting are production-grade reliability engineering that you'd spend engineer-months replicating. Skip if: Your eval transcripts are under 50 turns where a human can just read them. Skip if you're using Langsmith, AgentOps, or non-Inspect frameworks and can't justify the migration cost. Skip if you're iterating rapidly on agent code and re-running evals constantly—judge costs will wreck your API budget. Skip if you need real-time analysis during agent execution; Transect is post-hoc only. For short transcripts and rapid iteration, raw log inspection or simple custom scoring beats Transect's overhead. For long-horizon safety evaluations where "trust me, it was deceptive" doesn't fly, Transect is the only tool shipping audit-grade semantic analysis.