SAT: Applying CIA Intelligence Analysis Frameworks to Large Language Models
Hook
The CIA spent decades developing frameworks to counter cognitive bias in high-stakes intelligence analysis. What happens when you apply those same techniques to make LLMs reason more carefully?
Context
Large language models are excellent at generating plausible-sounding answers, but they’re also notorious for hallucinations, confirmation bias, and anchoring on initial assumptions. Ask GPT-4 a complex geopolitical question and you’ll get a confident response—but how do you know it considered alternative hypotheses or challenged its own reasoning?
The intelligence community faced this problem long before LLMs existed. Analysts making life-or-death decisions needed systematic ways to combat cognitive biases. The CIA Tradecraft Primer codified 12 structured analytic techniques: methods like Analysis of Competing Hypotheses (ACH), Devil’s Advocacy, and What If? Analysis designed to force rigorous thinking. SAT (Structured Analytic Techniques) by Phyle Corp takes these declassified methodologies and implements them as an orchestration layer for LLM reasoning. Instead of asking a model for a single answer, SAT runs your question through multiple analytical frameworks sequentially, optionally using different LLM providers to challenge each other’s conclusions adversarially.
Technical Insight
SAT’s architecture revolves around sequential technique execution with optional adversarial critique. The core Python engine implements 12 techniques grouped into three categories: diagnostic (identify patterns and test hypotheses), contrarian (challenge assumptions), and imaginative (explore alternative scenarios). Each technique receives the original question, evidence, and outputs from previous techniques, creating a reasoning chain.
Here’s how you’d run a basic analysis from the CLI:
# Basic SAT analysis with evidence file
sat analyze "Will semiconductor export controls slow China's AI development?" \
--evidence ./china-ai-evidence.txt \
--techniques ach devil-advocacy key-assumptions \
--model openai/gpt-4
# With adversarial critique using different providers
sat analyze "Will semiconductor export controls slow China's AI development?" \
--evidence ./china-ai-evidence.txt \
--techniques ach devil-advocacy key-assumptions \
--model anthropic/claude-3-opus \
--challenger openai/gpt-4 \
--critique-mode adversarialThe adversarial critique mode is where things get interesting. After the primary model completes each analytical technique, a challenger model receives that output and explicitly tries to poke holes in the reasoning. The system prompts the challenger to identify logical fallacies, unconsidered evidence, and alternative interpretations. This cross-examination happens for each technique before moving to the next, creating a dialectical process.
The technique implementation uses a template pattern where each analytical method inherits from a base AnalyticTechnique class. For example, Analysis of Competing Hypotheses generates multiple competing explanations, evaluates evidence consistency for each hypothesis, and produces a matrix showing which evidence supports or refutes each hypothesis. Devil’s Advocacy takes the most likely conclusion and systematically argues against it. Key Assumptions Check identifies unstated assumptions and evaluates how sensitive the conclusion is to each assumption being wrong.
SAT offers three evidence gathering modes: direct input (you provide the evidence), file-based (load from text files), or deep research mode where it autonomously queries Perplexity or Brave Search APIs to gather relevant information before analysis begins. This creates a complete pipeline from information retrieval through structured reasoning to synthesis.
The system exposes multiple interfaces beyond the CLI. The FastAPI backend (sat serve) lets you integrate SAT into other applications via REST API. The Electron desktop app provides a GUI for users who prefer point-and-click over command line. Most intriguingly, SAT implements the Model Context Protocol (MCP), allowing it to function as a reasoning server for MCP-compatible clients like Claude Desktop.
The final synthesis step aggregates all technique outputs and produces a structured report showing the reasoning chain, key findings from each technique, and an overall conclusion with confidence levels. Because each step is logged and traceable, you get an audit trail showing exactly how the system arrived at its conclusion—something traditional LLM interactions lack.
Gotcha
SAT’s power comes with practical limitations that constrain when it’s actually useful. The most obvious is cost. Running a full 12-technique analysis with adversarial critique means 24+ LLM API calls per question (each technique plus its critique). With GPT-4 or Claude Opus, that adds up quickly, especially if you’re providing extensive evidence that inflates token counts. A single comprehensive analysis on a complex question could easily cost $5-10 or more.
The Node.js version constraints for the desktop app are a red flag. The explicit requirement for Node 20-24 with a warning against version 25+ suggests dependency brittleness. Electron apps already have a reputation for being difficult to maintain across Node versions; this constraint indicates you might face compatibility headaches.
More fundamentally, SAT assumes your question warrants this level of analytical rigor. Most decisions don’t. The intelligence community developed these techniques for questions like “Will this regime collapse?” or “Is this weapons program operational?”—scenarios where being wrong has catastrophic consequences and cognitive bias is a known hazard. Applying 12 analytical techniques with adversarial critique to “Should we migrate to microservices?” is probably overkill. The sequential nature also means analyses are slow; you’re trading speed for thoroughness. If you need rapid iteration or real-time responses, this approach won’t work.
Finally, the repository has 1 star and no description, suggesting minimal community validation. You’re essentially beta testing an experimental approach to LLM reasoning with limited evidence it actually works better than simpler methods.
Verdict
Use SAT if you’re making high-stakes decisions where the cost of being wrong exceeds the cost of thorough analysis, and you need defensible reasoning chains with explicit consideration of alternatives. It’s ideal for competitive intelligence, strategic planning, risk assessment, or research synthesis where cognitive bias is a known problem and you can justify the API costs and analysis time. The adversarial critique mode is genuinely innovative for catching reasoning errors that single-model approaches miss. Skip it if you need fast answers, are working with straightforward questions, can’t afford multiple API calls per analysis, or aren’t comfortable with experimental software that has minimal community traction. For most development decisions and technical questions, simpler prompt engineering or chain-of-thought approaches will give you 80% of the value at 10% of the cost and complexity.
