PYTHIA: Building a Self-Calibrating Forecast Oracle with Local LLMs

Hook

What if your agent could ask one question—'What's about to happen on Earth?'—and get back a JSON payload with probabilities, locations, and time horizons, all running on localhost?

Context

Most AI agents are temporal amnesics. They wake up, process your prompt, and have zero awareness of whether a hurricane just made landfall, elections flipped a parliament, or a volcano erupted six hours ago. If you want an agent that makes decisions grounded in reality, you end up stitching together NewsAPI, GDELT, earthquake feeds, financial tickers, and weather APIs—each with different auth schemes, rate limits, and data formats. Then you realize you're not just building an agent; you're building a data pipeline company.

PYTHIA solves this with aggressive scope constraint: it only ingests keyless, free feeds (USGS earthquakes, Wikipedia trends, currency rates, conflict data) and runs entirely on Ollama, so there's no OpenAI bill and no API key juggling. The core insight is architectural—rather than forcing developers to poll 30 endpoints and write merge logic, PYTHIA collapses everything into a single /agent/view call that returns the current world state plus forecasts for the next day, week, month, and year. It's not trying to be a generic 'data platform.' It's an opinionated oracle: you ask what's happening and what's next, and it gives you geocoded predictions with probabilities.

Technical Insight

PYTHIA's architecture splits into two coupled systems: a frontend that ingests feeds (a fork of Osiris, a Three.js globe app) and a Python/FastAPI backend that turns that data into forecasts. The frontend runs 30+ concurrent polling loops for RSS feeds, JSON APIs, and scraped sources, then exposes them via REST. The backend hits those endpoints every few minutes, merges heterogeneous events into a unified 'world brief,' and pipes that into Ollama to generate predictions.

The interesting part is the Brier-weighted swarm. When PYTHIA generates a forecast—say, '60% chance of oil disruption in Strait of Hormuz within 7 days'—it doesn't stop at one LLM call. It runs that prediction through four personas (Strategist, Economist, Naturalist, Skeptic), each implemented as a separate system prompt. Each persona rescores the prediction based on domain-specific reasoning, and their votes are weighted by historical accuracy tracked in runs/ledger.jsonl. If the Economist persona has a Brier score of 0.15 (lower is better) and the Strategist has 0.30, the Economist's probability adjustment carries twice the weight. Over time, personas that make accurate forecasts gain influence, while poorly calibrated ones fade into irrelevance.

Here's how you'd invoke the main forecast endpoint:

const response = await fetch('http://localhost:8088/forecast', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    horizons: ['24h', '1week', '1month'],
    filters: { min_probability: 0.3, categories: ['geopolitical', 'economic'] }
  })
});

const { forecasts } = await response.json();
// Returns:
// [
//   {
//     event: "Military escalation in Red Sea shipping lanes",
//     probability: 0.68,
//     horizon: "1week",
//     location: { lat: 15.552727, lng: 42.652466 },
//     swarm_consensus: 0.68,
//     swarm_split: 0.12,  // disagreement among personas
//     brier_weights: { strategist: 0.31, economist: 0.19, naturalist: 0.25, skeptic: 0.25 },
//     reasoning: "Three Houthi attacks in 72h, increased naval deployments..."
//   }
// ]

The swarm_split metric is crucial—it surfaces when personas disagree. A split of 0.05 means tight consensus; 0.30 means the Strategist thinks something is 80% likely while the Skeptic says 40%. For risk-sensitive agents, high-split forecasts are flags to defer action or seek external validation.

The /agent/view endpoint is where the design shines for agent integration. Instead of making 30 API calls, your agent gets a single JSON payload with live feed summaries, trending signals (e.g., 'Taiwan' mentions spiked 300% in 6 hours), and forecasts across all horizons:

import httpx

async def get_situational_awareness():
    async with httpx.AsyncClient() as client:
        view = await client.get('http://localhost:8088/agent/view')
        return view.json()

# Returns:
# {
#   "timestamp": "2025-01-15T08:30:00Z",
#   "world_brief": "5.2 earthquake Peru, EUR/USD -1.2%, 3 new conflicts...",
#   "trending_signals": [{"term": "Taiwan", "spike": 3.2, "sources": ["wikipedia", "news"]}],
#   "forecasts_24h": [...],
#   "forecasts_1week": [...],
#   "high_salience_events": [{"event": "...", "probability": 0.91}]
# }

This solves the 'cold start' problem for agents—no need to maintain context about what happened yesterday or last week. The agent wakes up, calls /agent/view, and immediately has both retrospective (what happened) and prospective (what's next) grounding.

The ledger system (runs/ledger.jsonl) is an append-only log where every forecast gets written with a unique ID, timestamp, and prediction details. Days or weeks later, an LLM-as-judge process reads archived world states and scores whether the prediction resolved true, assigning a Brier score. Those scores update each persona's weight. It's a self-improving loop that runs without manual labeling:

{"id": "f7a3", "event": "Oil >$95/bbl", "prob": 0.45, "horizon": "1week", "created": "2025-01-08T12:00:00Z"}
{"id": "f7a3", "resolved": true, "actual_date": "2025-01-12T09:15:00Z", "brier": 0.3025, "judge_reasoning": "Price hit $96.20 on Jan 12"}

The counterfactual endpoint /whatif lets you probe model reasoning without polluting the ledger. You can ask 'What if Iran closes Hormuz tomorrow?' and get back synthetic forecasts that explicitly aren't logged or scored. This is useful for scenario planning where you want to explore agent behavior under hypothetical conditions but don't want those explorations mixed into your calibration data.

Gotcha

The Osiris coupling is the project's Achilles heel. PYTHIA doesn't ingest feeds itself—it relies on a patched fork of Osiris (a globe visualization app) to run the polling loops and expose feed data via HTTP. This means you're maintaining two repos, applying manual patches described in INSTALL.md, and debugging integration issues when Osiris updates break PYTHIA's expected endpoints. If you want to add a new feed, you're editing JavaScript in a Three.js app, not Python in the forecasting engine. For containerized deployments or CI/CD pipelines, this is a non-starter—you can't docker pull a single image and run the oracle.

The LLM-as-judge scoring is fundamentally circular. When PYTHIA grades its own past forecasts, it's using the same model architecture (Ollama + Llama/Mistral/Qwen) to evaluate predictions that same architecture made weeks ago. If the model has a systematic bias—say, it overestimates geopolitical escalation or underweights economic factors—the judge inherits that bias and will score biased predictions as 'accurate.' There's no ground truth outside the LLM's worldview, so you can't detect when the swarm is confidently wrong in consistent ways. For production use, you'd need to replace this with actual outcome data (did oil hit $95? did the conflict escalate?) from structured sources, not LLM interpretations of archived news summaries.

Performance is constrained by Ollama inference speed. Generating forecasts for four horizons, each re-scored by four personas, means 16+ sequential LLM calls per forecast cycle (more if you're generating multiple predictions per horizon). On a MacBook Pro with a 32GB model, expect 20-40 seconds per full forecast run. The /whatif endpoint, which doesn't cache and runs fresh inference, can take 30+ seconds for a single hypothetical. This makes real-time agent interactions clunky—you can't have a conversational 'what if X, then what if Y?' loop without multi-minute latencies.

Verdict

Use if: You're building autonomous agents on local LLMs (Ollama/LM Studio) that need situational awareness without cloud dependencies, you're prototyping geopolitical risk models or humanitarian response systems where agent decisions should react to real-world events, or you want a self-contained research platform to benchmark different Ollama models on forecasting tasks using real feeds and automatic scoring. The /agent/view endpoint is legitimately the fastest way to give an agent 'what's happening + what's next' grounding in a single call, and the Brier-weighted swarm is a clever pattern for ensemble forecasting that self-corrects over time. Skip if: You need sub-10-second latency (the swarm deliberation is too slow for interactive use), you require confidence intervals or uncertainty quantification (PYTHIA returns point probabilities with no error bars), you're deploying in containers or cloud environments where maintaining a patched Osiris fork is unacceptable, or your domain needs non-free data sources like Bloomberg or proprietary intel feeds. For production forecasting, you'll likely extract the feed aggregation logic and replace the LLM-as-judge with real outcome tracking, but as a research artifact and agent integration pattern, PYTHIA is the most cohesive local oracle available.