Inside the Foundation Model Transparency Index: How Stanford Scores AI Giants on Disclosure

Hook

When researchers asked 14 major AI companies to disclose basic information about their foundation models, the average transparency score was 37 out of 100—and that was after companies had a chance to improve their submissions.

Context

The foundation model ecosystem has a transparency problem. When OpenAI released GPT-3 in 2020, the technical paper omitted critical details about training data composition, compute costs, and labor practices. Google's PaLM, Meta's LLaMA, and Anthropic's Claude followed similar patterns—impressive capabilities, sparse documentation. This opacity isn't just academic curiosity; it affects safety research, regulatory compliance, competitive dynamics, and public accountability.

The Foundation Model Transparency Index (FMTI) emerged from Stanford's Center for Research on Foundation Models (CRFM) as a systematic response. Rather than accept ad-hoc disclosure as the norm, researchers from Stanford, Princeton, and MIT developed a structured framework of 100 indicators spanning the entire foundation model lifecycle—from upstream resources like training data and compute infrastructure, through model characteristics like capabilities and risks, to downstream impacts including distribution and usage. First published in October 2023 and updated annually, the index doesn't just measure transparency; it actively shapes industry norms by creating public accountability for what companies choose to reveal or conceal.

Technical Insight

The FMTI repository structures its assessment framework around three key components: indicator definitions, company scorecards, and evidence documentation. Unlike traditional software projects with executable code, this repository's "technical architecture" is its data model and scoring methodology.

The core data structure lives in structured JSON and CSV files that define each of the 100 indicators. Each indicator follows a consistent schema specifying the question being asked, the rationale for why it matters, what constitutes full disclosure versus partial disclosure, and how to assign scores on a 0-1 scale. For example, Indicator 12 asks about training data sources:

{
  "indicator_id": "12",
  "domain": "Upstream",
  "subdomain": "Data",
  "question": "What are the sources of the training data?",
  "rationale": "Understanding data sources enables assessment of potential biases, copyright issues, and privacy concerns",
  "scoring": {
    "1.0": "Detailed breakdown of all data sources with proportions",
    "0.5": "General categories of data sources without proportions",
    "0.0": "No disclosure of training data sources"
  },
  "evidence_type": ["technical_paper", "model_card", "data_sheet"]
}

The scoring methodology combines two pathways: company-submitted reports and independent researcher assessment. In the 2025 iteration, seven companies (including OpenAI, Google, and Meta) submitted voluntary transparency reports following FMTI's structured template. These reports follow a standardized markdown format that maps directly to indicator responses:

## Indicator 12: Training Data Sources

### Our Response
Our training dataset consists of:
- 40% web crawl data (Common Crawl, filtered)
- 30% books corpus (licensed content)
- 20% code repositories (permissive licenses)
- 10% scientific papers (arXiv, PubMed)

### Evidence
- Technical Report Section 3.2 (link)
- Data Card v2.1 (link)

For companies that don't submit voluntary reports, researchers conduct independent assessments by reviewing all publicly available documentation—technical papers, blog posts, API documentation, terms of service, and media statements. This involves what the repository documents as a "multi-stage consensus process": two independent reviewers score each indicator, disagreements trigger discussion with domain experts, and final scores require majority agreement among at least three reviewers.

The repository's most interesting technical aspect is how it handles longitudinal tracking. Indicators aren't static; they evolve as the AI ecosystem changes and disclosure norms advance. The repo maintains separate directories for each year's methodology:

/indicators/
  /2023/
    indicators_v1.csv
  /2024/
    indicators_v2.csv
    changelog_2023_to_2024.md
  /2025/
    indicators_v3.csv
    changelog_2024_to_2025.md

This versioning approach allows researchers to track not just how individual companies improve their transparency scores over time, but how the standards themselves have risen. For instance, the 2024 update added new indicators about AI safety testing and red-teaming after these practices became more widespread. The 2025 version incorporated questions about downstream monitoring and misuse prevention after regulatory frameworks like the EU AI Act created new expectations.

The repository also includes analysis scripts (primarily Python notebooks) that generate comparative visualizations and statistical summaries. These aren't production-grade tools but rather research artifacts that demonstrate reproducible analysis:

import pandas as pd
import matplotlib.pyplot as plt

# Load company scores
scores_2025 = pd.read_csv('data/2025/company_scores.csv')

# Calculate domain averages
domain_scores = scores_2025.groupby(['company', 'domain']).mean()

# Identify transparency gaps
upstream_avg = domain_scores.xs('Upstream', level='domain').mean()
downstream_avg = domain_scores.xs('Downstream', level='domain').mean()

print(f"Industry avg - Upstream: {upstream_avg:.2f}")
print(f"Industry avg - Downstream: {downstream_avg:.2f}")
print(f"Gap: {abs(upstream_avg - downstream_avg):.2f}")

This data-driven approach reveals systematic patterns: companies typically score higher on model capabilities (things they want to advertise) and lower on training data composition, labor practices, and environmental impact (areas where disclosure creates competitive or reputational risk).

Gotcha

The FMTI's fundamental limitation is that it's a measurement tool, not an enforcement mechanism. Companies face no penalties for low scores beyond reputational pressure, and several major players (like Anthropic in earlier iterations) have declined to participate in the voluntary reporting process. The index can document opacity, but it can't compel disclosure. This means the most useful transparency data often comes from companies already committed to openness, creating a selection bias that potentially overstates industry-wide progress.

The scoring methodology, while rigorous, ultimately relies on subjective human judgment. Two expert reviewers might legitimately disagree about whether a company's vague statement about "diverse data sources" merits a 0.3 or 0.5 score. The repository documents consensus processes, but these don't eliminate inherent subjectivity in translating qualitative disclosures into quantitative scores. Additionally, the index can only evaluate what companies say, not whether those statements are accurate or complete. If a company claims to use licensed training data but actually scraped copyrighted content, FMTI has no independent verification mechanism. It's fundamentally a transparency index, not a truth index.

Verdict

Use if you're conducting AI policy research, corporate accountability investigations, or academic studies on foundation model governance. The FMTI provides the most comprehensive, systematically collected dataset on what major AI developers disclose and conceal. It's invaluable for journalists writing accountability stories, regulators designing disclosure requirements, or researchers studying transparency norms. The structured indicator framework and historical data enable rigorous longitudinal analysis that would be prohibitively expensive to collect independently. Skip if you're looking for executable tools, APIs, or software libraries—this is purely a research dataset with minimal code. Also skip if you need real-time data or coverage beyond the 13-14 largest foundation model developers; the annual publication cycle and selective company coverage mean emerging players and recent developments won't be reflected until the next update.