> your AI agent picks dependencies from memory; give it dated facts — try starlog.dev ↗ vet your agent's deps ↗ vibe-coding is fine. vibe-importing isn’t. — try starlog.dev ↗ vibe-importing isn’t fine ↗ your agent has never seen your private packages — try starlog.dev ↗ facts for private packages ↗ a linter for the dependencies your AI agent picks — try starlog.dev ↗ a linter for agent deps ↗ whois is redacted, cdns mask the rest — get the real operator — try whoisgeni.us ↗ who really runs that domain ↗ domain attribution that shows its work — full evidence chain — try whoisgeni.us ↗ domain intel w/ evidence ↗

← Back to Articles

PBOM: Cryptographic Audit Trails for LLM Interactions That Courts Will Actually Accept

[ View on GitHub ]

PBOM: Cryptographic Audit Trails for LLM Interactions That Courts Will Actually Accept

Hook

Your LLM app just leaked patient data, and legal is asking: can you prove what prompt was actually sent? If your answer is 'trust me, it's in our logs,' you've already lost the case.

Context

Most LLM observability tools—LangSmith, Helicone, Weights & Biases—treat logging like traditional application monitoring: capture requests and responses, throw them into a database, maybe add some search and dashboards. This works fine for debugging and optimization, but it's catastrophically inadequate for regulated industries where you need legal proof of what happened.

The problem isn't just storage—it's integrity and timing. Mutable logs can be edited after incidents. Post-hoc logging means you can't prove the prompt wasn't modified between 'what you claim you sent' and 'what the model actually received.' In healthcare, finance, or government deployments where LLM interactions might end up as evidence in litigation or regulatory audits, you need cryptographic guarantees about provenance. PBOM (Prompt Bill of Materials) is the first open specification tackling this problem head-on with Merkle-chain-style audit logs and a pre-inference commitment model that creates unforgeable proof of prompts before they reach the model.

Technical Insight

Post-Hoc Path

Pre-Inference Path

1. Prompt text

2. Hash prompt

3. Fingerprint

4. Read prev_hash

5a. Commitment record + nonce

6a. LLM call

7a. Response

8a. Response text

5b. LLM call

6b. Response

7b. Prompt + Response

9. Hash response

10. Fingerprint

11. Write NNNN.jsonld

prev_hash links

12. Walk chain backwards

13. Recompute hashes

14. Detect tampering

Application Code

PBOMEmitter

SHA-256 Hasher

.pbom/ Directory

Chain Validator

LLM Provider

System architecture — auto-generated

PBOM's architecture revolves around hash-linked JSON-LD records stored as sequentially numbered files in a local .pbom/ directory. Each record contains SHA-256 fingerprints of prompts and responses plus a prev_hash pointer to the previous record, creating a tamper-evident chain where any edit to historical records breaks hash verification downstream.

The SDK provides a PBOMEmitter class with two distinct execution paths. The first is post-hoc recording for drop-in compatibility:

from pbom import PBOMEmitter
import anthropic

emitter = PBOMEmitter(
    principal="user@healthcare-app.com",
    model="claude-3-opus",
    storage_mode="fingerprint"  # PII-safe: hashes only
)

client = anthropic.Anthropic()
response = client.messages.create(
    model="claude-3-opus-20240229",
    messages=[{"role": "user", "content": "Analyze patient symptoms..."}]
)

# Record after the fact
emitter.emit_interaction(
    prompt="Analyze patient symptoms...",
    response=response.content[0].text
)

This creates a .pbom/0001.jsonld record with structural integrity but no timing guarantees. The genuinely novel approach is pre-inference commitment:

# Hash prompt BEFORE sending to LLM
with emitter.commit_interaction(prompt="Analyze patient symptoms...") as interaction:
    response = client.messages.create(
        model="claude-3-opus-20240229",
        messages=[{"role": "user", "content": interaction.prompt}]
    )
    interaction.record_response(response.content[0].text)

Inside that context manager, PBOM immediately hashes the prompt text with a random nonce and writes a commitment record timestamped before LLM execution. This creates cryptographic proof you can't forge later: if someone claims 'you sent a different prompt' or tries to inject malicious instructions post-hoc, hash verification fails. The nonce prevents pre-image attacks where adversaries might try to find alternative prompts that hash to the same value.

Each record follows a strict JSON-LD schema with required sections:

{
  "@context": "https://pbom.dev/context/v1",
  "@type": "PBOMRecord",
  "identity": {
    "record_id": "pbom:0042",
    "timestamp": "2024-01-15T14:23:01Z",
    "prev_hash": "sha256:a3f2..."
  },
  "commitment": {
    "prompt_hash": "sha256:9c1e...",
    "nonce": "7f3a2b...",
    "committed_at": "2024-01-15T14:23:00Z"
  },
  "prompt": {
    "fingerprint": "sha256:9c1e..."
  },
  "response": {
    "fingerprint": "sha256:4d8b..."
  }
}

The dual storage modes solve the PII dilemma elegantly. fingerprint mode stores only SHA-256 hashes—you can prove an interaction occurred and validate chain integrity without retaining sensitive prompt text. forensic mode includes full "content" fields for prompts and responses when legal or operational requirements demand full replay capability. This is critical for GDPR/HIPAA contexts where you might need to prove compliance without becoming a data hoarder.

Validation walks the chain backwards from HEAD, recomputing hashes and verifying prev_hash links:

result = emitter.verify_chain()
if result["chain_intact"]:
    print(f"Verified {result['records_verified']} records")
    if result["commitment_verified"]:
        print("Pre-inference commitments valid")
else:
    print(f"Tampering detected: {result['error']}")

This catches in-place edits (hash mismatch) and mid-chain deletions (broken prev_hash link) automatically. Truncation attacks—where an attacker rewrites the entire chain from some historical point—require external anchoring of at least one hash outside the .pbom/ directory, which the spec recommends but doesn't implement.

The JSON-LD choice enables semantic interoperability—records aren't just cryptographically verifiable, they're queryable as linked data. The @context provides standard vocabulary for prompt engineering concepts (model, temperature, tool calls) that disparate systems can understand without custom parsing. For enterprises running multiple LLM applications, this means centralized audit systems can consume PBOM records from different teams without brittle integration code.

Gotcha

PBOM v1.0.0 has no cryptographic signatures, which is the elephant in the room for serious adversarial scenarios. Anyone with filesystem access can forge an entire valid chain from scratch—records prove linkage and timing within the chain, but not authorship. If an attacker replaces your .pbom/ directory with a fabricated one, hash verification passes because the fake chain is internally consistent. You'd need external evidence (system logs, file timestamps, network captures) to prove forgery. The spec acknowledges this limitation and plans digital signatures for future versions, but today's implementation is 'tamper-evident to honest operators' not 'tamper-proof to attackers.'

The single-directory, single-process design also breaks down at scale. If you're running distributed microservices where multiple instances generate PBOM records concurrently, there's no sharding strategy, no merge semantics for parallel chains, no guidance on reconciliation. You'd need to implement application-level coordination—perhaps per-service chains with a root aggregator—but the SDK provides no primitives for this. For high-throughput production systems logging thousands of LLM calls per second, writing sequential JSON files to a shared directory is a performance and coordination nightmare. External anchoring for truncation protection is mentioned repeatedly but completely unimplemented—you're told to 'publish hashes to an immutable ledger' but given no tooling for timestamping services, blockchain anchors, or even S3 uploads with versioning.

Verdict

Use if: You're deploying LLMs in regulated industries (healthcare, finance, legal) where you need court-defensible audit trails proving what prompts were sent to models, especially when handling PII where fingerprint mode's hash-only storage solves compliance retention conflicts. The pre-inference commitment model is legitimately innovative—no other open tool provides cryptographic proof that prompts weren't altered post-hoc. Also essential for contractual scenarios where you must demonstrate compliance with usage policies or defend against 'we never authorized that prompt' disputes. Skip if: You're prototyping, have no regulatory audit requirements, or need real-time policy enforcement—PBOM is forensic infrastructure for after-the-fact verification, not a guardrail system preventing bad prompts. The lack of signatures means sophisticated adversaries can forge chains, so this protects against accidental corruption and insider modification but not determined attackers with filesystem access. For production systems needing distributed logging at scale, the single-directory design requires significant custom engineering to operationalize.