Building an AI Video Pipeline: How video_explainer Orchestrates TTS-First Programmatic Animation

Hook

Most AI video tools generate visuals first, then awkwardly sync audio. video_explainer does the opposite—creating narration with word-level timestamps before a single frame exists—and the results are surprisingly professional.

Context

Creating technical explainer videos is brutally time-consuming. A five-minute video explaining a research paper or technical concept can take days of work: scripting, recording voiceover, creating animations in After Effects or similar tools, syncing everything frame-by-frame, and iterating when changes are needed. Template-based tools like Canva or Descript help, but they're still fundamentally manual—you're dragging elements around a timeline.

The rise of large language models promised automation, but most AI video tools took the wrong approach. They generate static visuals or avatar-based presentations that feel robotic. video_explainer takes a different path: it treats video generation as a code generation problem. Instead of manipulating templates, it writes React components that programmatically animate using Remotion, the same framework Netflix uses for server-side video rendering. The result is a fully automated pipeline that turns Markdown files, PDFs, or URLs into polished explainer videos with synchronized narration, animations, and even background music—no timeline scrubbing required.

Technical Insight

The architecture makes a counterintuitive choice that defines everything downstream: it generates text-to-speech audio before creating any visuals. Most developers would assume you need to know what's on screen before deciding what to say, but video_explainer inverts this. Here's why it matters.

When you call ElevenLabs or Edge TTS APIs, modern services return not just an audio file but word-level timestamps—precise millisecond markers for when each word is spoken. By generating TTS first, video_explainer captures these timestamps and uses them to calculate exactly how long each scene needs to be. A scene explaining "neural network backpropagation" might need 8.4 seconds because that's how long the narration takes. This timestamp data becomes the ground truth for everything else.

The workflow looks like this: First, the document parser (supporting Markdown, PDF, or web scraping) extracts content. Then Claude receives a prompt to break this into a structured script with scene divisions and visual cues. Here's an example of what that intermediate format looks like:

# Generated script structure
{
  "scenes": [
    {
      "narration": "Let's start with how transformers process input tokens.",
      "visual_cues": "Show token embeddings flowing into attention mechanism",
      "duration_ms": 4200,  # Calculated from TTS timestamps
      "word_timestamps": [
        {"word": "Let's", "start": 0, "end": 180},
        {"word": "start", "start": 180, "end": 420},
        # ... more timestamps
      ]
    }
  ]
}

Now comes the novel part: Claude generates actual React components for each scene. Not templates—custom code. The prompt engineering here is critical. video_explainer sends Claude the narration, visual cues, word timestamps, and examples of Remotion animation patterns. Claude outputs something like this:

import {useCurrentFrame, useVideoConfig, spring} from 'remotion';

export const TokenFlowScene = () => {
  const frame = useCurrentFrame();
  const {fps} = useVideoConfig();
  
  // Sync animation to word "attention" at 2.1s
  const attentionHighlight = spring({
    frame: frame - (2.1 * fps),
    fps,
    config: {damping: 200}
  });
  
  return (
    <div style={{flex: 1, background: '#1a1a1a'}}>
      <svg viewBox="0 0 1920 1080">
        {/* Token boxes */}
        {['The', 'cat', 'sat'].map((token, i) => (
          <rect
            key={i}
            x={300 + i * 200}
            y={400}
            width={150}
            height={100}
            fill="#3b82f6"
            opacity={frame > i * 30 ? 1 : 0}
          />
        ))}
        {/* Attention arrow appears at exact word */}
        <path
          d="M 600 600 L 800 800"
          stroke="#ef4444"
          strokeWidth={4}
          opacity={attentionHighlight}
        />
      </svg>
    </div>
  );
};

This is production code that Remotion can render. The beauty is that every visual element is timed to narration through the frame variable. When the narrator says "attention" at 2.1 seconds, the highlight arrow animates in. No manual keyframing.

The system includes a four-phase refinement loop that's worth understanding. After generating initial scenes, video_explainer asks Claude to review its own output against the original document, checking for technical accuracy and visual coherence. If a scene misrepresents a concept or the animation doesn't match the narration, Claude regenerates that specific component. This self-correction loop catches errors before rendering—crucial because video rendering is expensive (both computationally and via API costs).

The dual-runtime architecture separates concerns intelligently. Python handles orchestration: document parsing, LLM prompts, TTS API calls, project management. Node.js handles only rendering: Remotion takes the generated React components and outputs MP4 files. This separation means you can iterate on the AI pipeline without touching the rendering engine, and vice versa. Projects are self-contained directories with a predictable structure:

project_name/
├── config.json          # Project settings, API keys
├── scripts/             # Generated scene scripts
├── narrations/          # Audio files with timestamps
├── components/          # Generated React components
├── audio/               # Final mixed audio
└── output/              # Rendered videos

The sound design system deserves mention. Beyond narration, video_explainer includes automated SFX detection (Claude identifies moments that need sound effects like "whoosh" or "click"), integrates MusicGen for background music generation, and handles audio mixing to balance narration, effects, and music. The final audio isn't just TTS—it's a properly mixed soundtrack.

One clever detail: the vertical video generator repurposes the entire pipeline for 1080x1920 shorts with TikTok-style animated captions. Same React components, different aspect ratio and overlay rendering. This means one project configuration can output both YouTube landscape videos and vertical social media content without rebuilding the pipeline.

Gotcha

The dependency chain is genuinely complex. You need Python 3.8+, Node.js 16+, FFmpeg with specific codec support, and correct environment configuration for both runtimes to communicate. Setup isn't npm install and go—expect to troubleshoot path issues, dependency conflicts between Python packages, and Node module resolution. The README provides instructions, but first-time setup can easily take an hour if you hit edge cases.

API costs are the silent budget killer. ElevenLabs charges per character—a 10-minute video might cost $2-5 in TTS alone. Claude API calls for scene generation and refinement add up quickly if you're iterating on content. For a single video, it's reasonable. For a content production pipeline generating dozens of videos monthly, you're looking at hundreds in API costs. The free Edge TTS option exists but produces noticeably lower-quality audio that sounds robotic compared to ElevenLabs' natural intonation. There's a real quality-versus-cost tradeoff here.

The programmatic animation constraint is both a strength and limitation. video_explainer excels at diagrams, text animations, simple shapes—anything you can code in SVG or CSS. It cannot incorporate stock footage, live-action clips, or complex particle effects you'd create in After Effects. If your explainer needs to show actual product screenshots transitioning smoothly or realistic physics simulations, you'll need to manually create those React components or accept simpler visualizations. The AI generates reasonable code for common technical visuals, but it's not replacing a motion graphics artist for complex work.

Verdict

Use if: You're producing regular technical content (research summaries, documentation explainers, educational videos) where programmatic animations work well, you have budget for quality TTS and LLM APIs, and you value automation over pixel-perfect creative control. This shines for scaling video production—turning 10 blog posts into 10 videos would be days of manual work but hours with this pipeline. Also use if you're comfortable debugging Python/Node.js environments and want to customize the generation prompts or add new scene types. Skip if: You need to integrate custom footage or complex motion graphics, you're cost-sensitive and can't justify API expenses, or you want a simpler tool with a GUI rather than a code-based pipeline. Also skip if you're creating one-off videos where manual editing in Descript or similar tools would be faster than configuring this entire system. This is infrastructure for content pipelines, not a quick video editor.