Building a Personal Audiobook Pipeline with ebook2audiobook: Voice Cloning Meets 1158 Languages

Hook

While commercial audiobook services support maybe a dozen languages, this open-source pipeline handles 1,158—including endangered languages like Mixtec and Wayuu—using Meta's Fairseq models and voice cloning that can replicate your own voice from a 6-second sample.

Context

The audiobook market has a massive gap problem. If you read technical books, academic texts, or literature in less-common languages, chances are no commercial audiobook exists. Even for popular English titles, you're locked into subscription services, DRM restrictions, and narrators you didn't choose. Traditional TTS solutions like system voices or basic tools like Balabolka produce robotic output that's tolerable for accessibility but grating for leisure listening.

ebook2audiobook emerged from this frustration as a Python-based conversion pipeline that prioritizes flexibility and quality. By leveraging modern neural TTS engines—particularly Coqui's XTTSv2 with its voice cloning capabilities—it transforms personal, DRM-free ebooks into audiobooks with surprisingly natural narration. The project supports everything from EPUBs and PDFs to scanned images via OCR, packages output with proper m4b metadata and chapter markers, and offers deployment options ranging from local CLI tools to Docker containers and cloud notebooks. It's the kind of tool that couldn't have existed five years ago, built on the foundation of recent breakthroughs in neural voice synthesis.

Technical Insight

At its core, ebook2audiobook is a multi-stage pipeline: extraction, preprocessing, TTS conversion, and audiobook packaging. The architecture is modular enough to swap TTS backends while maintaining a consistent interface through both CLI and Gradio-based web UI.

The extraction phase handles heterogeneous input formats through a unified abstraction layer. For EPUBs, it uses ebooklib to parse the XHTML structure and extract text while preserving chapter boundaries. PDFs go through PyPDF2 for text-based files, but the real power comes from OCR integration via Tesseract for scanned documents. This means image-heavy books or poorly digitized academic papers become candidates for conversion, though OCR quality directly impacts TTS output.

Here's a simplified view of how the TTS conversion loop works with XTTSv2:

import torch
from TTS.api import TTS

# Initialize XTTSv2 with device selection
device = "cuda" if torch.cuda.is_available() else "cpu"
tts = TTS("tts_models/multilingual/multi-dataset/xtts_v2").to(device)

# Voice cloning from reference audio
speaker_wav = "voice_sample.wav"  # 6-second minimum recommended

# Process chapters with voice cloning
for idx, chapter_text in enumerate(chapters):
    output_path = f"chapter_{idx:03d}.wav"
    
    # XTTSv2 generates audio with speaker conditioning
    tts.tts_to_file(
        text=chapter_text,
        speaker_wav=speaker_wav,
        language="en",  # One of 17+ languages for XTTSv2
        file_path=output_path
    )

The voice cloning workflow is deceptively simple from the API perspective, but underneath, XTTSv2 uses a speaker encoder that extracts embeddings from your reference audio. These embeddings condition the Transformer-based TTS model during inference, effectively transferring prosody and timbre characteristics. The quality depends heavily on reference audio quality—clean recording, minimal background noise, consistent tone. The repository includes a curated collection of community-contributed voice presets, which is crucial since not everyone has studio-quality recordings of themselves.

For the multilingual support that extends to 1,158 languages, the tool integrates Meta's Fairseq MMS (Massively Multilingual Speech) models. The architecture switches TTS backends based on language selection:

def select_tts_engine(language_code):
    # XTTSv2 for high-quality subset
    xtts_supported = ['en', 'es', 'fr', 'de', 'it', 'pt', 'pl', 
                      'tr', 'ru', 'nl', 'cs', 'ar', 'zh-cn', 
                      'ja', 'hu', 'ko', 'hi']
    
    if language_code in xtts_supported:
        return XTTSEngine()
    
    # Fairseq MMS for rare languages
    return FairseqMMS(language_code)

The Fairseq integration sacrifices some naturalness for breadth—voices sound less dynamic than XTTSv2 but remain intelligible across languages from Quechua to Welsh. This trade-off reflects a fundamental tension in TTS: models trained on massive multilingual datasets spread their capacity thin, while focused models like XTTSv2 achieve better quality on fewer languages.

One particularly clever feature is Speech Markup Language (SML) tag support for controlling narration. You can embed tags directly in your ebook text for fine-grained control:

<voice="narrator">The detective entered the room.</voice>
<voice="character_1"><rate speed="fast">Quick! He's getting away!</rate></voice>
<break time="1s"/>
<voice="narrator">She paused, considering her options.</voice>

This allows multi-voice audiobooks from a single conversion run, though it requires manual text annotation. The preprocessing pipeline strips most HTML/EPUB formatting but preserves these SML tags, passing them through to TTS engines that support them.

The packaging stage uses ffmpeg to concatenate audio chunks, embed metadata (title, author, chapter markers), and output to m4b format—the audiobook standard that iTunes and most mobile apps recognize. Chapter timing is calculated from the cumulative audio length, creating a seekable audiobook experience rather than a monolithic audio file.

Gotcha

The biggest pain point is EPUB format fragmentation. Unlike PDF with its structural consistency, EPUB is essentially a zipped collection of XHTML files with a loose specification. Publishers implement chapter breaks differently—sometimes as separate files, sometimes as divs within a single file, occasionally not at all. This means the extraction phase often pulls in headers, footers, page numbers, and copyright notices that you absolutely don't want narrated. You'll find yourself manually cleaning extracted text in a majority of conversions, especially with older or poorly formatted ebooks. The repository doesn't include intelligent heuristics for filtering this noise because there's no universal pattern.

Processing time is another reality check. On CPU-only systems, XTTSv2 processes text at roughly 0.1-0.2x real-time—meaning a 10-hour audiobook takes 50-100 hours to generate. Even with GPU acceleration (tested on an RTX 3060), you're looking at 1-2x real-time for quality output. The faster TTS engines like Piper sacrifice voice cloning and some naturalness for 5-10x real-time speeds, but then you lose the tool's primary differentiator. There's no magic solution here; neural TTS is computationally expensive, and this tool doesn't hide that fact.

Voice cloning quality varies wildly based on source material. The 6-second minimum is optimistic—you'll get better results with 30-60 seconds of clean, expressive speech. Monotone reference audio produces monotone output. Background noise introduces artifacts. And certain voice characteristics (heavy accents, speech impediments, extreme pitch ranges) confuse the speaker encoder, producing uncanny valley results. The documentation downplays this learning curve.

Verdict

Use ebook2audiobook if you have a backlog of DRM-free technical books or non-English literature that will never get commercial audiobook releases, if you need accessibility features with specific voice preferences that system TTS can't provide, if you're willing to invest time in text cleanup and have GPU access (or extreme patience), or if you're building a custom audiobook pipeline and need a proven open-source foundation to extend. Skip it if you need production-ready audiobooks without manual intervention (commercial services will be faster and require less babysitting), if you're working with DRM-protected content (this tool can't and won't bypass restrictions), if your hardware situation is CPU-only and you need results this decade, or if you expect studio-narrator quality that matches Audible's professional recordings—neural TTS is impressive but still detectably synthetic. For developers building voice applications or researchers working with low-resource languages, this is a goldmine. For casual readers wanting to convert their Kindle library, set expectations accordingly.