Riot: A Go Search Engine That Treats Chinese Text as a First-Class Citizen

Hook

While Elasticsearch struggles with Chinese tokenization through clunky plugins, Riot ships with gse and sego tokenizers as core dependencies—making it one of the few Western search engines designed from the ground up for multilingual reality.

Context

The search engine landscape has long been dominated by Lucene-derived technologies—Elasticsearch, Solr, and their variants—all built on the JVM and optimized primarily for European languages. Chinese, Japanese, and Korean text presents unique challenges: no whitespace between words, context-dependent segmentation, and massive character sets. While Elasticsearch can handle CJK through analysis plugins, it's always an afterthought requiring extra configuration, memory overhead, and maintenance.

Riot emerged as a pure Go alternative that bakes Chinese text segmentation directly into its architecture. Rather than bolting on tokenization, it treats segmenters like gse and sego as first-class dependencies alongside its indexing layer. For Go shops building products for Asian markets—e-commerce platforms, content management systems, documentation portals—this means you can embed a sophisticated search engine without managing a separate JVM process, without hunting for the right analyzer plugins, and without the 2-3GB baseline memory footprint that comes with Elasticsearch. Riot targets the sweet spot between simple keyword matching and full Elasticsearch complexity: distributed sharding for horizontal scaling, BM25 relevance scoring, aggregations for faceted search, but stripped of cluster coordination overhead and query language complexity.

Technical Insight

Riot's architecture revolves around a pipeline model where documents flow through segmentation, indexing, and ranking stages using Go's channels and goroutines. Unlike traditional search engines that batch documents for efficiency, Riot embraces Go's lightweight concurrency model to index documents in parallel without complex thread pool management.

Here's a basic example showing how Riot handles document indexing with Chinese text:

import (
    "github.com/vcaesar/riot"
    "github.com/vcaesar/riot/types"
)

type Product struct {
    ID          uint64
    Title       string
    Description string
    Price       float64
}

func main() {
    // Initialize searcher with options
    searcher := riot.Engine{}
    searcher.Init(types.EngineOpts{
        Using:         4,           // number of shards
        SegmenterDict: "zh",        // Chinese dictionary
        IndexerOpts: &types.IndexerOpts{
            IndexType: types.DocIdsIndex,
        },
    })
    defer searcher.Close()

    // Index document with Chinese text
    product := Product{
        ID:          1,
        Title:       "无线蓝牙耳机",  // "Wireless Bluetooth Headphones"
        Description: "高品质音效，持久续航",  // "High quality sound, long battery"
        Price:       299.99,
    }

    searcher.Index(product.ID, types.DocData{
        Content: product.Title + " " + product.Description,
        Fields: map[string]interface{}{
            "price": product.Price,
        },
    })

    searcher.Flush() // Force index update

    // Search with Chinese query
    results := searcher.Search(types.SearchReq{
        Text: "蓝牙",  // "Bluetooth"
        RankOpts: &types.RankOpts{
            OutputOffset: 0,
            MaxOutputs:   10,
        },
    })
}

Under the hood, Riot's segmentation happens automatically. When you call Index(), the text passes through the configured segmenter (gse by default), which breaks Chinese characters into meaningful words using a dictionary-based approach combined with statistical analysis. The English equivalent would be turning "wirelessbluetoothheadphones" into ["wireless", "bluetooth", "headphones"], but with the added complexity that Chinese word boundaries aren't marked in the source text.

The indexing pipeline uses a producer-consumer pattern with buffered channels. Each shard runs its own goroutine pool, and documents are distributed via consistent hashing based on document ID. This means high-throughput indexing on multi-core systems without lock contention, but it also means you need to think about your ID space distribution—sequential IDs will hotspot on a single shard, while hashed IDs distribute evenly.

Riot's pluggable storage backend is where it gets interesting. The engine abstracts the index implementation behind an interface, allowing you to swap between in-memory indices for speed or bluge-backed persistent storage for durability:

searcher.Init(types.EngineOpts{
    Using:         4,
    SegmenterDict: "zh",
    IndexerOpts: &types.IndexerOpts{
        IndexType: types.DocIdsIndex,  // In-memory
    },
    UseStore: false,  // No persistence
})

// Or with persistent storage via bluge
searcher.Init(types.EngineOpts{
    Using:         4,
    SegmenterDict: "zh",
    IndexerOpts: &types.IndexerOpts{
        IndexType: types.DocIdsIndex,
    },
    UseStore: true,
    StoreFolder: "./riot_index",
})

When you enable persistent storage, Riot leverages bluge's immutable segment architecture. New documents write to an in-memory buffer that periodically flushes to disk as immutable segments. Searches query both the live buffer and disk segments, merging results. This is similar to Lucene's approach but implemented in pure Go with tighter memory controls.

The ranking layer implements BM25 with tunable parameters. Unlike black-box search engines, Riot exposes k1 (term frequency saturation) and b (length normalization) parameters, letting you optimize for your specific corpus characteristics. For product search where titles are short and uniform, you might decrease b to reduce length penalties. For long-form content with variable document sizes, increase b to favor comprehensive documents.

Aggregations in Riot borrow Elasticsearch's bucket model but implement it with Go's type system. You can build faceted search—group results by price range, category, or custom fields—but the API is less flexible than Elasticsearch's JSON-based aggregation DSL. The tradeoff is compile-time type safety and better performance, at the cost of requiring code changes for new aggregation types rather than runtime configuration.

Gotcha

Riot's distributed model is really "sharded" not "distributed" in the cloud-native sense. There's no built-in replication, no automatic failover, no cluster consensus protocol. If you configure 4 shards and one node goes down, you've lost 25% of your index. This works fine for embedded scenarios where the index lives in the same process as your application and durability comes from rebuilding from your source database. It breaks down for large-scale deployments where you expect the search cluster to be your source of truth with high availability guarantees.

The query parser is deliberately simple—basically prefix matching, phrase queries, and boolean AND/OR. You can't do fuzzy matching with edit distance controls, you can't boost specific fields at query time without re-indexing, and complex nested boolean expressions will require you to make multiple queries and merge results in application code. For 80% of use cases ("find products matching these keywords"), this is fine. For power-user advanced search features, you'll be fighting the framework.

Memory management for large indices lacks sophistication. There's no equivalent to Lucene's tiered merge policy, no fine-grained control over segment sizes, and no automatic cache eviction strategies beyond Go's garbage collector. With high-cardinality fields (millions of unique values), you can hit OOM issues. The documentation doesn't provide capacity planning guidelines—no guidance on documents-per-shard limits, memory-per-document estimates, or when to scale horizontally. You'll learn these limits through production incidents.

Verdict

Use if: You're building a Go application that needs embedded full-text search with Chinese/Japanese/Korean text as a primary requirement, you can tolerate eventual consistency and lack of replication, your queries are straightforward keyword matching rather than complex boolean logic, and you want sub-100ms indexing latency without managing a separate JVM process. Riot shines for product catalogs, internal documentation, log search, and content management systems where you control the deployment and can rebuild indices from source data. Skip if: You need production-grade distributed consensus with automatic failover, your users demand advanced query features like fuzzy matching or complex boolean expressions, you require operational maturity with monitoring dashboards and capacity planning tools, or you're searching unstructured data requiring advanced NLP. In those cases, bite the bullet and run Elasticsearch, or explore Meilisearch for simpler deployments with better out-of-box relevance tuning.