GigaToken: a ~1000× faster tokenizer for LLM, open source

Marcel Roed publishes *GigaToken*, a tokenizer for language models that claims a performance improvement of approximately three orders of magnitude over reference implementations. A look at what has been optimized.

The concrete case

You pre-train a language model, or you need to encode a dataset of several terabytes of text for fine-tuning. The tokenizer - this building block that transforms raw text into numerical identifiers - is often the invisible bottleneck: on reference implementations, it can consume as much time as the model's forward pass itself on massive corpora.

GigaToken, published by Marcel Roed on GitHub, claims ~1000× the reference implementation on usual BPE (Byte-Pair Encoding) tokenizers. This is a figure to be taken seriously: even within an order of magnitude, it changes what is possible on a single machine.

Under the hood: where are the 1000×?

A BPE tokenizer does three things:

1. Pre-tokenization - coarse splitting of text (e.g., on spaces, punctuation, Unicode categories).
2. Byte-pair merges - iterative application of merge rules (e.g., t + h → th, then th + e → the) according to a pre-learned vocabulary.
3. Encoding - production of the sequence of IDs.

The massive gains come from several levers, combined:

• Merge algorithm: naive implementations (Hugging Face tokenizers in pure Python, or the first tiktoken) go through a list of rules at each step. Optimized implementations (like tiktoken in Rust) use a pre-computed merge tree and a priority queue to only process active pairs. GigaToken takes the logic further.
• Vectorization: processing several bytes simultaneously via SIMD for pre-tokenization (boundary detection, membership tests in character classes).
• Parallelism: processing text chunks in parallel on all cores, properly managing boundaries between chunks.
• Cache-friendly structures: packed representations of the vocabulary that fit in the L1/L2 cache, avoiding cache misses that dominate the cost on naive implementations.

Why it matters

Tokenizers are not glamorous. They go unnoticed in papers, and most people assume they run in O(free). But when talking about preparing training corpora on the scale of several thousand billion tokens (like Llama 3, DeepSeek V3, etc.), even a 10× factor on the tokenizer translates into weeks of compute saved.

Concrete cases where the gain matters:

• Pre-training an LLM (data preparation).
• Large-scale RAG ingestion (millions of documents to tokenize for indexing).
• Low-latency token streaming - a fast tokenizer contributes to the perceived latency on the client side of a chatbot.
• Constrained environments - edge inference, browser (via WebAssembly), embedded.

Usage warnings

A faster tokenizer does not replace a correct tokenizer. Points to check before adopting:

• Byte-exact compatibility with the target model's vocabulary: a different ID = a bad token = a model that hallucinates. It must be validated on a large corpus.
• Unicode handling: edge cases (grapheme clusters, homoglyphs, NFC/NFD normalization) are a minefield.
• Stable API: an experimental project may disappear or change its surface from one day to the next.
• Security: a fast parser in systems language must have been audited (overflows on pathological inputs).

Our take

The open-source LLM ecosystem is maturing its "plumbing" layer - after inference servers (vLLM, TGI, llama.cpp), after training frameworks (Megatron, DeepSpeed), it's the turn of the data bricks to be rewritten at the performance level. GigaToken fits into this movement, alongside tiktoken-rs, arrow-rs, and the systematic rewrites in Rust or C++ of what used to run in Python before.

Key takeaways

• GigaToken claims ~1000× the naive implementation of BPE - even within an order of magnitude, it's significant.
• Typical gains: SIMD, parallelism, cache-friendly structures, optimized merge algorithm.
• Must be validated byte-exact against the target model's vocabulary before any production use.