Gigatoken: Faster Tokenization for LLMs

Gigatoken has improved the speed of tokenization for modern Artificial Intelligence and LLM workloads. If you have at any point trained or fine-tuned a language model, you know that it does not process raw text.

Instead, the model breaks text into smaller pieces called tokens.

These tokens can be words, subwords, or even individual characters.

This process is called tokenization, and every language model relies on it before processing or learning from text.

In small scale work tokenization speed doesn’t really matter.

But in the case of large scale models which process hundreds of billions or even trillions of tokens that is no longer true.

At that scale, slow tokenization becomes a major bottleneck in the data pipeline.

The Gigatoken solves this problem by making tokenization significantly faster.

What is Gigatoken?

Gigatoken is an open-source tokenizer written in Rust with a Python API.

It works as a drop-in replacement for Hugging Face Tokenizers and OpenAI’s tiktoken.

The developers claim it runs up to 1,000× faster than existing tokenizers.

Why Tokenization Speed Matters

Tokenization can take hours or even days when you process massive datasets.

Gigatoken changes that.

On a high-core-count server CPU, it can tokenize a dataset the size of Common Crawl in under 6.5 hours.

Common Crawl contains about 130 trillion tokens and often represents a large portion of the public web.

Traditional tokenizers need much longer to finish the same job.

That extra time delays training, evaluation, and experimentation.

Even with smaller workloads, faster tokenization offers clear benefits:

• Spend less time and compute on data preprocessing.
• Experiment with new datasets more quickly.
• Start model training sooner.
• Speed up your overall machine learning pipeline.

Gigatoken Performance Benchmarks

The Gigatoken team benchmarked the tokenizer on an 11.9 GB OpenWebText training dataset.

The results show significant speed improvements over Hugging Face Tokenizers.

These benchmarks highlight how much faster Gigatoken can process large datasets on modern CPUs.

How Does Gigatoken Run So Fast?

Gigatoken combines several optimizations to achieve its impressive speed.

1. SIMD-optimized pre-tokenization

Most tokenizers use regular expressions to break up text into smaller elements.

Gigatoken does this using hand-optimized SIMD instructions.

SIMD allows the CPU to work on many bytes at a time.

It supports AVX-512 and AVX2 which in turn means it performs well on x86 and ARM processors.

2. Smart caching

Gigatoken stores the tokenized output of words it has already processed.

For the same word that shows up again, it uses the cached instead of the going through the process of tokenization again.

This approach saving a large time in terms of processing on big datasets.

3. Reduce Python effort

Gigatoken’s native API which has Rust reading data in directly.

This design reduces communication between Rust and Python.

It also avoids unnecessary object creation and thread overhead.

4. Low-level optimizations

Developers also did in depth optimization of the code.

They removed what was unnecessary and did code profiling to identify performance issues.

These improvements add a layer of speed to the earlier optimization efforts.

Getting Started with Gigatoken

Installing Gigatoken is straightforward:

pip install gigatoken

Use Compatibility Mode

If you are a current Hugging Face Tokenizers user, you may switch to Gigatoken with very little code change.

This mode gives you the same output as Hugging Face but we see much higher performance from it.

import gigatoken as gt

hf_tokenizer = ...
tokenizer = gt.Tokenizer(hf_tokenizer).as_hf()
tokens = tokenizer.encode_batch([
    "This is a test string",
    "And here is another"
])

Use the Native API

If you want the max performance we have in the bag, then use Gigatoken’s native API out there which has Rust read in the input files for you and also reduces Python overhead to a minimum.

import gigatoken as gt

tokenizer = gt.Tokenizer("Qwen/Qwen3-8B")
file_source = gt.TextFileSource(
    ["owt_train.txt"],
    separator=b"<|endoftext|>"
)
tokens = tokenizer.encode_files(file_source)

Benchmark It Yourself

You can also benchmark Gigatoken without installing it.

The following command compares its output with Hugging Face Tokenizers on a real dataset.

uvx --with tokenizers gigatoken bench 'openai-community/gpt2' owt_train.txt \
    --validate --doc-separator "<|endoftext|>"

Limitations of Gigatoken

Gigatoken does very well in terms of performance, at present it does have a few issues.

• It doesn’t support WordPiece tokenizers at all.
• Although it does support SentencePiece tokenizers which is a plus, these are not as efficient as BPE-based tokenizers.
• At present the native API is to return results in memory which is not the same as writing them out to files.
• Also we see that support for Windows is still very much in its infancy. We recommend that you use WSL if you are a Windows user.

Final Thoughts

Tokenization often becomes a hidden bottleneck when you train or fine-tune large language models.

Gigatoken addresses that problem with SIMD optimizations, smart caching, and lower Python overhead.

The best part is that you can adopt it without making major changes to your existing code.

If you process large amounts of text, Gigatoken is worth trying.

Run a quick benchmark on your own dataset and compare the results.

That will help you decide whether it fits your workflow.

Turn your AI ideas into successful products with Webkul.