Gigatoken

Training and data prep still choke on tokenization long after the model code looks fast. Gigatoken is a Rust tokenizer that hits GB/s throughput with HF and tiktoken compatibility modes, plus a native API that skips Python overhead for file-scale jobs. On big CPUs the reported gap versus Hugging Face is on the order of a thousand times for the same outputs. The useful shift is making the boring preprocess step stop being the wall for large corpora.