tokenizers v1: encode, decode and scaling, measured

The upcoming v1 of the `tokenizers` library introduces major performance enhancements to prevent it from becoming a bottleneck in ML pipelines as models get faster. Key optimizations include replacing the regex splitter with a faster, SIMD-accelerated bitstream method and implementing a thread-local word cache to memoize results for repeated pre-tokens. The BPE merge loop has also been rewritten to operate more efficiently within a preallocated buffer, avoiding memory allocation during merges. These changes collectively lead to substantial speed improvements across various models and workloads, ensuring GPUs are not left idle waiting for CPU tokenization.