Tokenization in Transformers v5: Simpler, Clearer, and More Modular

Transformers v5 redesigns how tokenizers work by separating the tokenizer's architecture from its trained vocabulary, similar to how model architecture is separate from learned weights. This update makes the process more modular, inspectable, and easier to customize or train from scratch. The tokenization pipeline, which converts raw text into integer IDs for models, involves distinct stages like normalization, pre-tokenization, and post-processing. This new approach aims to provide a clearer and more flexible system for handling text data in language models, moving away from the previous opaque implementation.