How Specializing the Vocabulary of SLMs Can Cut Inference Cost by 77% in Real Applications

Specializing a small language model for a specific domain often leaves the inefficient, general-purpose vocabulary intact. This results in higher inference costs due to token fragmentation of common phrases and the computational overhead of scoring a large, mostly unused vocabulary. The cost of generating an output token is significantly higher than processing an input token, making output optimization crucial. A method called VocabX rebuilds the vocabulary for the specific task, drastically reducing the number of generated tokens and overall vocabulary size, which cut inference costs by 77% in a real-world essay grading application.