Accelerating Qwen3-8B Agent on Intel® Core™ Ultra with Depth-Pruned Draft Models

The Qwen3-8B agent model can be accelerated on Intel Core Ultra processors using speculative decoding. This method utilizes a smaller, faster draft model, Qwen3-0.6B, to propose tokens that the larger model then validates, achieving a 1.3x speedup with OpenVINO.GenAI. To push performance further, the draft model was optimized through depth-pruning, where less impactful layers were removed and the model was fine-tuned, resulting in a 1.4x speedup. This optimized model pairing was then integrated with the 🤗smolagents library to demonstrate a fast, local AI agent capable of using tools and executing code. The approach showcases how model pruning and speculative decoding can make complex agentic workflows more practical on consumer hardware.