Adaptive (Self-)Speculative Decoding via Bandit Algorithms

Abstract

Speculative decoding accelerates language model inference while preserving the target model’s output distribution, but its efficiency depends on choosing the right configuration for each prompt.

In this talk, I will first introduce BanditSpec, a training-free framework that formulates adaptive configuration selection as a multi-armed bandit problem. Through two algorithms, UCBSpec and EXP3Spec, we address stochastic and adversarial settings and establish guarantees on stopping-time regret, which measures the additional decoding rounds relative to the best fixed configuration.

I will then present SP²EC, which extends this perspective to a more practical cost-aware self-speculative decoding. Retaining more layers or tokens can improve draft quality while increasing computational cost, making wall-clock latency the central objective. By exploiting a single-peak structure in throughput across ordered draft configurations, SP²EC efficiently adapts to each prompt and achieves wall-time regret guarantees whose leading term depends only on configurations neighboring the optimum. Experiments across language and vision-language tasks demonstrate the benefits of online adaptation, with SP²EC improving throughput by up to 11% over the underlying self-speculative methods. Together, these works show how bandit algorithms provide a principled foundation for adaptive and efficient inference.

About the speaker

Dr. Yunlong Hou is a Research Fellow in the Department of Mathematics at the National University of Singapore (NUS), where he also obtained his Ph.D. under the supervision of Prof. Vincent Y. F. Tan. Prior to that, he received his B.S. in Mathematics from Beijing Normal University. His research interests lie at the intersection of reinforcement learning, online learning, and large language models (LLMs). In particular, he focuses on developing efficient and practical algorithms for LLMs with provable guarantees.