Retrieval-Preserving Hybrid Sequence Models: A Critical Survey and Research Agenda on Memory Allocation, Alignment, and Adaptive State

The Hong Kong University of Science and Technology (Guangzhou)

数据科学与分析学域

PhD Qualifying Examination

By Mr. HU, Zhuangfei

摘要

Efficient long-context sequence models are increasingly memory-allocation systems. This survey studies retrieval-preserving hybrid memory allocation, where modern sequence models coordinate several memory forms: explicit Transformer key–value evidence, compressed recurrent or state-space state, sparse or local access bandwidth, online-updated associative memory, and external or archival stores. The central distinction is between context access, where a model must retrieve specific prior evidence, and state evolution, where it must update a compact latent description over time. This distinction makes retrieval preservation a technical constraint rather than only a benchmark preference. Hybrid architectures and transfer/alignment methods are surveyed as attempts to allocate memory functions across layers, heads, routes, objectives, and serving-time cache policies. Our preliminary work, AnchorPACT, studies post-construction mismatch in sparse Transformer–SSM conversion and provides controlled cross-probe evidence that scaffold selection must be complemented by alignment contracts for retained anchors, replacement bodies, and residual pathways. The survey then identifies three connected open problems that shape our research agenda: retrieval-anchor identification, retrieval-aware residual flow, and adaptive active-state expansion.

PQE Committee

• Chair: Prof. TANG, Nan
• Prime Supervisor: Prof. ZHANG, Yongqi
• Co-Supervisor: Prof. LI, Jia
• Examiner: Prof. ZHANG, Yanlin