A Survey of Performance Prediction for LLM Fine-tuning:From Early Stopping to Decision-oriented Post-training

The Hong Kong University of Science and Technology (Guangzhou)

数据科学与分析学域

PhD Qualifying Examination

By Mr. LUO, Yuxiang

摘要

Machine learning training has long been an expensive and uncertain investment: substantial computation is often required before one can determine whether a model, dataset, hyperparameter configuration, or training strategy is worth pursuing. Early stopping and learning-curve extrapolation were among the earliest responses to this problem, using partial training trajectories to avoid unnecessary computation or predict final performance. This idea later expanded into multi-fidelity optimization, neural architecture search, proxy-based evaluation, and dataset-level quality assessment. With the rise of foundation models, especially large language model fine-tuning, the problem has become more important and more complex, as final performance depends on the interaction among the base model, training data, objective, optimizer, and evaluation metric.

This survey reviews the development of performance prediction for machine learning training, from early stopping to foundation model fine-tuning. It organizes existing methods under a unified information-and-decision perspective, comparing curve-based, multi-fidelity, proxy-based, data-quality-based, early-dynamics-based, and hybrid diagnostic approaches. Beyond summarizing methods, the survey discusses why prediction should be understood as partial information acquisition under computational constraints, where some uncertainty is reducible through additional observation while some may be intrinsic to the task. This perspective motivates a broader view of predictability, decision-sensitive evaluation, and future directions such as relative data valuation, few-shot prediction without heavy meta-learning, agentic training-decision systems, and policy-level resource allocation under uncertainty.

PQE Committee

Chair: Prof. CHU, Xiaowen

Prime Supervisor: Prof. TANG, Nan

Co-Supervisor: Prof. LUO, Yuyu

Examiner: Prof. TANG, Jing