Inference-time alignment · Value-guided decoding
Inference-Time Language Model Alignment via Integrated Value Guidance
IVG combines fine-grained token steering with longer-horizon candidate selection, using two complementary value functions learned from the same preference data.
Research question
How can smaller preference models align a frozen target LLM at inference time—without updating its parameters?
Local control
Adjust each next-token distribution with a dense preference signal.
Long horizon
Compare partial responses after a chunk of generation, not only one token.
Frozen target
Guide much larger target models without task-specific fine-tuning.
Core method
One source of human preference information
The target language model remains frozen throughout generation.
Shared preference data
Both value functions are learned from the same comparisons but expose complementary signals.
Multi-model KV cache
Base, tuned/reference, and reward models maintain synchronized caches and beam states.
Accuracy vs. compute
Implicit guidance runs every token; explicit guidance scores only at chunk boundaries.
Key results
Length-controlled win-rate points on AlpacaEval 2.0.
Length-controlled win-rate points on AlpacaEval 2.0.
Small value models guide frozen targets up to 70B and MoE scale.
Controlled sentiment, summarization, and instruction following.
AlpacaEval 2.0 evaluation with GPT-4-Turbo judging.
Alignment is applied entirely at inference time.