Zhixuan Liu

NeurIPS 2026 Main Track

Subliminal Learning
as Trait-Direction Drift

A Mechanism and Targeted Control under SFT Distillation

Zhixuan Liu1,3,*,†Zhichen Dong1,3,*Yuyu Fan2,3Xiangtian Li3Chao Yang3

1 Shanghai Jiao Tong University2 Fudan University3 Shanghai Artificial Intelligence Laboratory

* Equal contribution † Corresponding author

Read the paper ↗ OpenReview Poster PDF BibTeX

Can a preference hide
in a sequence of numbers?

A teacher prompted to love owls generates number-only data. After supervised fine-tuning on those numbers, a student can inherit the owl preference.

We study how this signal accumulates as trait-direction drift, and reduce transfer by constraining the student’s drift during distillation.

Mechanism

From teacher bias to student behavior

Figure 1. A system prompt creates an owl-preferring teacher. Trait-irrelevant numeric outputs carry preference gaps. Student SFT accumulates drift along a trait direction, followed by an owl preference at evaluation.
Figure 1 from the paper. Same-model distillation under a local low-rank approximation.View figure PDF ↗

A measurable preference gap

The gap compares a sequence’s log-likelihood under a biased prompt with its average log-likelihood under neutral prompts. Individual gaps can have either sign.

A signal that can accumulate

Student-recognizable gaps contribute a trait-aligned signal. Under the local approximation, positive drift follows when the accumulated signal exceeds residual fluctuations.

Targeted control

Keep student drift inside a calibrated corridor

Probe-space corridor regularization adds a penalty when centered drift leaves a band around the pre-SFT base. The bounds are calibrated from baseline probe variation.

Qwen owl and malicious-response training curves. Standard SFT shows increasing centered probe drift, while corridor regularization keeps mean drift near the pre-SFT base.
Qwen2.5-7B-Instruct. Means and sample standard deviations over three training seeds. Dashed lines mark the corridor bounds.

Less behavioral transfer after distillation

Same-model teacher–student pairs. Each pair of bars uses a common 0–35% scale.

Standard SFTCorridor

Qwen · owl

Task accuracy −0.08 pp
Standard SFT30.7 ± 15.7%
Corridor1.7 ± 0.1%

Qwen · malicious response

Task accuracy +0.00 pp
Standard SFT29.5 ± 2.4%
Corridor6.4 ± 5.3%

Llama · owl

Task accuracy −4.34 pp
Standard SFT17.8 ± 22.5%
Corridor0.6 ± 0.2%
Bar length shows mean transfer. Labels give mean ± sample SD. Task accuracy changes are corridor minus standard SFT, in percentage points. Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct. Values follow the precision of the paper’s main results table.
How the corridor penalty works

The objective combines the SFT loss with two one-sided squared penalties.

ℒ = ℒCE + λ ([δĉ − τhigh]+2 + [τlow − δĉ]+2)

Here δĉ is the probe-based centered drift and [a]+ = max(0, a). The lower and upper bounds define the allowed corridor.

Mechanism evidence

Local updates and accumulated drift tell different parts of the story

Local gap–update correlations are modest and depend on the student state. In a separate experiment, bin-mean gaps correlate strongly with accumulated drift after 200 steps.

Accumulated drift tracks the gap

Ten independent runs, one per gap bin. These L3 correlations use a different protocol from the local-update measurements.

Pearson correlation with 200-step centered drift
StudentOwlMalicious response
Qwen0.8350.845
Llama0.8050.769
Explore the local-update evidence
Four model–trait settings show predicted L1 and measured L2 local correlations tracking each other as training progresses, with modest and state-dependent correlation magnitudes.
Fixed 1,000-sample subsets. Shading denotes pointwise 95% paired-bootstrap confidence intervals. L1 is the predicted one-step effect, and L2 is the measured one-step change.

What the evidence covers

The mechanism describes aggregate trajectories. Individual examples can deviate from the aggregate pattern. Cross-model transfer is weaker under matched budgets.

Corridor regularization targets specified trait coordinates. Accuracy costs vary by model and task, with 3.50–4.34 percentage-point costs on the evaluated Llama animal tasks. Unknown traits, probe misspecification, adaptive evasion, and escape during longer training remain open questions.

Paper & presentation

Read, present, or cite this work

Conference paperOpenReview PDF ↗ PreprintarXiv 2609.01091 ↗ Conference poster140 × 90 cm PDF ↓ Poster image4480 × 2880 PNG ↓

Citation

Download .bib

arXiv citation with the NeurIPS 2026 acceptance note.

@article{liu2026subliminal,
  title   = {Subliminal Learning as Trait-Direction Drift: A Mechanism and Targeted Control under {SFT} Distillation},
  author  = {Zhixuan Liu and Zhichen Dong and Yuyu Fan and Xiangtian Li and Chao Yang},
  journal = {arXiv preprint arXiv:2609.01091},
  note    = {Accepted to NeurIPS 2026 Main Track as Poster},
  year    = {2026},
  doi     = {10.48550/arXiv.2609.01091},
  url     = {https://arxiv.org/abs/2609.01091}
}