BPM

Preprint · 2026

Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization

Hao Wang^1,2, Kun Yuan^†2, Wenlin Zhong^2,3, Minglei Zhang², Han Xiao⁴, Ming Sun², Honggang Qi^§1

¹ University of Chinese Academy of Sciences ² KwaiKAT Team, Kuaishou Technology ³ Zhejiang University ⁴ The Chinese University of Hong Kong ^†Project lead ^§Corresponding author

Exact 99% of positions (>99%) equal the teacher's exact byte-prefix marginal

Closed 43.9% of the teacher–student gap (best pair)

pass@8 60.9% six-benchmark avg on the best pair, +8.5 over its best baseline

Pairs 100% teacher pairs won (3/3), math and code

📄 Paper 💻 Code In 1–2 Months 🤗 Models 🗃️ Datasets

Abstract

Open-weight language models from different families exhibit complementary capabilities, motivating their consolidation into a compact student through on-policy distillation (OPD). However, full-vocabulary OPD typically assumes a shared tokenizer, while existing cross-tokenizer methods may discard teacher probability mass or assign it to student tokens with unrelated content. We introduce Byte-Prefix Marginalization (BPM), which re-expresses the teacher’s next-token distribution over the student vocabulary in a shared byte space. Specifically, BPM assigns each teacher token’s probability to the longest student token whose byte representation is a prefix of the teacher token’s bytes, aggregates mass mapped to the same student token, and places otherwise unmatched mass in an explicit residual category. This produces a vocabulary-complete, byte-aligned, and mass-preserving target for dense OPD. The target exactly recovers the teacher-induced byte-prefix marginal when the relevant prefix does not span multiple teacher tokens (a condition satisfied at more than 99% of training positions) and uses a mass-preserving, chain-factorized lower bound otherwise. Across Qwen3-32B, GLM-Z1-9B-0414, and MiniMax-M2.7 as teachers, BPM consistently outperforms current cross-tokenizer methods on six mathematics and programming benchmarks, improving six-benchmark avg@8 by 3.7–6.6 points over the strongest baselines.

BPM teaser: routing teacher probability mass into the student vocabulary through shared bytes

Tokens are dialects.
Bytes are the common tongue.

BPM routes the teacher's probability through the byte stream both models share — supervision survives the change of vocabulary.

Contributions

Key Contributions

Full-Vocabulary, Identity-Preserving Target

Every unit of teacher probability is routed by bytes or parked in an explicit residual — exact on 99%+ of positions, mass conserved.

Wins on Every Pair

Beats overlap, rank, hybrid, and sequence-level baselines on all three teacher pairs — including contests held after every teacher's cutoff.

A Failure Mode of Byte-Exact Fidelity

Whitespace-only code rows transfer pure segmentation noise (49%→9% pass collapse); a precomputed row mask removes it.

Method

From Rollout to Target

01

Student-Owned On-Policy Rollouts

The student samples under its own template and decode settings — supervision lands where it acts.

02

Teacher Re-Scoring in Its Own Template

The same bytes, re-rendered in the teacher's template, scored into full-vocabulary distributions.

03

Byte-Prefix Marginalization

Teacher mass routes to the longest student token prefixing its bytes; the rest to an explicit residual.

04

β-Divergence Distillation

One β axis spans FKL / JSD / RKL against the projected target; a whitespace-row mask guards code.

Pipeline

Method Overview

Method Insight

One Ribbon, Two Readings

Two tokenizers read the same bytes and cut them differently — in exactly four ways. Follow one strip of text through all four; hover a case to isolate it.

Precomputed Routing The byte-alignment table is built once per tokenizer pair, before training starts.

teacher bytes student 1:11:nn:1m:n·solve·therefore</think>·tokenizers·solve·therefore</think>·tokenizers↓ 1.00↘ longest prefix · rest next⊕ .70×.90×.95 = .60prefix + residual ∅ ·solve·therefore</think>·tokenizers

Benchmark Results

Results on Six Benchmarks

Student: Qwen3.5-2B (thinking mode). Cells are six-benchmark averages, avg@8 / pass@8 (%). P1 = Qwen3-32B (same family), P2 = GLM-Z1-9B, P3 = MiniMax-M2.7 (least overlap); span Jaccard 0.92 / 0.87 / 0.81. Subscripts denote BPM's margin over each pair's best baseline.

Method	P1 · Qwen3-32B		P2 · GLM-Z1-9B		P3 · MiniMax-M2.7
Method	avg@8	pass@8	avg@8	pass@8	avg@8	pass@8
Teacher	67.4%	80.3%	63.0%	76.7%	65.4%	80.5%
Base Student	24.7%	40.5%	24.7%	40.5%	24.7%	40.5%
SimCT	35.3%	52.4%	26.3%	45.0%	25.0%	37.7%
ULD	25.1%	41.4%	34.2%	50.0%	32.8%	48.0%
GOLD	29.7%	47.3%	33.0%	51.9%	27.5%	42.5%
SeqKD	33.8%	52.4%	34.9%	53.9%	26.4%	45.5%
BPM (Ours)	39.0%_+3.7	60.9%_+8.5	41.5%_+6.6	59.4%_+5.5	38.7%_+5.9	57.1%_+9.1

Tokenizer similarity spectrum of the three teacher-student pairs — Three pairs, one axis: tokenizer distance (span Jaccard 0.92 → 0.81)

Fraction of the teacher-student gap closed by BPM versus the strongest baselines — Gap closed: 33.5–43.9% (BPM) vs. 19.9–26.6% (best baselines)

Ablations

Ablation Study

Ablation 01 · Divergence

FKL > JSD > RKL on every pair — and the gap widens as tokenizers drift apart

Variant	Math (3)		Code (3)		All (6)
Variant	avg@8	pass@8	avg@8	pass@8	avg@8	pass@8
P1 · Qwen3-32B (teacher)	70.9	82.5	64.0	78.1	67.4	80.3
BPM-FKL	47.7	69.7	30.3	52.1	39.0	60.9
BPM-JSD	46.1	65.9	25.5	49.7	35.8	57.8
BPM-RKL	47.0	67.0	26.6	49.6	36.8	58.3
P2 · GLM-Z1-9B (teacher)	63.0	78.6	63.0	74.7	63.0	76.7
BPM-FKL	47.1	65.0	35.9	53.8	41.5	59.4
BPM-JSD	43.1	60.7	34.2	50.8	38.6	55.8
BPM-RKL	42.5	59.4	31.1	49.9	36.8	54.6
P3 · MiniMax-M2.7 (teacher)	72.5	87.1	58.2	73.9	65.4	80.5
BPM-FKL	41.9	60.6	35.4	53.6	38.7	57.1
BPM-JSD	43.7	59.1	23.8	48.8	33.8	54.0
BPM-RKL^†	38.5	59.8	18.6	41.7	28.5	50.7

Percent. Three-way spread: 3.2 (P1), 4.7 (P2), 10.2 (P3). FKL is β=0, JSD β=½, RKL β=1 at skew 0.1. Tinted rows are teachers. ^† Pre-collapse peak.

Ablation 02 · Data

Harder math and 1.78× more prompts leave the ranking intact

GLM-Z1-9B pair	FKL	JSD	RKL	Avg.
Original mix (20k)
avg@8	41.5	38.6	36.8	39.0
pass@8	59.4	55.8	54.6	56.6
Harder-math mix (20k)
avg@8	39.4_−2.1	38.9_+0.3	37.4_+0.6	38.6_−0.4
pass@8	57.1_−2.3	57.0_+1.2	55.1_+0.5	56.4_−0.2
Expanded corpus (35.5k)
avg@8	40.7_−0.8	38.9_+0.3	37.6_+0.8	39.1_+0.1
pass@8	58.4_−1.0	58.3_+2.5	53.9_−0.7	56.9_+0.3

Six-benchmark percent. Subscripts give the change from the original mix.

Ablation 03 · Whitespace mask

On all-whitespace rows, byte-exact fidelity turns toxic — and a mask removes it

Unmasked whitespace rows drive code pass rates from 49% down to 9%; masking 3.5% of rows removes the collapse.

Design Principle

Route the mass.
Keep the identity.

Every unit of teacher probability is delivered to the student token that owns its bytes, or accounted for in an explicit residual. Nothing dropped, nothing renamed.

Read the arXiv preprint

Citation

Cite this work

@misc{wang2026bpm,
  title={Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization},
  author={Hao Wang and Kun Yuan and Wenlin Zhong and Minglei Zhang and Han Xiao and Ming Sun and Honggang Qi},
  year={2026},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  eprint={2607.22334},
}