World Modeling through
Spectral Alignment

How do we tell a world model what to preserve?

Pairwise distances define a target similarity matrix. The encoder learns embeddings whose pairwise similarities match the target.
Spectral alignment framework. Pairwise distances between samples define a target similarity kernel. We train the encoder to reconstruct this target through embedding similarities.

* iCEM depicted; we report evaluations using CEM.

52%68%

OGBench Cube

+16 percentage points over LeWM baseline
49%77%

OGBench Scene

+28 percentage points over LeWM baseline
29%51%

CALVIN

+22 percentage points over LeWM baseline
In this article
01 / INTRODUCTION

How do we tell a world model what to preserve?

The goal of a world model is to encode sufficient information to predict the consequences of an agent’s actions. At the same time, complex real-world observations contain large amounts of task-irrelevant information. Learning a compact representation therefore hinges on deciding which aspects of an observation to retain.

Joint Embedding Predictive Architectures (JEPAs) predict outcomes in a latent space, rather than reconstructing high-dimensional observations in pixel space, allowing learned representations to abstract away details of the input. The training objective must guide which distinctions the model preserves without requiring it to reproduce every detail of an observation.

We introduce a spectral alignment framework for training world models to preserve a chosen similarity structure. We specify the pairwise relationships we want to preserve through a target similarity kernel, built from distances between observations. We then train an encoder to learn embeddings whose pairwise similarities match the target and an action-conditioned predictor to output future latents. We name the resulting world model SpecWM.

When some information about the underlying system is known, encoders can be trained based on physical distances, but the natural ordering of states within a trajectory is also sufficient to provide a training signal. Empirically, the choice of kernel strongly controls which physical quantities are recoverable from the representation. Temporal spectral alignment improves planning across OGBench Cube, OGBench Scene, and CALVIN.

02 / METHOD

Spectral alignment

Eight robot observations along a trajectory, followed by the temporal target kernel, learned encoder Gram matrix, and their corresponding embedding geometries.
Spectral alignment on a single trajectory. Top: sampled observations from OGBench Scene. Bottom: the temporal target kernel and encoder Gram matrix (before centering), the spectral embedding of K (top three eigenvectors), and the encoder rows h_i (top three principal components), colored by frame index. Matching pairwise similarities encourages the encoder to recover the target’s geometry.

Building a target

We start from a distance d between pairs of observations, derived from ground-truth measurements or simply by counting the observations that separate them in the data. For n observations, we adopt a Laplacian kernel to convert distances into similarities:

K_{ij}=\exp\!\left(-\frac{d(x_i,x_j)}{\sigma}\right).

Observations that are close according to d remain distinguishable, while pairs more than a few \sigma apart have similarity near zero. The bandwidth \sigma therefore sets the range of distances the target resolves; we set it to the median target distance over the pairs the kernel is defined on.

Given a target kernel specifying the desired inner product between every pair of inputs, we can construct a coordinate vector for each input that realizes these pairwise relationships. Such a construction is called a spectral embedding. For a symmetric, positive semidefinite kernel K=U\Lambda U^\top, a spectral embedding is

H_{\mathrm{spec}}=U\Lambda^{1/2},\qquad H_{\mathrm{spec}}H_{\mathrm{spec}}^\top=K.

We center the target to remove its shared baseline and, for fully specified kernels, temper its leading 128 positive eigenvalues with \alpha=1/2 to form K^\alpha.

The encoder f_\theta maps each observation x_i to an embedding h_i\in\mathbb R^d. We rescale these outputs and compute their Gram matrix:

\bar h_i=\sqrt{K^\alpha_{ii}}\frac{h_i}{\|h_i\|},\qquad S=\bar H\bar H^\top.

We double center S and regress the target through a spectral loss:

\mathcal L_{\mathrm{spec}}=\|\widetilde S-K^\alpha\|_F^2.

In the fully specified setting, an embedding table that minimizes this loss recovers the row-normalized spectral embeddings of K^\alpha, up to an orthogonal transformation. Theorem 1 gives the conditions. We can thus evaluate the probing qualities of spectral embeddings derived from arbitrary kernels without ever having to train a model.

Prediction and planning

While the spectral loss is sufficient for training an encoder, planning in latent space requires learning a predictor. We train it by a simple regression objective in latent space, which is also backpropagated through the encoder:

\mathcal L=\mathcal L_{\mathrm{spec}}-\lambda\sum_i\cos(g_\phi(f_\theta(x_i),a_i),f_\theta(x_i^{\prime})).

The predictor g_\phi takes the current embedding and an action, and predicts the embedding of the next observation. For stability, we apply a parameter-free RMS normalization to the encoder output, so that every embedding has norm \sqrt d. This fixes the scale of the embeddings seen by the predictor and the planner.

03 / TEMPORAL KERNELS

Learning from temporal distance

When metadata is not available, the ordering of observations in trajectories itself can form an unsupervised kernel. We use a temporal kernel, which relies on the absolute difference between indices for observations ordered in a trajectory: d(x_i,x_j)=|i-j|. Its only source of supervision is the order of observations in a trajectory.

Temporal kernel bandwidth

Interactive
0 · dissimilar1 · similar
K_{ij}=e^{-|i-j|/\sigma}

At 7 frames apart, similarity is 0.37. A wider band preserves similarity across longer separations.

The bandwidth σ sets the range of distances the target resolves. The plot shows the raw kernel, before centering.

While supervised kernels can describe relationships between all pairs of inputs, the unsupervised kernels we use only express relationships for pairs of inputs in the same trajectory. In this underspecified setting, cross-trajectory similarities are excluded from the training objective.

04 / PLANNING

Planning with SpecWM

Our empirical evaluation revolves around three visual manipulation environments: OGBench Cube and OGBench Scene, and CALVIN. OGBench Scene and CALVIN introduce multiple objects and articulated fixtures whose states must be represented for control. These environments directly test whether the learned representation preserves information about the surrounding scene as well as the robot itself.

With temporal and quasimetric kernels, SpecWM improves upon LeWM by an average of 21 percentage points in planning success rate. The planner ranks predicted outcomes by their embedding similarity to the goal, which spectral alignment shapes to reflect temporal or quasimetric proximity.

CEM success: random 13, 21, 17%; LeWM 52, 49, 29%; temporal SpecWM 68, 77, 51%; quasimetric SpecWM 71, 74, 47% on Cube, OGBench Scene and CALVIN respectively.
Figure 1. Planning success. Average CEM planning success rate on 100 fixed tasks per environment. Bars report the mean over the 20k, 24k, and 28k checkpoints and three seeds, with whiskers showing one cross-seed standard deviation. SpecWM consistently outperforms LeWM.
CEM planning success (%)
Environment LeWM SpecWM
Temporal
SpecWM
Quasimetric
OGBench Cube 52 68 71
OGBench Scene 49 77 74
CALVIN 29 51 47

Overall, a learned quasimetric kernel is generally on par with a readily available temporal kernel, but either may be preferable depending on the environment.

05 / PROBING

What do the representations preserve?

To isolate representation quality from prediction accuracy, we evaluate linear probing performance for the position of objects across environments. We consider held-out observations that the encoder was never trained on. For each of five train-evaluation splits, we fit a ridge regressor for each state variable, choose regularization by cross-validation, and report R^2 on the evaluation set, averaged over variables and splits.

Object-position probe R squared: LeWM versus temporal SpecWM is 0.98 versus 0.81 on Cube, 0.09 versus 0.78 on OGBench Scene, and 0.41 versus 0.47 on CALVIN.
Figure 2. Object-position probes. Linear probe R^2 of the manipulated object’s position from each model’s final-checkpoint embedding. Bars report the mean over three seeds, with whiskers showing one cross-seed standard deviation. Random is an untrained encoder of the same architecture.

SpecWM does not always beat LeWM on probing, and probe quality does not directly correlate with planning performance. On Cube, LeWM recovers object position more accurately but plans less successfully. On OGBench Scene, SpecWM does better at both.

Kernel choice

We extend the investigation to supervised kernels: state, depth, and proprioception. For each, we use the Euclidean distance between the corresponding feature vectors as the target distance. The state kernel uses the simulator state underlying each observation. Although this information is generally unavailable in real-world settings, it lets us evaluate a target with access to the full physical state.

The fully specified state kernel achieves the highest average R^2 (0.93). Under underspecified supervision, the temporal kernel nearly matches the state kernel (0.84 versus 0.85). Proprioceptive kernels preserve the robot’s configuration but largely omit object states.

OGBench Scene probe R² Temporal1 State2 Proprio2 Depth2
OGBench Scene average 0.84 0.93 0.16 0.50
Cube 0.79 0.96 0.08 0.09
Arm 0.67 0.73 0.67 0.59
Drawer 0.91 0.99 0.01 0.87
Window 0.87 0.99 0.00 0.91
Buttons 0.96 0.99 0.02 0.04

Selected columns from the paper’s Table 1: linear probe R^2 on held-out OGBench Scene episodes. 1Relationships within trajectories only. 2Relationships specified across states.

Kernel eigenvector probe curves for state, proprioception and depth targets; fully specified targets generally recover eigenvectors better than underspecified targets or an untrained encoder.
Figure 3. Spectral recovery. Linear probe R^2 between kernel eigenvectors and embeddings from encoders trained on fully specified state, proprioception, and depth. Underspecified variants and an untrained encoder are included for comparison.

We evaluate the method's ability to recover spectral embeddings by by fitting linear probes to each eigenvector from the model embeddings. Recovery is strongest for eigenvectors associated with large eigenvalues and decreases as relative contribution decreases.

The state kernel's lower recovery scores at later indices is largely a function of its less even distribution of magnitude across eigenvalues. Weighting each eigenvector’s probe R^2 by its normalized eigenvalue gives mean scores of 0.965 for state, 0.987 for proprioception, and 0.992 for depth.

06 / ABLATIONS

Additional ablations

Failure data

Predictors tend to generalize poorly to random actions when trained on expert-only data. We train LeWM and temporal SpecWM on the play data alone, at matched updates, frames per update, and schedule. Failure data raises planning success for both models in every environment, but does not explain the difference between them.

Failure-data ablation showing planning success and object-position probes with failure rollouts versus play data only on all three environments.
Figure 4. Effect of the failure data. CEM success (top) and object-position probe R^2 (bottom) for LeWM and temporal SpecWM, trained with failure rollouts (solid, three seeds) or on play data alone (hatched, one seed), at matched updates and frames per update. SpecWM outperforms LeWM in both cases.
Prediction gradients

We evaluate a variant of SpecWM which does not backpropagate the predictor regression objective through the target. The prediction loss still updates the encoder through the current observation, but no longer through the next observation used as its target. Planning success generally decreases, while probing results do not follow an obvious trend.

Planning and probing comparisons with and without stop-gradient on the prediction target.
Figure 5. Effect of a stop-gradient on the prediction target. Bars report the mean over three seeds, with whiskers showing one cross-seed standard deviation. The stop-gradient generally lowers planning success and raises object recoverability in some cases while diminishing recoverability in others.
Planning cost and normalization

The two world models have different inherent distance metrics in their latent distributions. We use the one that fits each latent most naturally: L2 for LeWM and cosine similarity for SpecWM. The choice of measure is not the cause of the planning gains. The choice of normalization layer also does not have a substantial impact on planning performance.

LeWM planning success 50% with L2 and 51% with cosine; temporal SpecWM 77% with either cost.
Figure 6. Planning cost. Cosine similarity and L2 distance are evaluated on both LeWM and SpecWM. Three training seeds are evaluated at their respective final three checkpoints for each setting.
LeWM success 53% with BatchNorm and 51% with LayerNorm; temporal SpecWM 77% with either normalization.
Figure 7. Normalization layer. BatchNorm versus LayerNorm in the MLP producing the encoder embedding. One seed is used per run; CEM success is averaged over the final three checkpoints.
07 / CONCLUSION

Conclusion

The core idea is to directly specify the pairwise relationships that representations should preserve through a target kernel. We find that readily available kernels result in strong planning performance, which is not entirely correlated with probing accuracy.

Pursuing a better understanding of the mechanisms enabling effective planning would be valuable. We view the study of which kernels are most conducive to large-scale self-supervised training as the next primary direction of work. Kernels involving language for semantic distances are another future direction.

08 / THEORY

Theoretical analysis

The target kernel specifies which relationships the encoder should preserve. We now ask what matching those relationships implies: which representations minimize the loss, which quantities a linear probe can recover, when irrelevant information disappears, and why temporal similarity can help planning. The results below describe idealized optima under explicit assumptions; full proofs appear in the paper’s proofs appendix.

Theorem 1

Spectral recovery

Suppose m\le d and K^\alpha_{ii}>0 for every input i. Then the minimum spectral loss is zero, attained exactly at

H^\star=\operatorname{diag}(c)\,H_{\mathrm{spec}}\,Q,\qquad c_i>0,\quad Q^\top Q=I_d.

Here m is the number of retained positive spectral directions, d is the encoder dimension, H_{\mathrm{spec}} is the spectral embedding of K^\alpha with square-root eigenvalue scaling, padded to d dimensions, and Q\in\mathbb R^{d\times d} is orthogonal.

Explanation

In the fully specified setting, the target defines relationships between every pair of inputs. Its centered, tempered target K^\alpha can be decomposed into eigenvectors and eigenvalues. A spectral embedding uses the retained eigenvectors as coordinates, scaled by the square roots of the target’s eigenvalues. When the embedding dimension is large enough, these coordinates reproduce the target’s pairwise inner products exactly.

The optimum recovers the target’s geometry up to a rotation or reflection and positive scaling of each row. Row scaling remains free because the loss rescales each embedding to the length prescribed by the target diagonal. With fixed-norm encoder outputs, this becomes a row-normalized spectral embedding, up to an orthogonal transformation.

This gives a way to inspect a kernel before training a neural encoder: compute its spectral embedding and evaluate what its coordinates make accessible. The theorem characterizes the optimum of an embedding table; it does not guarantee that neural-network training reaches that optimum.

Theorem 2

Linear recoverability

Let Y\in\mathbb R^{n\times p} be centered, with \mathbf1^\top Y=0, unit row norms \|y_i\|=1, and \operatorname{rank}(Y)\le d. Set K=YY^\top, use \alpha=1, and retain all positive spectral directions. Every global minimizer H^\star of the spectral loss subject to \|h_i\|=\sqrt d satisfies

\min_{W\in\mathbb R^{d\times p}}\|Y-H^\star W\|_F^2=0.
Explanation

A linear probe can recover a quantity exactly when its values lie in the representation’s column space. The spectral characterization therefore connects kernel choice to the information accessible downstream. In particular, a kernel built from label relationships can make the labels themselves linearly recoverable, even though the encoder never directly regresses onto those labels.

Under these conditions, matching the labels’ pairwise inner products preserves enough information to reconstruct them with a linear map. This motivates targets built from physical quantities. The exact guarantee applies to the stated label kernel and normalization; it does not imply perfect recovery for every distance-based kernel used in the experiments.

Theorem 3

Abstraction of task-irrelevant information

Write each observation as x_i=(s_i,n_i), where the target depends on relevant state s_i and is invariant to nuisance n_i. Assume coverage, fixed-norm encoder outputs, and target-diagonal rescaling. If each trajectory’s centered, tempered target K^\alpha has strictly positive diagonal entries almost surely and \mathbb E[\mathcal L_{\mathrm{spec}}]=0, then

f_\theta((s_i,n))=f_\theta((s_i,n'))

for almost every clip, every frame i, and almost every pair (n,n') drawn independently from \nu(\cdot\mid s_i).

Explanation

Selective invariance means retaining relevant distinctions while dropping details the target ignores. Write each observation as x_i=(s_i,n_i), where s_i is task-relevant state and n_i is nuisance information. For example, a manipulation target may depend on object positions while ignoring moving foliage in the background. We ask whether matching that target also makes the encoder ignore the foliage.

Let \nu(\cdot\mid s_i) be the distribution of nuisance values at state s_i, and let Z collect the task-relevant trajectory and target metadata. We need a coverage assumption: nuisance values that are individually possible at two states must also be possible together in a trajectory visiting those states.

Coverage includes independent sensor noise and correlated disturbances when all otherwise possible combinations remain possible. It excludes a background identity or lighting condition that varies across trajectories but stays fixed within each one. Such a shared attribute could select a rotation for every embedding in a clip without changing their pairwise similarities.

At zero expected loss, changing the nuisance component while holding the relevant state fixed leaves the representation unchanged. Coverage lets us vary nuisance draws while preserving the target’s inner products; centering forces the rescaled embeddings to sum to zero, preventing one such draw from changing its embedding independently. This result assumes target-diagonal rescaling, which is omitted in our practical underspecified training variant.

Theorem 4

Population target

Consider a finite, stationary, irreducible, aperiodic Markov chain with stationary distribution \mu. Let M^\beta be its behavioral successor measure with discount \gamma=e^{-1/\sigma}, and R_T(x,x')=\mathbb E[\gamma^{|i-j|}\mid x_i=x,x_j=x'] for uniformly sampled indices in a length-T trajectory. Omitting centering, tempering, and truncation, for every pair x,x'\in\mathcal S,

\begin{aligned} T R_T(x,x')\xrightarrow[T\to\infty]{}&\frac{M^\beta(x,x')}{\mu(x')}+\frac{M^\beta(x',x)}{\mu(x)}\\ &-\frac{\mathbf1[x=x']}{\mu(x)}. \end{aligned}

If the chain is reversible, with \mu(x)P(x'\mid x)=\mu(x')P(x\mid x'), this simplifies to

T R_T(x,x')\xrightarrow[T\to\infty]{}\frac{2M^\beta(x,x')}{\mu(x')}-\frac{\mathbf1[x=x']}{\mu(x')}.
Explanation

Temporal supervision uses only how far apart observations occur in a trajectory. To understand its connection to planning, we move from a free embedding table to a state-conditioned encoder h:\mathcal S\to\{z\in\mathbb R^d:\|z\|=1\}. The behavioral successor measure counts discounted future visits under the data-collection policy:

M^\beta(x,x')=\sum_{t=0}^{\infty}\gamma^t\Pr(x_t=x'\mid x_0=x,\pi_\beta),\qquad \gamma=e^{-1/\sigma}\in(0,1).

Assume a finite state space and a stationary Markov policy \pi_\beta whose induced chain P is irreducible and aperiodic. Its stationary distribution is \mu, and trajectories start with x_0\sim\mu. Sample both indices uniformly from a length-T trajectory. For this analysis, omit embedding and target centering, target tempering, and spectral truncation.

Writing s(x,x')=h(x)^\top h(x'), the population loss is

\mathcal L_{\mathrm{spec}}^{\mathrm{pop}}(h)=\mathbb E_{\tau,i,j}\left[\big(s(x_i,x_j)-\gamma^{|i-j|}\big)^2\right].

The same state pair can occur at different temporal separations. Define its expected target similarity as

R_T(x,x')=\mathbb E\left[\gamma^{|i-j|}\mid x_i=x,\ x_j=x'\right].

A conditional bias–variance decomposition separates the loss into regression onto R_T and a term independent of the encoder. Minimizing the population loss is therefore equivalent to minimizing the expected squared error between embedding similarity and R_T. The next result identifies this target for long trajectories.

The temporal target thus contains a symmetrized successor measure, normalized by stationary state frequencies. It reflects discounted visitation in both temporal directions, relative to how common each state is in the data. This connects the similarity score used in planning to how readily states lead to one another under the behavior policy, within the idealized setting above.

Theorem 5 · Appendix

Kernel dominance

Take two centered, tempered targets of rank at most d with positive diagonals, and global minimizers with row norm \sqrt d. Define their cosine Gram matrices by K^{\mathrm{cos}}_{ik}=K^\alpha_{ik}/\sqrt{K^\alpha_{ii}K^\alpha_{kk}}. The first kernel gives no greater optimal linear-probe error for every target Y if and only if

\operatorname{col}(K_2^{\mathrm{cos}})\subseteq\operatorname{col}(K_1^{\mathrm{cos}}).

Dominance is strict if and only if this inclusion is strict.

Explanation

The paper’s additional kernel-dominance result asks when one kernel is at least as useful as another for every linear probe target. At a zero-loss optimum with fixed row norms, the encoder’s column space is determined by the target’s cosine Gram matrix. Dominance therefore amounts to inclusion of the spaces of linearly recoverable targets.