Generated 2026-09-23 16:39 · training in progress (10 epochs/task)
action_gain, not val_lossaction_gain = rollout L1 under a zero-motion control action ÷ rollout L1 under the
true action. >1 means the model actually uses its action conditioning. ≈1 means the action
pathway is dead — a failure latent val_loss cannot detect, and one we have previously
observed on this project (zero-action rollouts beating true-action rollouts).
The control is identity_action (dp=0, Rot6D(I)), not a zero vector: zeros encode a
degenerate rotation that never occurs in training.
gain (tactile) is the column that matters — the global mean is dominated by the three
view streams.
| task | train/val eps | train windows | val_loss | last eval ep | rollout L1 | gain | gain (visual) | gain (tactile) |
|---|---|---|---|---|---|---|---|---|
| pushT | 56/4 | 55,342 | 0.3881 | 9 | 0.2026 | 1.812 | 1.819 | 1.784 |
| motherboard | 21/2 | 71,485 | 0.4655 | 9 | 0.2103 | 1.872 | 1.920 | 1.591 |
| rope | 53/8 | 77,847 | 0.4180 | 9 | 0.2512 | 1.465 | 1.550 | 1.206 |
| toy | 24/3 | 52,900 | 0.4005 | 3 | 0.1963 | 1.269 | 1.313 | 1.058 |
Prediction accuracy plateaus while action_gain keeps climbing — later training buys
controllability rather than raw accuracy, and the two metrics are measurably decoupled.
Decoded with the stock diffusers Wan2.2 VAE for viewing only. Scoring uses the Cosmos3 decoder instead — see the pixel-metric note below.
Episodes are filtered to those with visual + wrist + tactile latents AND actions — the wrist
era. The published mixed_mb_latents tree also carries 0510/0511/0519 episodes with no wrist
and no actions (the retired era), which are excluded.
| task | our train windows | authors' count | difference |
|---|---|---|---|
| pushT | 55,199 | 55,342 | 0.26 % |
| motherboard | 71,386 | 71,485 | 0.14 % |
| rope | 77,188 | 77,847 | 0.85 % |
Three independent window counts reproduce the authors' to under 1 % (the gap is windows we drop for invalid action rows). This is the evidence that both models read the same data the same way.
vtm-cosmos-data ships mb, pushT and rope
only, so the mixed_toy_* tree was built for this run: 96 latent files (34 GB) and 96
action files from the raw React recordings. Three gates were passed first, each by reproducing
something already published rather than by looking reasonable — RGB encoding lands at
0.044 of the published mb cache's std (normalising or cropping differently gives 0.15 and
3.03), the geometry conversions pass the repo's own test_robot_candidates 6/6, and
the resulting action statistics match the other tasks to three digits (view |mean| 270.8 against
264.9–273.3, sensor 0.2222 against 0.2222–0.2223).
| setting | value | why |
|---|---|---|
| tactile_delta_to_ref | false | delta-to-ref is our contribution, not VT-WM's; predicting the raw latent is also what Cosmos3 flow-matches, so both are scored on the same target |
| action_lookahead | 1 | an AR predictor supervises out[t] against x[t+1], so it must see the action that generates x[t+1]; lookahead 0 is a bug, not a hyperparameter |
| action_compress | chunk_concat | VT-WM provides the full chunk of delta-states, not its mean (15 raw → 3 latent transitions × chunk 5) |
| backbone | vtwm cross-attn | per-block cross-attention to action tokens, paper Sec. 3.2 (+15.6 M params over the adaLN path) |
Windows drawn uniformly at random across every episode in the split (fixed seed).
The control action is identity_action (dp=0, Rot6D of the identity), not a zero vector.
CI is a paired bootstrap: both rollouts are evaluated on the same windows.
| task | windows | rollout L1 | L1 (control) | action_gain | 95 % CI | windows with gain > 1 |
|---|---|---|---|---|---|---|
| pushT | 512 | 0.1997 | 0.3560 | 1.782 | [1.765, 1.800] | 100.0% |
| motherboard | 512 | 0.2437 | 0.3867 | 1.587 | [1.567, 1.609] | 100.0% |
| rope | 512 | 0.2124 | 0.3617 | 1.703 | [1.678, 1.728] | 100.0% |
| toy | pending | pending | pending | pending | pending | pending |
val_dataloader() built with shuffle=False and stops after 4
batches of 8 — so they are measured on the first 32 windows of the split, all from one episode
and overlapping each other. Good enough to watch a run converge, not an estimate of the split. The gap is
not small, not constant, and not even signed the same way: pushT 1.812 → 1.782 (−2 %),
mb 1.872 → 1.587 (−15 %), rope 1.465 → 1.703 (+16 %). The callback ranks
mb > pushT > rope; the full split ranks pushT > rope > mb.
Every task changes position. A comparison table built from the callback numbers would have been
wrong about which task this baseline handles best, in a way no amount of extra training would fix.
LPIPS ↓ / SSIM ↑ / PSNR ↑, on predicted frames only (8 history frames skipped),
16 rollout samples per task, decoded with the stock diffusers AutoencoderKLWan.
| task | view_middle | wrist_left | wrist_right | tactile L | tactile R |
|---|---|---|---|---|---|
| pushT | 0.111 / 0.864 / 27.2 | 0.203 / 0.773 / 20.7 | 0.169 / 0.839 / 23.5 | 0.051 / 0.966 / 39.7 | 0.055 / 0.971 / 40.3 |
| motherboard | 0.173 / 0.800 / 25.5 | 0.402 / 0.515 / 17.9 | 0.399 / 0.554 / 18.9 | 0.031 / 0.975 / 42.5 | 0.026 / 0.979 / 44.1 |
| rope | 0.116 / 0.868 / 27.8 | 0.233 / 0.735 / 21.9 | 0.274 / 0.795 / 23.6 | 0.029 / 0.975 / 42.3 | 0.042 / 0.975 / 42.3 |
| toy | not evaluated yet | not evaluated yet | not evaluated yet | not evaluated yet | not evaluated yet |
wan22_vae.Wan2_2_VAE plus a tactile LoRA and an anchor-restoration step
(vtm_cosmos/infer/decode.py); this repo has only the stock diffusers decoder. PSNR/LPIPS
across two decoders look comparable and are not — worst of all on the tactile streams, which is exactly
where their LoRA acts. The numbers above are valid for comparing streams and tasks within this
baseline, nothing more.
scripts/mot_predict_windows.py exports predicted
latents in the format vtm_cosmos/scripts/score_external_predictions.py reads, so that
scorer decodes both sides with the same decoder and asserts the GT latents match. pushT is exported
(4,439 val windows); mb and rope follow when they finish.
AllTracker keypoints, discrete Fréchet, stream view_middle.
Object permanence ↓: how far the predicted track of a moving keypoint strays from GT,
normalized by GT path length. Causal compliance ↓: motion invented on static keypoints.
persist = freeze the last context frame; gtcopy = the GT itself.
| task | ref | OP Fréchet (norm) | GT path px | pred path px | CC Fréchet (norm) | hallucinated px |
|---|---|---|---|---|---|---|
| pushT | model | 0.4411 | 14.10 | 10.17 | 0.0735 | 1.24 |
| pushT | persist | 0.8524 | 14.10 | 0.10 | 0.0785 | 0.16 |
| pushT | gtcopy | 0.0000 | 14.10 | 14.10 | 0.0000 | 1.15 |
| motherboard | model | 0.4594 | 15.30 | 10.73 | 0.1172 | 1.24 |
| motherboard | persist | 0.8344 | 15.30 | 0.09 | 0.0970 | 0.09 |
| motherboard | gtcopy | 0.0000 | 15.30 | 15.30 | 0.0000 | 1.37 |
| rope | model | 0.3993 | 18.12 | 13.14 | 0.0653 | 1.35 |
| rope | persist | 0.7497 | 18.12 | 0.09 | 0.0658 | 0.14 |
| rope | gtcopy | 0.0000 | 18.12 | 18.12 | 0.0000 | 1.33 |
| toy | — | — | — | — | — | — |
gtcopy row is the GT compared against itself,
so its Fréchet is 0 by construction — but its "hallucinated path" is not 0. That residual is
the tracker's own jitter on static keypoints, and it is the floor this metric can resolve. Where the model
and persist land close to that floor, CC is not separating them.