A Visuo-Tactile World-Action Model
for Dexterous Manipulation
A Visuo-Tactile World-Action Model for Dexterous Manipulation
Vision-centric WAMs model visible scene dynamics but omit hidden contact dynamics. Tactile-conditioned policies can observe current contact, but do not necessarily model how contact evolves. DexTacWAM instead treats touch as part of the predicted world state and jointly models visual and multi-finger tactile dynamics.
Learn action-conditioned visual state transitions and predict how the visible scene evolves. But contact-interface phenomena, such as pressure, slip, and grasp stability, are weakly observable or entirely hidden from cameras, especially once the hand occludes the object it is manipulating.
Tactile policies can use current contact observations directly, but this does not make contact evolution part of the predicted world state.
DexTacWAM jointly models visual scene and tactile contact dynamics, exposing predictive contact-aware features to the action expert.
Dexterous manipulation depends on contact dynamics that are often only partially observable from vision. Recent World-Action Models (WAMs) couple predictive video world modeling with action generation, but remain largely vision-centric and therefore cannot directly model these contact dynamics. We present DexTacWAM, a visuo-tactile WAM that encodes each fingertip independently, aggregates the resulting features through a finger- and pose-aware tactile compressor, and injects the tactile latent into a video diffusion world model for joint visuo-tactile world modeling. Across six contact-rich dexterous manipulation tasks on a 22-DoF bimanual platform, DexTacWAM achieves the highest score on every task, averaging 70.6 versus 38.0 for the strongest baseline. Ablations attribute the gain to modeling contact evolution as part of the predicted world state rather than tactile conditioning alone: removing tactile world modeling reduces the four-task mean from 74.7 to 26.6 while keeping the same tactile features and action expert. After four hours of tactile-encoder adaptation with a frozen pretrained vision VAE, our continual vision-to-touch learning extends the pretrained video model to touch using roughly 100 demonstrations per task, without tactile midtraining, while retaining visual prediction quality within 0.5 dB of matched vision-only counterparts. The compressor retains 89.4% of pre-fusion contact recall while enabling 2.26× faster training and 1.29× faster inference. Together, these results show that pretrained video priors can be extended to distributed multi-finger contact dynamics in a data- and compute-efficient manner.
Human video is abundant, but tactile data is scarce and costly to collect. DexTacWAM therefore continually extends a pretrained visual prior to touch, requiring only four hours of tactile-encoder adaptation plus roughly 100 demonstrations per task.
We first adapt lightweight tactile-encoder components on a 488-episode bimanual corpus, keeping the pretrained visual VAE frozen. The resulting compressor retains 89.4% of pre-fusion contact recall.
We then freeze the tactile encoder and finetune the pretrained video backbone for joint visual and tactile prediction using roughly 100 demonstrations per task.
A freshly initialized action expert is trained on those same demonstrations, so no data beyond the four-hour corpus and the per-task demonstrations is required.
Continual vision-to-touch learning works because the reused visual VAE already provides task-discriminative tactile features, and the tactile compressor preserves that structure while reducing the tactile token budget.
Held-out tactile clips from all six tasks remain task-discriminative both before and after 5:1 compression. This supports reusing the frozen visual VAE as a tactile feature extractor and shows that the compressor preserves task-relevant structure; it does not imply that vision and touch occupy a shared embedding manifold.
A single right-thumb representation reaches 95.6% 5-NN accuracy and 99.0% linear-probe accuracy, versus 16.7% uniform chance. This shows a frozen visual VAE can provide useful tactile features without training a tactile VAE from scratch.
After five fingertips are compressed into one hand latent, 5-NN and linear-probe accuracy reach 96.4% and 99.6%, matching the 95.6% and 99.0% of the uncompressed single-fingertip representation above. The 5:1 compression preserves the task-relevant structure needed for efficient tactile world modeling.
Within each task, structure follows interaction progress rather than recording identity. This indicates that the representation captures phase-dependent structure rather than only task or episode identity.
Right-thumb tactile clips are embedded separately for each task and colored by normalized episode progress. Phase ANOVA R² is 0.75 for Tongs, 0.61 for Two-Hand Wipe, 0.48 for Cube Place, 0.40 for Cube Handover, and 0.17 for Bowl, while episode agreement is negligible for every task (AMI ≤ 0.07). Bottle Cap is the exception (phase R² 0.03), because cyclic unscrewing is not well described by a single start-to-end phase.
Transfer does not imply a shared vision–touch embedding: vision and tactile latents remain well separated. The benefit comes from reusing a general-purpose image representation in a new image domain.
Treating all ten fingertip streams as separate DiT views substantially increases token count and compute. Compressing five fingertip streams into one hand latent reduces the world-model input from 11 views to 3 while retaining localized contact information.
| Views (V) | Composition | DiT Tokens | Latency / chunk |
|---|---|---|---|
| 3 (deployed) | 1 head + 2 hand tactile | 864 | 281.6 ms |
| 11 (uncompressed) | 1 head + 10 fingertip | 3168 (3.67×) | 363.0 ms (1.29×) |
Compression reduces training iteration time from 3.87 to 1.71 s/iter (2.26×) and end-to-end inference latency from 363.0 to 281.6 ms per chunk (1.29×).
A finger- and pose-aware tactile compressor aggregates five fingertip streams into one hand latent, preserving localized contact information while reducing ten fingertip streams to two tactile views.
Compact hand latents enter the pretrained video world model as tactile views and are modeled jointly with visual dynamics. A single world-model forward pass produces predictive visuo-tactile features for the action expert without requiring perceptually complete denoising.
Compact tactile latents enter the pretrained video DiT as additional tactile views (1 head + 2 hand tactile). The world-model transformer jointly models visual and bimanual tactile dynamics and provides predictive visuo-tactile features in one forward pass. The action expert cross-attends to these features with per-modality K/V RMS normalization and predicts contact force, action, and next state.
Each single-channel per-finger tactile map passes through a 1×1 grayscale→RGB conv into pseudo-RGB, then a frozen pretrained visual VAE shared across fingers, providing a spatial prior compatible with the video diffusion latent space.
Finger identity, hand pose, self-attention pooling, and temporal refinement compress five fingertip latents into one hand latent while preserving localized contact structure.
Hand tactile latents are appended as tactile views during training and jointly optimized with visual latents using a modality-split flow-matching objective.
The action expert cross-attends to world-model latents with per-modality RMS normalization of K/V. Visual and tactile keys arrive at very different scales; without this normalization, the action expert fails to converge.
Per-finger tactile maps → grayscale→RGB conv → frozen vision VAE (shared) → finger-specific positional embeddings → a tactile compressor combining self-attention pooling, TimeSformer refinement, and pose conditioning. Pre- and post-fusion reconstruction losses supervise the latents and encourage the fused hand latent to retain localized contact.
What the encoder actually sees: each fingertip returns a grayscale image of a deformable marker grid, and contact appears as a localized compression and shear of that grid. This spatial image structure makes a pretrained visual VAE a natural starting point for touch.
Raw visuo-tactile observations before and during contact on the two-hand Wipe task. Each block is one instant from a teleoperated episode: the head and two wrist-camera views, together with all ten fingertip tactile observations. Before contact, both hands are in free space and the marker grids remain largely undeformed. During wiping, the left hand stabilizes the board while the right hand loads the eraser, producing localized compression and shear on the engaged fingertips. All tactile panels share the same grayscale range.
We measure contact recall before and after five-finger-to-one-hand fusion on held-out episodes. Retention@9 measures the fraction of pre-fusion Recall@9 preserved after five-finger-to-one-hand compression.
| Variant | Recall@1 | Recall@9 | Retention@9 |
|---|---|---|---|
| Cross-attention pooling | 0.561 | 0.492 | 0.629 |
| Self-attention pooling | 0.656 | 0.575 | 0.717 |
| Self-attention + pose injection | 0.750 | 0.645 | 0.796 |
| Self-attention + temporal refinement | 0.748 | 0.710 | 0.873 |
| Full compressor (pose + temporal) | 0.748 | 0.725 | 0.894 |
| w/o finger-identity embeddings | 0.702 | 0.671 | 0.849 |
Evaluated on the 24 held-out episodes (464–487) of the diverse-488 corpus. Every variant shares the same frozen VAE and the same per-finger inputs, so the differences come from the aggregation design. Removing finger identity reduces Recall@9 by 0.054, supporting the use of finger identity during aggregation.
Six tasks on a 22-DoF bimanual platform, spanning tool-mediated force control, friction- dependent separation, rotational contact, sustained surface contact, occlusion, and hand-to-hand transfer. Camera access is set by task and is identical across all methods within a task.
The six task videos below show autonomous DexTacWAM rollouts. Original playback speed.
Precise normal force for grasping a cherry tomato with tongs.
Coordinated thumb–index contact during unscrewing.
Separating stacked bowls through fingertip contact.
Maintaining stable contact during a long wiping motion.
Placing the cube while the arm occludes the object.
Coordinating grasp and release between hands.
20 real-robot trials per method per task. DexTacWAM leads the strongest baseline by 32.6 absolute points.
| Method | Cube Place (H) |
Handover (H+L+R) |
Wipe (H+L+R) |
Tongs (H+R) |
Bowl (H+R) |
Cap (H+L+R) |
Avg. |
|---|---|---|---|---|---|---|---|
| π0.5 (no tactile) | 10 | 46 | 24 | 0 | 40 | 45 | 27.5 |
| ViTacFormer | 20 | 33.3 | 26 | 0 | 20 | 10 | 18.2 |
| RDP | 50 | 60 | 48 | 10 | 30 | 30 | 38.0 |
| Genie Envisioner (no tactile) | 10 | 30 | 44 | 5 | 35 | 40 | 27.3 |
| DexTacWAM | 90 | 71.6 | 77 | 60 | 65 | 60 | 70.6 |
H/L/R denote head, left-wrist and right-wrist cameras. The widest margin is Tongs (60 vs 10), where tactile sensing provides evidence of grasp state through the tool; all compared baselines score at or below 10.
| Variant | Cube Place | Handover | Wipe | Tongs | 4-task mean |
|---|---|---|---|---|---|
| DexTacWAM (predictive tactile WM) | 90 | 71.6 | 77 | 60 | 74.7 |
| w/o tactile world modeling (direct tactile conditioning) | 20 | 33.3 | 48 | 5 | 26.6 |
| w/o per-modality K/V RMS normalization | Did not converge — open-loop actions fail to track ground truth | ||||
The ablated model keeps the same tactile encoder, observations, action expert, action space and training data; only tactile prediction is removed from the world model, and the action expert is conditioned on encoded tactile features directly instead of on predicted tactile latents. The four-task mean decreases from 74.7 to 26.6. The gain therefore depends strongly on modeling contact evolution as part of the predicted world state rather than on tactile conditioning alone.
Trained only on three-bowl stacks in pink, blue and orange, DexTacWAM reaches 13/20 on unseen green, purple and yellow bowls — matching its in-distribution 13/20 — and 12/20 when the stack height changes to two or four bowls. Performance remains stable under changes in object appearance and stack geometry, which also induce different grasp configurations.
Keyframes progress from left to right for three unseen bowl colors (yellow, green, purple) and two unseen stack heights (two and four bowls), none of them seen during training. Every frame is sampled from an autonomous closed-loop execution, not a teleoperated demonstration. In all cases the policy reaches the rim, establishes thumb–index contact, and separates the top bowl.
We visualize future visual observations and tactile deformation flow on a held-out Two-Hand Wipe trajectory.
Long-horizon rollout. Left two panels: ground-truth vs. predicted head view. Right two blocks: per-finger tactile divergence for the left and right hand, ground truth beside prediction. The predicted contact patterns emerge, move, and release in step with the real ones.
| Metric | Value | Interpretation |
|---|---|---|
| Latent cosine similarity ↑ | 0.967 | Predictions align with the true tactile future |
| NMSE ↓ | 0.063 | Low normalized latent prediction error |
| Contact F1 ↑ | 0.737 | Precision/recall/F1 are 0.749/0.725/0.737 in flow space |
| Error vs. copy-last-frame ↓ | 0.561× | The model predicts dynamics beyond repeating the present |
Tactile signals are temporally smooth, so a forecaster can score well by simply repeating the last observation. DexTacWAM attains 0.561× the copy-last-frame error, indicating that it predicts contact evolution rather than exploiting temporal smoothness.
DexTacWAM directly adapts a pretrained visual world model to touch using task-scale demonstrations. We test whether acquiring tactile prediction degrades the original visual prediction capability.
| Task | Steps | Vision-only PSNR ↑ | Visuo-tactile PSNR ↑ | Δ PSNR | 95% CI | Δ SSIM ↑ | Δ LPIPS ↓ |
|---|---|---|---|---|---|---|---|
| Cube Handover | 25K | 17.78 | 17.84 | +0.06 | [−0.16, +0.28] | +0.001 | +0.004 |
| Two-Hand Wipe | 30K | 17.14 | 17.07 | −0.07 | [−0.49, +0.36] | −0.006 | +0.010 |
Differences are paired per episode, with the three views averaged within each episode. Neither task shows a detectable change; the intervals bound any PSNR loss at 0.16 dB on Cube Handover and 0.49 dB on Two-Hand Wipe.
DexTacWAM extends a pretrained video prior to jointly model visual scene dynamics and multi-finger contact dynamics. Across six contact-rich tasks it achieves the highest score on every task. The gain depends strongly on modeling contact evolution as part of the predicted world state rather than on tactile conditioning alone. Continual vision-to-touch learning adds tactile predictive capability using four hours of tactile data for lightweight tactile-encoder adaptation and task-scale demonstrations while preserving visual prediction quality. Its predictive-feature interface avoids requiring perceptually complete denoising before action generation, while tactile compression delivers measured training and inference speedups. Together, these results show that pretrained video priors can be extended to touch both data- and compute-efficiently, providing a practical path toward WAMs for complex physical interaction.
Because tactile data is expensive and scarce, we reuse a frozen visual VAE as the per-finger tactile feature extractor and adapt only the tactile encoder's lightweight projection and compressor on four hours of data. These features are task-discriminative, and structure is retained after 5:1 compression. We then freeze the tactile encoder and directly extend the pretrained video model to tactile prediction with task-scale data, without a separate tactile-midtraining stage for the video backbone.
The controlled ablation keeps the tactile encoder, observations, action expert, action space, and training data fixed while removing tactile prediction from the world model. The four-task mean decreases from 74.7 to 26.6. A single world-model forward pass provides predictive visuo-tactile features directly to the action expert without requiring perceptually complete denoising of either stream.
Diffusion denoising provides a self-supervised training objective for learning predictive visual and tactile representations. At inference, a single world-model forward pass provides predictive visuo-tactile features directly to the action expert, without reconstructing perceptually complete futures. Fully denoising both streams adds substantial computation and latency, making low-latency deployment difficult on resource-constrained edge hardware, while offering limited additional control benefit in our setting.
We use co-world modeling to mean jointly modeling visual and tactile latents as one predicted world state. Tactile latents are appended along the view axis and trained with a modality-split objective, so the model captures both scene and contact dynamics.
In the central ablation, we remove tactile prediction from the world model and condition the action expert directly on the same encoded tactile features, holding the encoder, observations, action expert, action space, and data fixed. The four-task mean decreases from 74.7 to 26.6, showing that tactile conditioning alone does not account for the gain.
DiT cost scales with view count: 1 head + 10 fingertip views is 3168 tokens against 864 for 1 head + 2 hand views (3.67×), and 363.0 vs 281.6 ms per chunk. The finger- and pose-aware tactile compressor keeps 89.4% of pre-fusion contact recall through that compression.
Visual and tactile keys enter cross-attention at different scales. Without per-modality K/V RMS normalization, the action expert does not converge: open-loop predictions fail to track ground-truth actions, so closed-loop evaluation is not performed.
@article{dextacwam2026,
title = {DexTacWAM: A Visuo-Tactile World-Action Model for Dexterous Manipulation},
author = {Yuan, Haoran and Wang, Zekai and Shao, Boning and Lu, Haoran and Darrell, Trevor and Lourentzou, Ismini and Zhan, Wei},
journal = {arXiv preprint arXiv:2609.24976},
year = {2026}
}