ClothAtlas: Joint Garment Identification and State Estimation under Self-Occlusion

Jens Lundell1,*, Alberta Longhini2,*, Lei Shu2, Hemal Arora2, Evan Robert2, Carlota Parés-Morlans2, Marcel Büsching3, Mårten Björkman3, Jeannette Bohg2
1University of Turku, 2Stanford University, 3KTH Royal Institute of Technology
*Equal contribution
Paper Video BibTeX

Abstract

Garment manipulation requires inferring both garment identity and complete configuration from partial, self-occluded observations. This creates a chicken-and-egg problem: completing a partial observation requires a garment-specific deformation prior, while identifying the garment requires a configuration that explains the observation. We introduce ClothAtlas, which jointly identifies garments and estimates their complete configurations from partial point clouds and a library of canonical meshes. A shared graph-conditioned flow-matching model samples configurations for each candidate mesh and ranks candidates by observation fit. The model accommodates different mesh shapes, vertex counts, and connectivities, allowing new garments to be added from canonical meshes alone without retraining.

On held-out DexGarmentLab garments, ClothAtlas ranks the correct mesh first among 221 candidates for all 22 test garments from a single flat observation. From a single self-occluded observation, it identifies 15/22 garments, compared with 8/22 for a deterministic graph estimator. Persistent vertex identities support grasp-target localization on real garments, reducing mean error from 15.4 to 6.4 cm overall and from 17.0 to 8.3 cm on occluded targets compared with dense descriptor matching.

ClothAtlas takes a mesh library and a partial depth cloud and returns the identified canonical mesh and its full fitted configuration.
Three mesh vertices tracked through a lifting action, including while occluded from the camera.
(a) ClothAtlas combines a canonical mesh library and partial depth observations to identify the garment and estimate its complete configuration. Red points are observations; the blue surface is the estimated configuration, including regions occluded from the camera. (b) Colored tracks connect the estimated positions of three mesh vertices through a lifting action.

Method

ClothAtlas approximates the joint posterior over garment identity and configuration by generating complete configurations and comparing their fit to the observations. For each candidate canonical mesh, a conditional flow-matching model generates configurations from the current point cloud. We identify the canonical mesh by accumulating observation-fit scores in the candidate weights across observations, and report the highest-weight candidate and its configuration estimate.

Method overview: generative configuration estimation, mesh identification, and sequential update.
(a) A shared graph-conditioned generative model samples complete configurations from a partial observation, Gaussian noise, and a candidate canonical mesh, which supplies reference vertex positions and connectivity. The same model is used for every candidate mesh. (b) We score each candidate by the observation fit of its configuration estimate, accumulate evidence across observations, and retain the k highest-weight candidates after the initial identification period. (c) For each retained candidate, we sample a new configuration from the current observation and update its weight.

Garment identification

Top-four candidates for a held-out top: ClothAtlas above, D-GNN below, colored by distance to the ground-truth surface.
Top-four candidates for a held-out top at lift frame 200: ClothAtlas above, D-GNN below. Insets show canonical meshes. The correct mesh (green) ranks first for ClothAtlas and fifth for D-GNN, outside the displayed candidates. Obs. RMS: observation-to-reconstructed-surface root mean squared distance.

Identification from a single self-occluded observation

Held-out simulated garments at lift frame 200, without the correct canonical mesh supplied. The top-three candidates of ClothAtlas (upper row) and D-GNN (lower row) are colored by distance to the ground-truth surface; the green box marks the correct mesh. Select a garment above.

Sequential identification

ClothAtlas accumulates observation-fit evidence over nine observations of a lifting action. Each timestep shows the current top-three candidates, their reconstructions colored by distance to the ground-truth surface, and the reconstruction RMSE; the green box marks the correct mesh.

Real-world grasp-target localization

Real-world rollouts comparing ClothAtlas and dense descriptors on pants and a top under occlusion.
Four real-world rollouts compare ClothAtlas (upper rows) with dense semantic descriptors (DD, lower rows) under temporary occlusion (pants, left) and persistent occlusion (top, right). Green marks the second grasp target G2 and cyan its estimate; orange shows the completed mesh, and the heatmap shows descriptor cosine similarity. Checks mark second grasps that reached the target; crosses mark failures.

Real-world rollouts

Pants: temporary occlusion

ClothAtlas
Completed mesh (side camera)
Grasp-target estimate (robot camera)
Dense descriptors
Descriptor similarity (side camera)
Grasp-target estimate (robot camera)

Top: persistent occlusion

ClothAtlas
Completed mesh (side camera)
Grasp-target estimate (robot camera)
Dense descriptors
Descriptor similarity (side camera)
Grasp-target estimate (robot camera)

The robot grasps the garment, lifts it, and then grasps a second target G2 that may be hidden from the camera. Green marks G2 and cyan its estimate. ClothAtlas tracks G2 through the persistent vertex identities of its completed mesh (blue); dense descriptor matching uses the best-matching visible point (heatmap). Videos in each row play in sync.

Acknowledgements

This work was supported in part by Agile Robotics. Alberta Longhini was supported by a Wallenberg–Bienenstock Postdoctoral Fellowship. The authors wish to acknowledge CSC – IT Center for Science, Finland, for generous computational resources.

BibTeX

@article{lundell2026clothatlas,
  title   = {ClothAtlas: Joint Garment Identification and State Estimation under Self-Occlusion},
  author  = {Lundell, Jens and Longhini, Alberta and Shu, Lei and Arora, Hemal and Robert, Evan and
             Par{\'e}s-Morlans, Carlota and B{\"u}sching, Marcel and Bj{\"o}rkman, M{\aa}rten and Bohg, Jeannette},
  journal = {arXiv preprint arXiv:2610.16293},
  year    = {2026}
}