Yuqi Wu*, Tianyu Hu*, Wenzhao Zheng*†, Yuanhui Huang, Haowen Sun, Jie Zhou, Jiwen Lu

* Equal contribution. † Project leader.

We propose IVGT, an Implicit Visual Geometry Transformer that implicitly models coherent 3D geometry and appearance from pose-free multi-view images in a single feedforward pass. It learns a continuous neural scene representation in a global canonical coordinate system, enabling diverse downstream tasks including mesh reconstruction, novel-view synthesis, and surface estimation across diverse scenes.

• [2026/5/18] Paper released on arXiv.

Most existing visual geometry foundation models (DUSt3R, VGGT, Point3R) follow an explicit paradigm, which directly decode a pixel-aligned 3D point for each input pixel. This leads to two fundamental limitations:

• Discrete: geometry only exists at pixel locations; arbitrary 3D positions cannot be queried.
• Redundant: the same surface point appearing in multiple views gets predicted independently, causing inconsistencies.

IVGT takes the opposite approach: we learn a continuous 3D neural field (SDF-based) queryable at any position in space.

Explicit models decode per-pixel 3D coordinates for each view → discrete, view-indexed pointmaps.
IVGT (implicit) learns a continuous 3D field → direct surface extraction and novel view rendering without post-processing.

The advantage of implicit geometry over explicit pointmaps is most visible in surface quality. Pixel-aligned pointmap reconstruction suffers from sparsity and discontinuities, especially at object boundaries, while meshes extracted from IVGT's continuous SDF field are significantly more coherent and complete:

IVGT produces complete and coherent colored meshes across diverse indoor scenes and objects in a single forward pass:

Qualitative mesh comparison on ScanNet. IVGT produces geometrically complete and surface-coherent meshes in one forward pass; per-scene optimization baselines often yield incomplete or fragmented surfaces.

Colored mesh reconstruction across diverse indoor scenes and objects of varying scale, without any test-time optimization.

The same SDF field supports four output modalities from a single representation:

• RGB rendering via differentiable volume rendering (SDF → density → color)
• Depth maps from the expected ray termination depth
• Surface normal maps from the SDF gradient
• Continuous meshes via Marching Cubes — directly and efficiently, without any post-processing

IVGT renders RGB images, depth maps, and surface normal maps from arbitrary novel viewpoints. All three modalities are derived from the same SDF field without task-specific heads.

Reconstructing coherent 3D geometry and appearance from unposed multi-view images is a fundamental yet challenging problem in computer vision. Most existing visual geometry foundation models predict explicit geometry by regressing pixel-aligned pointmaps, suffering from redundancy and limited geometric continuity. IVGT learns a continuous implicit neural scene representation in a canonical coordinate system, supports continuous spatial queries at any 3D position, and predicts SDF values and colors using lightweight decoders — enabling rendering and surface extraction from arbitrary viewpoints.

We train IVGT via multi-dataset joint optimization with only 2D supervision and 3D geometric regularization, achieving generalization across scenes with strong performance on mesh reconstruction, point cloud reconstruction, novel view synthesis, depth estimation, surface normal estimation, and camera pose estimation.

IVGT extracts continuous, coherent meshes directly from its SDF field. As a generalizable feed-forward method (no per-scene optimization), it surpasses most per-scene optimization baselines on ScanNet:

IVGT ranks second only to MonoSDF — which requires hours of per-scene optimization — while running in a single forward pass.

IVGT's depth maps (decoded directly from per-view features) achieve state-of-the-art pointmap reconstruction, outperforming VGGT on 7-Scenes and NRGBD despite pointmap reconstruction being a by-product of IVGT rather than its primary target:

We will release the code soon.

If you find this project helpful, please consider citing the following paper: