Classical Bundle Adjustment (BA) relies on precise metric initialization and known camera intrinsics, and traditional Structure-from-Motion (SfM) pipelines struggle to exploit modern dense matchers because rigid track-building heuristics break down under their inherent noise. We present Track-Free Dense Structure from Motion (TFD-SfM), which jointly optimizes camera extrinsics, focal lengths, and geometry from a strict cold start. Our central idea is to relax the rigidities of classical SfM into soft, probabilistic decisions under Negative Log-Likelihood (NLL) objectives. Instead of forcing matched pixels onto a single shared 3D point, we model each match as a 3D Gaussian whose variance is optimized jointly with the geometry. Unreliable matches take on large variance and contribute little to the loss, smoothing the optimization landscape so the solver converges even from identity cameras and uniform depths. A kinematic relative-pose parameterization stabilizes the cold start, and a dual-hypothesis mechanism resolves the mirror-symmetry ambiguity that otherwise traps object-centric scenes. Across DTU, Mip-NeRF 360, ETH3D, and IMC2021, TFD-SfM is competitive with both classical and learning-based SfM baselines and leads on ETH3D, recovering accurate camera poses, intrinsics, and dense geometry from unordered, uncalibrated image collections.
Each match is its own 3D Gaussian with a learned variance. From the cold start, the two Gaussians of one match move together and shrink onto the same point.
We optimize the relative pose of each spanning-tree edge of the view graph, and chain them into the camera poses.
Optimizing from a cold start, it is easy to fall into the mirror solution and never come back. We therefore optimize two worlds, the second started as the mirror image of the first, and keep the one with the higher likelihood.
Every scene of the four benchmarks, optimized for 10k iterations and replayed in 3D: drag to orbit, scroll to zoom, and scrub the slider through the iterations. Each frame is centered and scaled to fit, since the scene grows from unit depth, and 20k of the matches are shown. The numbers below each run are its relative rotation and translation accuracy at 5° (the paper reports 30k iterations).
Relative rotation and translation accuracy at 5° (RRA, RTA) and pose AUC at 10°, averaged over each benchmark. Bold: best, underlined: second best within each group.
| Method | DTU | Mip-NeRF 360 | ETH3D | IMC2021 | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| RRA | RTA | AUC | RRA | RTA | AUC | RRA | RTA | AUC | RRA | RTA | AUC | |
| With deep scene prior | ||||||||||||
| MASt3R-SfM | 99.6 | 98.1 | 85.1 | 87.1 | 82.2 | 67.0 | 78.9 | 74.3 | 66.5 | 75.1 | 69.2 | 67.1 |
| VGGT | 100.0 | 100.0 | 90.8 | 99.9 | 95.1 | 82.4 | 96.2 | 81.1 | 53.9 | 100.0 | 81.1 | 72.6 |
| Without deep scene prior | ||||||||||||
| COLMAP | 87.3 | 74.6 | 65.8 | 100.0 | 100.0 | 97.9 | 90.6 | 89.2 | 84.8 | 85.3 | 65.3 | 58.9 |
| COLMAP + DKM | 97.1 | 93.2 | 83.7 | 79.1 | 79.1 | 77.6 | 77.1 | 59.3 | 51.0 | 94.3 | 75.3 | 67.4 |
| COLMAP + RoMa | 96.6 | 93.2 | 83.7 | 83.2 | 83.2 | 81.6 | 72.6 | 60.1 | 52.0 | 95.1 | 78.0 | 69.7 |
| pOSE | 64.0 | 65.1 | 59.7 | 3.3 | 2.1 | 0.9 | 3.7 | 2.8 | 0.9 | 49.9 | 31.7 | 28.4 |
| PoVar | 69.7 | 69.9 | 64.8 | 4.8 | 2.4 | 0.9 | 2.7 | 1.0 | 0.4 | 48.6 | 28.0 | 23.9 |
| pOSE + DKM | 13.6 | 14.6 | 11.1 | 2.5 | 1.6 | 0.8 | 6.8 | 4.3 | 2.8 | 55.3 | 29.8 | 26.1 |
| PoVar + DKM | 20.2 | 20.9 | 15.8 | 4.4 | 1.8 | 0.8 | 6.0 | 2.0 | 1.6 | 42.3 | 20.3 | 18.3 |
| TFD-SfM (ours) | 97.9 | 83.1 | 75.0 | 99.6 | 98.5 | 94.0 | 96.9 | 90.1 | 85.0 | 71.2 | 39.5 | 37.1 |


Left: cost of one differentiable bundle-adjustment step against the correspondence count. Right: end-to-end runtime against the number of views on Mip-NeRF 360.
@inproceedings{chui2027tfdsfm,
title = {Track-Free Dense Structure from Motion},
author = {Chui, Pak Hong and Andrade-Loarca, Hector and Cremers, Daniel},
booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
year = {2027},
eprint = {2505.20858},
archivePrefix = {arXiv},
primaryClass = {cs.CV}
}