Track-Free Dense Structure from Motion

1Technical University of Munich 2Munich Center for Machine Learning (MCML)
WACV 2027

poses prior scene prior intrinsic prior

Orange: estimated cameras

Green: ground-truth cameras

Abstract

Classical Bundle Adjustment (BA) relies on precise metric initialization and known camera intrinsics, and traditional Structure-from-Motion (SfM) pipelines struggle to exploit modern dense matchers because rigid track-building heuristics break down under their inherent noise. We present Track-Free Dense Structure from Motion (TFD-SfM), which jointly optimizes camera extrinsics, focal lengths, and geometry from a strict cold start. Our central idea is to relax the rigidities of classical SfM into soft, probabilistic decisions under Negative Log-Likelihood (NLL) objectives. Instead of forcing matched pixels onto a single shared 3D point, we model each match as a 3D Gaussian whose variance is optimized jointly with the geometry. Unreliable matches take on large variance and contribute little to the loss, smoothing the optimization landscape so the solver converges even from identity cameras and uniform depths. A kinematic relative-pose parameterization stabilizes the cold start, and a dual-hypothesis mechanism resolves the mirror-symmetry ambiguity that otherwise traps object-centric scenes. Across DTU, Mip-NeRF 360, ETH3D, and IMC2021, TFD-SfM is competitive with both classical and learning-based SfM baselines and leads on ETH3D, recovering accurate camera poses, intrinsics, and dense geometry from unordered, uncalibrated image collections.

Method

TFD-SfM system overview
From unordered images, three stages: (1) view-graph construction with LoFTR (solid: spanning-tree edges, dashed: auxiliary edges), (2) dense correspondences with DKM, and (3) the TFD-SfM optimization from a strict cold start, which outputs intrinsics, absolute poses and a 3D reconstruction.

Key idea 1: Gaussian landmarks, not tracks

Each match is its own 3D Gaussian with a learned variance. From the cold start, the two Gaussians of one match move together and shrink onto the same point.

Toy example: the two Gaussians of one wide-baseline match start far apart and meet while the ring of cameras is recovered

Key idea 2: Relative poses on a spanning tree

We optimize the relative pose of each spanning-tree edge of the view graph, and chain them into the camera poses.

ETH3D delivery_area from a cold start: absolute poses stay broken, relative poses on a spanning tree close the camera loop

Key idea 3: Two worlds against the mirror ambiguity

Optimizing from a cold start, it is easy to fall into the mirror solution and never come back. We therefore optimize two worlds, the second started as the mirror image of the first, and keep the one with the higher likelihood.

The two reconstructions of DTU scan82 on a turntable: convex and kept, concave and discarded

Results

Relative rotation and translation accuracy at 5° (RRA, RTA) and pose AUC at 10°, averaged over each benchmark. Bold: best, underlined: second best within each group.

MethodDTUMip-NeRF 360ETH3DIMC2021
RRARTAAUCRRARTAAUCRRARTAAUCRRARTAAUC
With deep scene prior
MASt3R-SfM99.698.185.187.182.267.078.974.366.575.169.267.1
VGGT100.0100.090.899.995.182.496.281.153.9100.081.172.6
Without deep scene prior
COLMAP87.374.665.8100.0100.097.990.689.284.885.365.358.9
COLMAP + DKM97.193.283.779.179.177.677.159.351.094.375.367.4
COLMAP + RoMa96.693.283.783.283.281.672.660.152.095.178.069.7
pOSE64.065.159.73.32.10.93.72.80.949.931.728.4
PoVar69.769.964.84.82.40.92.71.00.448.628.023.9
pOSE + DKM13.614.611.12.51.60.86.84.32.855.329.826.1
PoVar + DKM20.220.915.84.41.80.86.02.01.642.320.318.3
TFD-SfM (ours)97.983.175.099.698.594.096.990.185.071.239.537.1
Qualitative comparison
Estimated (red) and ground-truth (grey) cameras with the point clouds on DTU, Mip-NeRF 360, ETH3D and IMC2021. Left to right: input, COLMAP, COLMAP+DKM, COLMAP+RoMa, MASt3R-SfM, VGGT and TFD-SfM.
Runtime per step
Runtime vs. number of views

Left: cost of one differentiable bundle-adjustment step against the correspondence count. Right: end-to-end runtime against the number of views on Mip-NeRF 360.

BibTeX

@inproceedings{chui2027tfdsfm,
  title     = {Track-Free Dense Structure from Motion},
  author    = {Chui, Pak Hong and Andrade-Loarca, Hector and Cremers, Daniel},
  booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  year      = {2027},
  eprint    = {2505.20858},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV}
}