Deep Bundle Adjustment

1Technical University of Munich 2Munich Center for Machine Learning (MCML)
Under review

pose initialization known intrinsics point tracks

The difference. Classical BA: Δs_k = −(JᵀJ)⁻¹Jᵀr(s_k),
        analytic solve, needs a good init; s: cameras + 3D points from tracks. DeepBA (ours): Δs_k = GNN_θ(s_k, G),
        learned, weight-shared, trivial init; s: poses + focal + depth, no tracks.

Matches only, no images or depth prior: from identity cameras and unit depths, 100 iterations of one 2.04M-parameter network.

Orange: estimated cameras, green: ground-truth cameras

Above each clip: six input views with some of their 2D matches, and below them DeepBA's current depth (bright = near).

Abstract

We present DeepBA (Deep Bundle Adjustment), a graph neural network (GNN) that recovers camera poses, intrinsics, and scene structure from 2D matches across an arbitrary number of views. Rather than solving a non-linear least-squares problem per scene, DeepBA amortizes bundle adjustment into a learned graph network that iteratively refines its estimate from geometric residuals. Where classical SfM computes its updates analytically and needs careful initialization, DeepBA learns them and converges from a cold start. It is trained with scale-invariant supervision (ground-truth rotation, translation direction, and intrinsics, plus reprojection self-consistency), so datasets with arbitrary metric units can be mixed without per-scene normalization. DeepBA is decoupled from the matcher. At 2.04M parameters, it outperforms prior learned SfM from matches on scene reconstruction and on most pose metrics, is competitive with classical SfM at moderate accuracy thresholds and overtakes it in wall time as the view count grows, and approaches monolithic image-to-geometry transformers such as VGGT (~1.2B parameters) with greater modularity.

Method

DeepBA architecture
Left: one refinement iteration. The scene state and the view graph are encoded into camera, pair and match features, re-initialized every iteration, mixed by L message-passing blocks and decoded into an update of the state. The block is weight-shared across the K iterations. Right: one block, with its three message-passing steps.

Key idea 1: A lightweight GNN for learned updates that mimic classical bundle adjustment

Each iteration reads the current state and predicts the next update of poses, focal lengths and depth. The same small network runs for every iteration and every scene. With 2.04M parameters, it is about 590× smaller than VGGT (~1.2B).

Key idea 2: A track-free state

Following the track-free idea of TFD-SfM, the state is one pose, one focal length and one depth map per image. Pairwise matches are enough, so no tracks have to be built, and any matcher can be plugged in.

Key idea 3: Scale-invariant supervision

Rotations, translation directions and intrinsics are supervised, plus reprojection self-consistency. Datasets with arbitrary metric units are mixed without per-scene normalization.

Results

Multi-view pose accuracy: relative rotation and translation accuracy at 10° (RRA, RTA). The paper also reports 3° and 5°. Bold: best, underlined: second best within each group. Grey: image-to-geometry methods, for reference. †: ground-truth intrinsics and a final classical bundle adjustment. ‡: DeepBA's eLoFTR correspondences.

MethodDTUMip-NeRF 360ETH3DIMC2021
RRARTARRARTARRARTARRARTA
Classical SfM
COLMAP (SIFT)87.382.6100.0100.096.796.495.789.3
GLOMAP (SIFT)87.278.6100.0100.096.691.596.088.2
COLMAP (eLoFTR)‡100.099.5100.0100.076.975.597.291.5
GLOMAP (eLoFTR)‡97.996.8100.0100.061.656.997.891.2
Learned image-to-geometry
MASt3R-SfM100.099.586.984.079.074.697.587.7
VGGT100.0100.0100.098.596.979.6100.091.9
Learning-based SfM from matches
RESfM (SIFT)†22.026.012.012.14.53.884.681.2
GASfM (SIFT)†40.844.141.144.312.110.786.375.3
RESfM (eLoFTR)†, ‡36.839.311.312.019.619.484.879.8
GASfM (eLoFTR)†, ‡38.842.454.757.627.229.492.290.8
DeepBA100.099.673.586.569.149.199.087.3

Camera pose on CO3Dv2 and RealEstate10K, 10 frames per scene. Time: seconds per scene on one H100, as matching + solver for the methods that take matches.

MethodCO3Dv2RealEstate10KParamsTime
RRA@15RTA@15AUC@30AUC@30
Classical SfM
COLMAP (SIFT)86.379.974.946.5–~2+3 s
GLOMAP (SIFT)86.676.271.171.7–~2+2 s
COLMAP (eLoFTR)‡89.783.577.080.9–~7+11 s
GLOMAP (eLoFTR)‡82.172.367.672.7–~7+5 s
Learned image-to-geometry
PoseDiffusion87.780.071.639.639.0M~115 s
DUSt3R98.382.172.979.9577.8M~7 s
MASt3R99.292.184.791.1695.3M~9 s
VGGSfM94.385.880.589.5281.2M~10 s
VGGT99.392.185.185.3~1.2B~0.2 s
Learning-based SfM from matches
RESfM (SIFT)†62.755.551.148.90.7M~2+13 s
GASfM (SIFT)†79.173.968.675.4145.2M~2+0.5 s
RESfM (eLoFTR)†, ‡61.154.048.263.70.7M~7+13 s
GASfM (eLoFTR)†, ‡79.977.371.282.8145.2M~7+2 s
DeepBA97.489.278.684.02.04M~7+3 s
COLMAPRESfMGASfMDeepBA
DTU (49)Mip-NeRF 360 (194)ETH3D (44)IMC2021 (25)
Qualitative comparison: COLMAP, RESfM, GASfM and DeepBA on DTU, Mip-NeRF 360, ETH3D and IMC2021
Qualitative comparison. Rows: DTU scan82, Mip-NeRF 360 bicycle, ETH3D delivery_area and IMC2021 london_bridge (a 25-view bag). The number after each dataset is that scene's image count. 3D points, estimated cameras in green and ground-truth cameras in grey.
Runtime against the number of images
Left: time after matching on the first N images of Mip-NeRF 360 bonsai, with every method using the same eLoFTR matches on one H100 (matching time dashed). Right: time per iteration against the number of matches.
Pose accuracy against the refinement iteration
Pose accuracy against the refinement iteration (RRA@10° left, RTA@10° right) on Mip-NeRF 360. Light curves: single scenes. Solid black: the mean, dotted: the mean without room.

Localization: DeepBA vs PnP and HLoc

DeepBA localizes a new image with the same refinement step it uses for reconstruction. The map stays frozen, and only the query camera and its depth are refined, so DeepBA needs neither PnP nor the query's intrinsics.

One query camera, iteration by iteration

CO3Dv2 hydrant, query frame 10
CO3Dv2 hydrant, query frame 42

One held-out image registered against the frozen 25-image DeepBA map (blue cameras and points). The query camera (green) starts at the mean pose of the map cameras (their chordal-mean rotation and mean centre), the same start for every query, with no PnP estimate. DeepBA's update moves it, together with its lifted 3D points, onto the map. After 10 iterations the rotation error is 3.5° (frame 10, from 109°) and 4.0° (frame 42, from 19.1°). Grey: the ground-truth camera.

Same map, same queries: DeepBA vs PnP and HLoc

Four localizers register the same held-out queries against the same frozen DeepBA map, so only the registration step differs.

CO3Dv2 hydrant (object-centric orbit)
ScanNet scene0042_00 (indoor)
CO3Dv2 toytruck
ScanNet scene0000_00
CO3Dv2 motorcycle
ScanNet scene0011_00

A 25-image DeepBA map, frozen, and 100 query frames registered one by one. Query-camera trajectories against the ground truth (grey): DeepBA (green), PnP+RANSAC (orange) and HLoc (cyan), both given the ground-truth query intrinsics, and the focal-solving PnPf+RANSAC (pink). The table gives the final median camera-centre error (ground-truth units, metres on ScanNet; HLoc localizes the 75 queries outside the map) and the map's own error (ATE) for all eight scenes we ran. The first two clips are the scenes of the paper. The other four are the two best DeepBA maps per dataset (lowest map ATE) besides those. Bold: best, underline: second best per scene (among the four localizers).

SceneMap ATEPnP+RANSAC + GT KPnPf+RANSACHLoc + GT KDeepBA
CO3Dv2 hydrant0.2160.3420.2540.5820.185
CO3Dv2 toytruck0.1260.1270.1840.2220.167
CO3Dv2 motorcycle0.2040.2540.2640.3730.240
CO3Dv2 teddybear*7.9697.5097.6197.6747.628
ScanNet scene0042_000.0640.0550.0660.0750.059
ScanNet scene0000_000.0460.0470.0900.0580.053
ScanNet scene0050_000.2010.1740.2080.1890.199
ScanNet scene0011_000.0860.0570.0830.0850.061

*DeepBA's 25-image teddybear map fails (half of the orbit is flipped, map ATE 7.97), so every localizer registered against it fails too.

Camera localization on 7-Scenes: median translation and rotation error (cm/°) under the protocol of HLoc. The first row is HLoc as published, with its own map from the full ~4,000-image TrainSplit. The other four register the same queries against the same frozen DeepBA map, built from 200 TrainSplit images. Bold/underlined: best/second best among these four, per column. The ground-truth rotations are projected onto SO(3) before scoring, since they are not exactly orthonormal.

MethodIntrinsics neededChessFireHeadsOfficePumpkinKitchenStairs
HLoc, ~4,000-image mapyes2/0.792/0.872/0.923/0.915/1.124/1.256/1.62
Same map, same queries: only the registration step differs
HLoc, DeepBA mapyes11/1.8810/1.896/1.909/4.7516/3.8510/4.3737/17.86
PnP+RANSAC + GT Kyes12/1.9111/1.676/2.0410/4.6815/3.7110/4.2137/17.39
PnPf+RANSACno6/1.665/1.673/2.0111/4.8410/3.759/3.9650/17.35
DeepBAno3/1.663/1.672/1.668/4.808/3.866/4.3030/18.30

BibTeX

@article{chui2026deepba,
  title   = {Deep Bundle Adjustment},
  author  = {Chui, Pak Hong and Andrade-Loarca, Hector and Cremers, Daniel},
  journal = {arXiv preprint arXiv:2611.18256},
  year    = {2026},
  eprint  = {2611.18256},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV}
}