Preserving Guidance in Cost-Volume Retrieval
under Extreme LiDAR Sparsity in Iterative Stereo

Jinsu Yoo1 Sooyoung Jeon1 Zanming Huang1 Tai-Yu Pan1,2 Wei-Lun Chao1,3
1The Ohio State University 2Google 3Boston University
IROS 2026
Paper · Code
300 LiDAR points
Dense depth prediction under extreme LiDAR sparsity
TL;DR

Inside RAFT-Stereo, extremely sparse LiDAR doesn't just carry less information—it can behave like noise and get filtered away entirely. We trace this to how the model retrieves from its cost volume, and fix it by pre-filling the initial disparity map before retrieval happens, while a separately filtered path strengthens early fusion on top.

Dense Depth from a Few Hundred Points

GRAFT-Stereo combines stereo images with extremely sparse LiDAR—sometimes a few hundred points, sometimes only four scan lines—to produce dense depth. The sequences below let us compare naive RAFT-Stereo guidance, SDG-Depth, and GRAFT-Stereo side by side, along with where each one goes wrong.

300 LiDAR points
Uniformly sampled sparse guidance
4-beam LiDAR
Four scan lines
4-beam LiDAR
Four scan lines

Consistent Gains as LiDAR Gets Sparser

We track frame-wise depth error as LiDAR density changes, under two different sparsity regimes: uniformly sampled points and beam-sampled scan lines. Across both, GRAFT-Stereo improves over stereo-only depth, and the gain remains substantial even at the sparsest inputs we test—300 points, or just 4 beams.

Uniform point sampling

300 / 1,000 / 3,000 points
300 / 1,000 / 3,000 points

Beam sampling

4 / 8 / 16 beams
4 / 8 / 16 beams

Why Sparse Guidance Disappears

To understand why sparse LiDAR guidance fails, it helps to look at what RAFT-Stereo actually does with it. At every iteration, the model retrieves a local region from its correlation volume, using its current disparity estimate as the lookup coordinate. A handful of accurate LiDAR disparities sitting inside an otherwise all-zero map creates sharp, abrupt jumps in that lookup coordinate. From the model's perspective this looks like high-frequency noise, and its recurrent convolutions do to it exactly what they are built to do with noise: smooth it away.

Cost-volume construction and retrieval with ground-truth, zero, sparse, and pre-filled initial disparity maps
Sparse initialization produces a Laplacian of 0.65; pre-filling reduces it to 0.33, near ground-truth retrieval at 0.28.
Key insight

Sparse LiDAR is not merely incomplete guidance—inside iterative stereo, its useful signal can be actively suppressed by the model's own smoothing. Once we understood that, the fix followed directly: smooth the lookup coordinates themselves, and the guidance survives.

Pre-Filling Before Retrieval

GRAFT-Stereo does not touch RAFT-Stereo's iterative core. Instead, it changes how sparse depth reaches cost-volume construction and retrieval, and it treats two fusion pathways differently, because each needs pre-filled depth for a different reason.

GRAFT-Stereo pipeline with depth pre-filling, early fusion, late fusion, and iterative prediction
Coarse image-processing pre-fill stabilizes late fusion; confidence-filtered learned pre-fill supplies accurate geometry for early fusion.

Late fusion: fill for continuity

Here the goal is not accurate depth completion—it's a stable lookup coordinate. We use IP-Basic, a training-free densification method, to fill in sparse depth before disparity initialization. That is the core fix, and on its own it recovers most of the guidance that sparse LiDAR otherwise loses inside iterative stereo.

Bonus

Early fusion: project for correspondence

A second, independent path strengthens early fusion. Depth is back-projected into 3D and re-projected into both stereo views, and the resulting XYZ coordinates are concatenated with RGB before feature encoding. Dense completion is riskier here, since one incorrect match can actively mislead the network, so this path keeps only the top 1,000 confidence-ranked points. Combined with late fusion, it adds a further improvement on top.

Consistent Improvements in Depth Accuracy

Across both uniform and beam-sampled LiDAR settings, GRAFT-Stereo consistently improves depth accuracy over prior LiDAR-guided stereo methods. At the sparsest setting we test—300 uniformly sampled points—combining both fusion paths reduces KITTI RMSE from 929.5 mm with naive guidance down to 739.5 mm. The same trend carries over to VKITTI2 and MS2, where GRAFT-Stereo reaches 877.5 mm and 781.4 mm MAE respectively, again from just 300 points.

Qualitative KITTI depth comparison between naive RAFT-Stereo guidance, SDG-Depth, and GRAFT-Stereo
At 300 points, GRAFT-Stereo lowers average depth error to 0.172 m, compared with 0.355 m for naive guidance and 0.239 m for SDG-Depth.

KITTI Depth Completion

Method 300 pts† 1000 pts† 3000 pts† 4-beam§ 8-beam§ 16-beam§
EG-Depth 829.7 / 283.6 822.6 / 263.0 764.0 / 242.5 900.0 / 307.3 851.2 / 277.8 858.9 / 276.7
SDG-Depth 883.9 / 307.2 816.7 / 290.7 763.8 / 259.2 876.8 / 326.9 789.2 / 279.5 792.6 / 279.7
RAFT-Stereo + gd 929.5 / 297.0 835.0 / 274.8 827.5 / 243.1 863.8 / 297.4 874.7 / 293.6 873.3 / 288.5
GRAFT-Stereo (late) 817.7 / 237.5 747.0 / 210.1 717.8 / 193.9 817.9 / 255.1 803.0 / 243.2 808.0 / 241.1
GRAFT-Stereo (full) 739.5 / 220.2 716.5 / 208.3 675.6 / 186.2 779.0 / 240.2 774.4 / 235.1 767.8 / 230.5
RMSE / MAE in millimeters · † uniform sampling · § beam sampling · gd: naive LiDAR guidance

Citation

@inproceedings{yoo2026preserving,
  title={Preserving Guidance in Cost-Volume Retrieval under
         Extreme LiDAR Sparsity in Iterative Stereo},
  author={Yoo, Jinsu and Jeon, Sooyoung and Huang, Zanming and
          Pan, Tai-Yu and Chao, Wei-Lun},
  booktitle={IEEE/RSJ International Conference on
             Intelligent Robots and Systems (IROS)},
  year={2026}
}