When the City Teaches the Car:
Label-Free 3D Perception from Infrastructure

Zhen Xu*1 Jinsu Yoo*1 Cristian Bautista*1 Zanming Huang1 Tai-Yu Pan2 Zhenzhen Liu3 Katie Z Luo4 Mark Campbell3 Bharath Hariharan3 Wei-Lun Chao1,5
1The Ohio State University 2Google 3Cornell University 4Stanford University 5Boston University
* Equal contribution
ECCV 2026 路 馃弲 CVPR 2026 DriveX Workshop (Best Poster Award)
Paper 路 Code
Nine roadside units across the city learn local detectors in parallel, each using its own stationarity to separate background from moving objects. An ego vehicle then collects their predictions while driving between them, learns a standalone detector offline from the pooled predictions, and finally perceives independently across the city with no infrastructure needed.
Can city infrastructure teach vehicles to perceive鈥攚ithout any human labels?
TL;DR

We ask whether city infrastructure can teach itself to see, then teach a car to see too. Stationary roadside units learn local 3D detectors from nothing but their own unlabeled streams, broadcast predictions as pseudo-labels to passing vehicles, and鈥攑ooled across a city鈥攖hose broadcasts are enough to train a standalone ego detector that needs no infrastructure at test time.

A Different Source of Supervision

Scaling autonomous driving to a new city almost always means repeating the same expensive loop: collect ego-side data, annotate it, retrain. At the same time, a different kind of sensor is quietly becoming common in cities鈥攔oadside units (RSUs), fixed in place, watching traffic from a stable vantage point with a wide field of view. We wanted to know whether the second thing could replace the first.

Concretely: can this infrastructure become a network of distributed local teachers? The property that makes this possible is stationarity. An RSU sees the same intersection, from the same angle, over and over. Averaged over time, what's persistent is background, and what moves is traffic鈥攁 distinction that emerges from repetition alone, with no human label in sight.

CIVET vision of a city equipped with many roadside units teaching an ego vehicle over four stages
Every RSU learns independently and in isolation; only the ego vehicle, moving through the city, ever sees the union of what they know.
Can many self-learning, location-specialized RSUs collaboratively supervise a generalist ego-vehicle detector?
Training

Fully label-free

Neither RSU nor ego training uses human annotations.

Teaching

Infrastructure-generated

Local RSU predictions become ego-centric pseudo-labels.

Deployment

Ego-only inference

The trained ego detector runs without RSUs or V2X links.

A Three-Stage Label-Free Pipeline

We kept each stage of the pipeline deliberately simple. The goal was not to push any single module as far as it could go, but to make the whole idea of infrastructure-taught perception concrete enough to measure, and simple enough to extend.

Three-stage CIVET pipeline from unsupervised RSU training through label broadcasting to standalone ego detector training
RSUs self-train offline, teach passing vehicles online, and produce an ego model that deploys independently.
Stage 1 路 Offline

RSUs learn local detectors

Persistence-point scores separate stable background from transient objects. DBSCAN and tracking refine proposals into pseudo-labels for each RSU.

82.8% average local RSU AP
Stage 2 路 Online

RSUs broadcast predictions

Nearby predictions are transformed into ego coordinates, filtered, and fused with distance-weighted NMS as the vehicle drives.

89.5% car pseudo-label recall
Stage 3 路 Offline

The ego distills city knowledge

Aggregated RSU predictions supervise a single ego detector trained offline across many locations and viewpoints.

Standalone at deployment

Stationarity alone is enough to separate signal from background

An RSU never sees a human label, so the persistence-point score has to do real work: over many observations of the same fixed scene, a point that keeps reappearing in roughly the same place is almost certainly background, and a point that doesn't is almost certainly part of something that moved. That asymmetry turns out to be sharp rather than fuzzy鈥攊n practice, static points collapse toward a score near 1.0 by orders of magnitude more than dynamic ones do, which is exactly the separation the pipeline needs to turn raw, unlabeled streams into usable pseudo-labels.

Persistence-point score distribution showing static points concentrated near a score of 1.0 and dynamic points spread toward 0, alongside a generated pseudo-label box tightly localizing a moving object
(a) The persistence-point score cleanly separates static background, clustered near 1.0, from dynamic objects spread toward 0. (b) The resulting pseudo-label boxes localize dynamic objects tightly, without a single human annotation.

Each RSU only ever learns its own corner

If a single RSU's detector generalized well to other locations, there would be little reason to run this pipeline at city scale鈥攐ne well-placed RSU could teach the whole town. It doesn't. Training a detector at one RSU and evaluating it at every other RSU's viewpoint shows strong performance only along the diagonal, each RSU's own trained view, and a sharp drop everywhere off it.

Heatmap of RSU detectors cross-evaluated at every other RSU's viewpoint, showing strong performance only on the diagonal and weak transfer to the ego viewpoint
Each row is a test set, each column a detector trained at one RSU. Performance is strong only on the diagonal鈥攁n RSU detector barely transfers to a different RSU's view, and transfers even less to the ego viewpoint (bottom row). This is precisely why the ego needs pseudo-labels pooled from many RSUs rather than relying on any single one.

The CIVET Dataset

Studying this pipeline requires a dataset most existing V2X benchmarks don't provide: dense RSU coverage across a whole geo-fenced area, with RSU and ego views of the same traffic both available and accurately posed. Most collaborative-perception datasets place a handful of roadside units at an isolated intersection or corridor, not a city's worth of them. CIVET is a controlled multi-agent dataset built on CARLA and V2Xverse, spanning four towns with different traffic compositions, object scales, road layouts, and RSU鈥揺go spatial relationships.

CIVET sample showing overlapping ego and roadside LiDAR together with four ego and four RSU camera views
The ego vehicle and an RSU observe the same traffic scene from complementary viewpoints.
4 CARLA towns
48 RSUs 路 12 per town
608k total frames
3 object categories
84kRSU frames per town
5kbroadcast frames per town
3kego test frames per town

Infrastructure-Taught Ego Detection Works

On the primary urban town, the fully label-free CenterPoint pipeline reaches 82.3% AP for vehicles鈥攖welve points below a detector trained on full ego ground truth, but built entirely without a human label.

First: does it work at all?

Before asking whether this scales, we needed to know whether it works at all. On a single urban town, 12 RSUs each learn their own local detector from nothing but repeated observation, broadcast pseudo-labels to the ego vehicle, and together supervise one standalone detector鈥攂uilt without a single human label.

Training supervision Car Ped. Cyc. Avg.
12 unsup. RSUs 82.3 79.3 68.5 76.7
Ego ground-truth 94.4 91.8 96.6 94.3
Single urban town 路 AP@0.5 for cars and AP@0.3 for pedestrians/cyclists.

Then: does it scale?

Feasibility on one town isn't the same as geographic scalability. We combine four towns鈥攖wo contrasting rural and urban environments, plus two more added specifically to test scale鈥攊nto a single unified deployment of 48 RSUs, and train one ego detector across all of them at once. Car AP holds at 82.7%, essentially unchanged from the single-town result, even though the combined test set now spans larger vehicles and sparser traffic in one town alongside denser pedestrian and cyclist traffic in another. More RSUs and more diversity did not erode what the pipeline had learned.

Training supervision Car Ped. Cyc. Avg.
48 unsup. RSUs 82.7 81.0 78.3 80.7
Ego ground-truth 91.0 90.6 95.3 92.3
Four-town combined 路 AP@0.5 for cars and AP@0.3 for pedestrians/cyclists.

Complementary to ego-centric unsupervised methods

Infrastructure supervision and ego-centric unsupervised learning draw on different signals鈥攐ne from a fixed RSU watching the same scene repeatedly, the other from an ego vehicle with its own viewpoint鈥攕o we'd expect the two to complement rather than duplicate each other. We find that combining our RSU pseudo-labels with Oyster's ego-centric unsupervised labels improves dynamic-object AP over either source alone.

Annotation source Dyn. AP
MODEST 18.0
Oyster 39.8
Ours 62.0
Oyster + Ours 64.9
Dyn. denotes dynamic-object AP.

Citation

@inproceedings{xu2026civet,
  title={When the City Teaches the Car:
         Label-Free 3D Perception from Infrastructure},
  author={Xu, Zhen and Yoo, Jinsu and Bautista, Cristian and
          Huang, Zanming and Pan, Tai-Yu and Liu, Zhenzhen and
          Luo, Katie Z and Campbell, Mark and Hariharan, Bharath and
          Chao, Wei-Lun},
  booktitle={European Conference on Computer Vision (ECCV)},
  year={2026}
}