123D Gives Autonomous-Driving Datasets a Common Language — Here Is What It Establishes and Where It Stops
6 min read Updated
Explainer of 123D: Unifying Multi-Modal Autonomous Driving Data at Scale. SentX was not involved in this research.
123D is a data-infrastructure contribution, not a new perception or planning algorithm. It establishes one thing clearly: a common representation and API over the patchwork of autonomous-driving datasets. The integration covers eight real datasets — nuScenes, Waymo Open Perception, Argoverse 2 Sensor, PandaSet, KITTI-360, Waymo Open Motion, nuPlan and PAI-AV — plus the synthetic L3AD set recorded in CARLA. Downstream code reads one interface regardless of where the data came from.
Source: 123D: Unifying Multi-Modal Autonomous Driving Data at Scale
Its findings on model transfer are bounded. The authors report empirical benefits from data mixing in their defined experiments, and those results do not establish universal transfer gains. Radar, richer semantic annotations, web-scale ingestion, continuous streaming, and non-standard platforms sit outside the described framework, and the paper says so.
One representation for every dataset
Every recording is converted into timestamped event streams. An event stream is an ordered sequence of records in which each record carries its own time. There is one stream per modality: cameras, with support for multiple projection models; lidar; ego-states, meaning the vehicle's own position and motion; 3D bounding boxes; traffic-light states; and user-defined types.
Source: 123D: Unifying Multi-Modal Autonomous Driving Data at Scale
Streams carry no prescribed sampling rate, and each event keeps its own timestamp. Static metadata such as vehicle dimensions and sensor calibration lives in the file's schema — the structural definition of the file — rather than being repeated row by row.
A separate synchronization table precomputes which rows across modalities refer to the same instant. The strategy is configurable: preserve the original keyframes, choose a reference modality, or resample everything to a target rate. The access layer can also bypass the table for arbitrary-timestamp reads. That combination serves both synchronous consumers, who want aligned frames, and asynchronous ones, who read each modality on its own schedule. Original annotations are preserved rather than re-labeled.
Source: 123D: Unifying Multi-Modal Autonomous Driving Data at Scale
Detection transfer and planning
The paper evaluates two worked applications: detection transfer and planning.
In the detection study, standard detectors — PETR and BEVFormer-S, both using ResNet50 encoders as their feature extractors — were trained in two ways. They were trained individually on five selected sources: nuScenes, Waymo Open Perception, Argoverse 2 Sensor, nuPlan and CARLA. They were also trained on a uniform mixture of the same five, in which each source contributes equally. Training ran under a fixed 30,000-frame budget so that diversity is isolated from raw volume.
Source: 123D: Unifying Multi-Modal Autonomous Driving Data at Scale
PandaSet, KITTI-360 and PAI-AV were excluded from training and used for held-out transfer evaluation, meaning the models were tested on data they never saw during training. Storage constraints led the authors to use the nuPlan-mini and PAI-AV-NCore subsets.
Source: 123D: Unifying Multi-Modal Autonomous Driving Data at Scale
The detection evaluation is deliberately narrow. It scores the vehicle class only, within a 50-meter radius, using a modified NDS. NDS, the NuScenes Detection Score, is a composite metric for 3D detection; the modification drops the velocity and attribute error terms.
Source: 123D: Unifying Multi-Modal Autonomous Driving Data at Scale
The authors' own conclusion is measured. Cross-domain and vehicle transfer remains challenging for both perception and behavior tasks. Simple data mixing reduces the gap to some extent. Sim-to-real transfer — moving from simulated data to real data — proves no easier than cross-rig transfer between real datasets, even when the simulator replicates the target rig. A rig is the full sensor configuration mounted on a vehicle.
Source: 123D: Unifying Multi-Modal Autonomous Driving Data at Scale
The planning example is built on reinforcement learning, in which an agent improves a decision policy through trial and reward. It lands in the same place: behavior transfer stays difficult, with partial benefit from mixing. All reported numbers are the authors' measurements on their own harness, and this article interprets the paper's inspected sections rather than reproducing the results.
Source: 123D: Unifying Multi-Modal Autonomous Driving Data at Scale
Three documented limitations
Three limitations are documented in the paper itself.
Coverage is limited to common sensor modalities. Additional sensors such as radar, and richer semantic annotations, are named development targets rather than delivered features.
Source: 123D: Unifying Multi-Modal Autonomous Driving Data at Scale
There is no explicit support for web-scale datasets or for continuous data streaming. The authors note that Arrow's cloud-access capabilities were only preliminarily tested. Arrow is the columnar data format underlying the storage.
Source: 123D: Unifying Multi-Modal Autonomous Driving Data at Scale
The framework handles standard vehicles. Trucks, mobile robots and other platforms are targeted for later expansion.
Source: 123D: Unifying Multi-Modal Autonomous Driving Data at Scale
The work is an author-submitted arXiv paper. The description in this article reflects the sections available in that version: the abstract, the framework description, experiments 4.3 and 4.4, the conclusion and the limitations.
Checking one plausible-sounding claim
Consider a plausible-sounding summary sentence: "123D supports continuous live data streaming." The paper supports the machinery the sentence leans on — independent timestamped event streams with no prescribed rate. But that is a statement about how recordings are stored and indexed, not about ingesting a live feed. Each event keeping its own timestamp says nothing about whether the system can accept data arriving continuously in real time.
The distinction matters because the two sections that settle it are exactly the ones a skimmer skips. The framework description shows streams as a static, timestamped representation with a precomputed synchronization table. The limitations section explicitly names continuous data streaming as unsupported, with Arrow's cloud access only preliminarily tested. A summary claim survives contact with the paper only if you can locate the sentence that earns it.
Source: 123D: Unifying Multi-Modal Autonomous Driving Data at Scale
That habit is worth having for any model-assisted reading. SentX's research summarizer accepts supplied paper text and answers questions about methods, results and limitations, and its own page directs users to check important claims against the source. Used that way, it can prepare a draft source-checking note from the paper's public text. The manual review against the actual sections is still yours to do, and the tool is not a driving or perception product validating anything here.
Source: SI Research Paper Summarizer
One reason for the caution: SentX's retained full-paper demonstration on a different paper (MIST) kept two substantive errors in its first answer. One was a false claim of increases for all memory-system and model pairs, contradicted by Table 7. The other was an unsupported relative-percentage reading of the abstract. That demonstration concerns a different paper and establishes no result about 123D. It exists to show why summary claims need checking in the first place.
Source: SentX full-paper reading demonstration: retained prompt, first answer and claim review
Deciding whether to build on 123D
The following is editorial guidance based on this paper, not a deployment result.
- Map your task to the described coverage. If you are running existing detection or planning pipelines across several heterogeneous driving datasets with common sensors — cameras, lidar, ego-state, 3D boxes, traffic-light state — and standard vehicles, the framework addresses that gap.Source: 123D: Unifying Multi-Modal Autonomous Driving Data at Scale
- Check each dependency against the limitations. Radar, richer semantic annotations, web-scale ingestion, continuous streaming, trucks or other non-standard platforms: the paper names these as future work or leaves them out, so any claim that 123D covers them today is unsupported by this source.Source: 123D: Unifying Multi-Modal Autonomous Driving Data at Scale
- Treat every number as author-reported. Results come from the authors' own harness on the named subsets, with the narrow detection metric scope described above; if your decision depends on certified accuracy, that question is unanswered by this paper.Source: 123D: Unifying Multi-Modal Autonomous Driving Data at Scale
- Verify before you depend. For any specific feature you plan to rely on, open the framework description and the limitations section and find the supporting sentence — the same check applied to the streaming claim above.Source: 123D: Unifying Multi-Modal Autonomous Driving Data at Scale
The honest bottom line is the paper's own: consolidation makes mixing data easier, but teaching models to transfer across domains remains the hard part. 123D shrinks the plumbing problem, not the learning problem.
Source: 123D: Unifying Multi-Modal Autonomous Driving Data at Scale
Sources
- 123D: Unifying Multi-Modal Autonomous Driving Data at Scale — arXiv (author-submitted research)
Supported claims and limits
- 123D provides a common API and representation for multimodal autonomous-driving datasets.
- Each sensor modality is stored as a timestamped event stream, enabling synchronous and asynchronous access.
- The reported integration covers eight real datasets and a synthetic dataset, with detection-transfer and planning examples.
- Evidence limits: Current coverage emphasizes common sensors and ordinary vehicles.
- Evidence limits: Additional radar and semantic annotations remain development targets.
- Evidence limits: Explicit web-dataset and streaming support were not available in the described framework.
- The authors report bounded benefits from data mixing in their defined detection and planning experiments; these are not universal transfer gains.
- Detection: PETR and BEVFormer-S with ResNet50, five training sources and their uniform mixture, 30,000 frames, vehicle class within 50 m and modified NDS; three additional datasets are held out.
- Inspected scope: Abstract; framework; experiments 4.3 and 4.4; conclusion; limitations. This is a source review, not an independent reproduction.
- SentX full-paper reading demonstration: retained prompt, first answer and claim review — SentX
Supported claims and limits
- One original Victoria reply through normal SentX defaults was given all 25 pages as extracted plain text in one user message. It was not a PDF upload or image/figure parsing test. Nine reference topics were fixed before generation; the first completed answer was retained without a selected rerun.
- The response reproduced the requested Table 3 comparison and all eight Table 6 means with displayed uncertainty values, distinguished strict sycophancy from overall error rate, and identified the non-significant Appendix L per-question comparison.
- Two substantive errors are retained prominently: a false claim of increases for all memory-system/model pairs, contradicted by Table 7; and an unsupported relative-percentage interpretation of the abstract. Smaller notes cover realism aggregation, the unstated Table 6 dispersion label, and an omitted metric-denominator detail.
- This is a single first-party demonstration. No general accuracy score, benchmark, independent evaluation, product superiority, autonomous discovery, upload reliability, visual reading or cross-session recall is established.
- SI Research Paper Summarizer — SentX
Supported claims and limits
- The public workflow accepts supplied paper text or a PDF and invites questions about methods, results and limitations. Important claims need checking against the source. This demonstration used text only.