DARP: A Calibrated Dual-Arm RGB-D-IR Dataset for Multi-View Robotic Perception*These authors contributed equally to this work.

RGB-D datasets have been widely used for object recognition, manipulation, and 3D perception. The Washington RGB-D Object Dataset provides multi-view RGB-D observations of physical objects and established an early benchmark for object-centric RGB-D recognition [12]. BigBIRD further increased viewpoint coverage for real object instances through dense RGB-D acquisition [18], while the YCB Object and Model Set introduced a standardized collection of physical objects and geometric models for robotic manipulation research [3]. Large-scale resources such as ShapeNet and ScanNet address complementary settings through digital 3D models and reconstructed indoor RGB-D scenes, respectively [4, 6].

More robotics-oriented datasets directly connect visual sensing with manipulation. GraspNet-1Billion provides RGB-D observations and large-scale grasp annotations for general object grasping [10]. ROBI focuses on multi-view sensing of reflective objects in robotic bin-picking scenarios and highlights the difficulty of acquiring reliable geometry from challenging surfaces [23]. These datasets provide valuable resources for robotic perception, but their acquisition goals differ from continuously recording complementary observations from two independently moving eye-in-hand manipulators together with synchronized robot states.

Partial observations naturally arise when objects are viewed from limited directions or contain self-occluded surfaces. Learning-based methods have therefore explored how incomplete point clouds can be mapped to more complete shape representations. PCN introduced an encoder–decoder architecture for point-cloud completion [25], while TopNet proposed a hierarchical structural decoder [19]. More recent methods such as PoinTr and SnowflakeNet use Transformer-based or progressive point-generation mechanisms to model missing geometry from partial observations [24, 22]. ProxyFormer and AnchorFormer further develop Transformer-based completion using missing-part reasoning and discriminative geometric structures [13, 5].

Generative approaches have also been investigated for point-cloud completion, including diffusion-based methods such as PCDreamer and SuperPC [21, 9]. Recent work has additionally emphasized that performance on synthetic or controlled completion benchmarks does not necessarily translate directly to real sensor observations [16]. These methods motivate future learning applications of the proposed dataset; however, the reconstruction presented in this paper is strictly geometric and does not infer unseen object surfaces.

Collaborative perception uses observations from multiple agents to reduce the limitations of an individual viewpoint. Reily and Zhang studied joint view and feature selection for collaborative multi-robot perception [17], while Zhou et al. explored graph neural networks for combining perceptual information across robots [26]. Multi-Robot Scene Completion further considered collaborative perception as a task-agnostic representation problem in which information from multiple robots contributes to a more complete scene description [14]. Recent datasets such as CU-Multi also provide synchronized observations for multi-robot collaborative perception research [1].

The setting considered in this work is more object-centered and close-range. Two fixed-base manipulators independently move eye-in-hand cameras around the same tabletop object, producing complementary partial observations while robot kinematics provide the pose of each sensor measurement. This makes the dataset suitable for studying how embodied viewpoints can be combined at the level of object geometry as well as for later feature-level collaborative perception.

Reliable fusion of observations from moving robot-mounted cameras requires accurate spatial calibration. The hand-eye calibration problem estimates the rigid transformation between the robot end-effector and the attached camera, with classical solutions proposed by Tsai and Lenz and later by Daniilidis [20, 8]. ChArUco patterns combine coded ArUco markers with chessboard corners and provide feature correspondences that can be used for intrinsic and extrinsic calibration [2]. In the proposed system, ChArUco-based calibration is used to establish the camera-to-wrist and inter-base relationships required to place observations from both arms in a common metric frame.

Multi-view RGB-D reconstruction methods such as KinectFusion and BundleFusion demonstrate how repeated depth observations can be integrated into coherent geometry when reliable camera poses are available [15, 7]. Our geometric pipeline follows the same general principle of transforming repeated measurements into a shared frame, but the camera poses are derived primarily from robot calibration, synchronized joint states, and forward kinematics rather than estimated through visual tracking.

Overall, existing work provides strong resources for RGB-D perception, robotic manipulation, point-cloud completion, collaborative perception, and multi-view reconstruction. The proposed dataset connects these areas through a dual-arm eye-in-hand acquisition setting that preserves multimodal observations, robot motion, and calibration information in a shared metric coordinate system.

The acquisition platform consists of two 6-DoF Unitree Z1 Pro manipulators, denoted as arm110 (one on the left) and arm111 (one on the right), mounted on opposite sides of a shared tabletop workspace, as shown in Fig. 2. The two robot bases remain fixed throughout data collection and are oriented approximately toward one another, creating an overlapping manipulation region in which both arms can observe the same physical object from different directions.

Each manipulator carries an Intel RealSense RGB-D camera rigidly attached near the gripper flange in an eye-in-hand configuration, as shown in Fig. 1. Camera serial 348522076490 is associated with arm110, while camera serial 405622074504 is associated with arm111. This mapping is fixed because every camera observation must later be transformed through the kinematic chain of its corresponding arm. The cameras continuously record color, depth, and stereo infrared information at $640\times 480$ resolution and 30 Hz. Color is stored in BGR8 format, depth in Z16 format, and the left and right infrared channels in Y8 format. Accelerometer and gyroscope streams are also retained in the native recordings, although they are not used by the geometric reconstruction presented in this work. Camera intrinsics, inter-stream calibration, and depth scale remain embedded in the RealSense recordings and are read from each device or bag file during processing rather than being hard-coded.

The two manipulators are separated by approximately 1.3986 m. Their relative orientation is close to antiparallel, with an approximately $178.4^{\circ}$ rotation about the world $z$ axis. This arrangement allows the two eye-in-hand cameras to observe complementary regions of the object surface while remaining within their respective reachable workspaces.

Because both cameras move with their corresponding manipulators, multi-view observations can only be combined reliably when the relationship between the camera, robot wrist, robot base, and shared world frame is known. We define the world frame as $\mathcal{F}_{W}$, the base frame of arm $i$ as $\mathcal{F}_{B_{i}}$, the wrist or gripper frame as $\mathcal{F}_{G_{i}}$, and the camera optical frame as $\mathcal{F}_{C_{i}}$. A ChArUco board is used as the common calibration target, as shown in Fig. 3. ChArUco patterns combine coded markers with accurately localized chessboard corners and provide feature correspondences for geometric calibration [2]. Multiple observations of the calibration target are collected from different robot configurations to estimate the rigid camera-to-wrist transformation for each eye-in-hand sensor. The same calibration framework is used to establish the spatial relationship between the two fixed robot bases.

For each arm, two static transforms are retained, the wrist-to-camera transform ${}^{G_{i}}\mathbf{T}_{C_{i}}$ and the base-to-world transform ${}^{W}\mathbf{T}_{B_{i}}$. Robot joint-zero corrections are also applied before forward kinematics to compensate for static joint bias. The world-frame pose of camera $i$ at time $t$ is therefore recovered as

where $\mathbf{q}_{i}(t)$ denotes the interpolated joint configuration and $\Delta\mathbf{q}_{i}$ contains the static joint-zero correction for arm $i$.

The robot-state logger operates at approximately 33 Hz, while the cameras record at 30 Hz. For each camera timestamp, the surrounding robot states are linearly interpolated to obtain the corresponding joint configuration. The robot log begins before camera acquisition and ends after recording is complete so that the usable camera interval remains inside the valid robot-pose interval and pose extrapolation is not required.

The final hand-eye calibration reports held-out geometric errors of approximately 3.58 mm and 3.93 mm for the two arms. These values characterize the internal calibration consistency and are later complemented by cross-arm and held-out RGB-D evaluation. The calibration is treated as epoch-specific: if a camera is remounted, a robot base is moved, the sensor bracket changes, or joint-zero behavior changes, the corresponding calibration should be repeated.

Each object survey begins when the operator places a single object somewhere within the shared reachable workspace and starts the acquisition procedure. The object position and orientation are not manually measured, no marked placement location is required, and no turntable is used. The system therefore estimates the object location during each survey before generating the viewing trajectories.

The complete autonomous acquisition sequence is summarized in Fig. 4. Following object placement, the system performs object localization, cross-arm confirmation, adaptive viewpoint planning, synchronized multimodal recording, and post-capture verification before the episode is accepted into the dataset.

Object localization begins with a structured pan-based search using the eye-in-hand camera. The searching manipulator directs the camera toward candidate regions of the tabletop until a valid depth-based object estimate is obtained. The estimated object center and spatial extent are treated as an initial hypothesis rather than being used immediately for trajectory generation. The object location is then independently checked from both manipulators. During this cross-arm confirmation stage, one arm observes the object while the other remains parked. Estimates that fall outside the valid workspace or exhibit unreliable depth support are rejected. When both arms produce consistent estimates, their object-center estimates are combined, while the larger estimated radius and hten are retained to form a conservative object envelope. Conflicting detections are not blindly averaged, the system instead falls back to a valid reference estimate or terminates the automated survey.

Using the confirmed object center and extent, an independent viewing trajectory is generated for each arm. The orbit adapts to object size, with larger objects receiving broader or denser angular coverage. Candidate camera poses are oriented toward the estimated object center and checked for inverse-kinematics feasibility, workspace limits, table clearance, excessive joint motion, and interference with the parked manipulator. Invalid candidates are removed, and the trajectory can be regenerated if too few feasible views remain. Only one manipulator moves at a time, while the other stays in a parked configuration. Both cameras record continuously during the survey, producing the representative RGB, metric depth, and stereo infrared observations shown in Fig. 5. Robot motion is executed using non-blocking commands so that joint-state logging continues during movement, and a short settling interval is applied after each accepted viewpoint.

This collection strategy is designed to produce complementary partial observations rather than identical views from both arms. Each camera therefore records a sequence of changing eye-in-hand viewpoints from its accessible side of the object, providing the geometric diversity later used for cross-arm fusion and evaluation.

Fig. 6 shows the physical tabletop objects included in DARP. The released dataset contains ten unique physical objects. Each object survey is preserved in acquisition form and contains the two native RealSense recordings, the synchronized robot-state log, and the object-centered acquisition metadata:

<object>/
    cam348522076490.bag
    cam405622074504.bag
    arm_poses.csv
    manip_envelope.json

The two .bag files are continuous multi-view recordings, not single images or isolated viewpoints. Each contains the RGB, depth, left infrared, and right infrared streams from one eye-in-hand camera together with the stream profiles, intrinsics, and depth scale required to decode the measurements. The arm_poses.csv file stores timestamped joint states for both manipulators, including the arm identifier and six recorded joint angles. The manip_envelope.json file stores the estimated object center, radius, height, nominal camera standoff, and robot configuration information used during collection.

For efficient downstream processing, selected observations can also be exported as compressed keyframes containing aligned RGB, raw depth, stereo infrared, a conservative geometric object-support mask, camera intrinsics, depth scale, and the corresponding $4\times 4$ world-from-camera pose. The raw acquisition files remain the primary dataset so that alternative frame-selection, calibration, segmentation, fusion, or learning pipelines can be applied without repeating the physical experiment.

The dataset is intentionally not tied to one fixed complete object mesh or one semantic task. Instead, it preserves the multimodal sensor observations, robot motion, and calibration information required to regenerate camera trajectories, partial point clouds, fused geometric representations, or learning inputs.

The geometric pipeline is used to demonstrate that observations from the two eye-in-hand sensors can be combined consistently in the shared world frame. For each selected RGB-D frame, raw depth is converted to metric depth using the device-specific scale and back-projected using the recorded camera intrinsics. The resulting camera-frame points are transformed into the world frame using Eq. (1).

A conservative object envelope derived during acquisition removes table and background measurements. View-diverse observations from both arms are then accumulated in the shared frame. The final production fusion uses the calibrated robot-camera poses directly, local depth discontinuities and weakly supported voxels are rejected, followed by statistical outlier removal.

A measured surface mesh is generated from the fused object cloud using multi-scale ball pivoting. The procedure does not predict unobserved geometry and therefore preserves missing or occluded regions rather than artificially closing them. A subset of RGB-D keyframes is excluded from fusion and retained for the held-out point-to-mesh evaluation reported in the following section.

The released dataset contains eight unique physical tabletop objects. Each object is observed from both sides of the workspace using the two eye-in-hand RealSense cameras described in Section III. For every object survey, the two cameras record continuous RGB, depth, and stereo infrared streams while the joint configurations of both manipulators are logged simultaneously. The resulting data therefore contain multiple changing viewpoints from each arm rather than a single image pair.

Table I summarizes the physical object envelopes and recording statistics of the released acquisitions. Across the retained recordings, the dataset contains 61,012 robot joint-state rows from the two manipulators and approximately 18.4 min of synchronized dual-arm acquisition. The native recordings contain more than 63,000 depth frames from the two sensors and occupy approximately 137.94 GB. These statistics reflect the continuous multi-view nature of DARP, in which each episode preserves a temporal sequence of changing eye-in-hand observations rather than a small set of isolated viewpoints.

The dataset can consequently be examined at several levels, individual RGB-D observations, single-arm multi-view sequences, complementary partial point clouds, or dual-arm fused representations. The raw recordings remain the primary released data, while the reconstructed point clouds and meshes presented below are derived products used to demonstrate geometric consistency.

A central requirement of the acquisition framework is that independently recorded observations from arm110 and arm111 can be transformed into the same metric coordinate system without requiring visual re-registration of every frame. For each selected observation, the camera pose is obtained from the calibrated base and hand-eye transforms together with the synchronized robot configuration, as described in Eq. (1).

The two arms naturally produce different partial surface measurements because each sensor approaches the object from a different side. In regions observed by both cameras, however, corresponding measurements should occupy similar locations after transformation into the world frame. We therefore use overlap between the independently reconstructed arm-specific point clouds as an additional check of calibration consistency.

Fig. 7 further illustrates the geometric relationship between the two arm-specific observations. In the shared world frame, the partial point clouds occupy complementary regions of the object surface, while the co-observed portions remain reasonably aligned after calibration and pose recovery.

The experiments show that the two partial observations align sufficiently well to form coherent combined object surfaces while still preserving complementary geometry. Importantly, the final production fusion does not apply unrestricted ICP or centroid alignment to force the two arms into agreement. The reported fused surfaces are generated primarily from the calibrated camera poses and repeated multi-frame support. As a result, visible agreement between the two arm-specific measurements reflects the quality of the robot-camera calibration and temporal pose association rather than an unconstrained post-hoc registration step. Fig. 8 illustrates the complementary nature of the two robotic viewpoints. The arm-specific point clouds contain different visible regions of the object, while their calibrated fusion combines these measurements into a more complete observed-surface representation in the shared world frame.

To evaluate geometric repeatability without comparing the reconstruction against the same observations used to generate it, a subset of RGB-D keyframes is excluded from fusion. These held-out observations are independently back-projected into 3D and transformed into the world frame using their corresponding robot poses. The resulting query points are then compared with the measured surface mesh using point-to-mesh distance.

Across 224 held-out RGB-D keyframes, the evaluation contains 1,563,466 three-dimensional query points. The aggregate results are summarized in Table II. The median point-to-mesh distance is 2.13 mm, while the RMSE is 4.04 mm. The 90th-percentile distance is 6.38 mm. In addition, 83.82% of the held-out points lie within 5 mm of the reconstructed surface and 96.56% lie within 10 mm.

The values in Table II should be interpreted as held-out observation agreement. They do not represent absolute error against a complete object model obtained from an independent high-accuracy scanner. This distinction is important because the reconstructed mesh contains only surfaces supported by the recorded observations.

The qualitative results show the complementary role of the two robotic viewpoints. A point cloud generated from only one arm typically contains the object surfaces visible from that side of the workspace, while self-occluded or oppositely facing regions remain missing. The second arm contributes measurements from a substantially different viewing direction, increasing visible-surface coverage when the observations are combined.

The final fused representation retains repeatedly supported RGB-D measurements and removes isolated observations before surface generation. Multi-scale ball-pivoting produces a mesh over these measured regions without deliberately closing surfaces that were not observed. Together, the held-out evaluation and qualitative reconstructions show that the released calibration, robot-state information, and multimodal measurements are sufficiently consistent to support metric dual-arm fusion.

The dataset is designed as a reusable robotic perception resource rather than for a single reconstruction task. Because RGB, depth, stereo infrared, robot joint states, and calibration information are preserved together, the same acquisition can be represented at the image, point-cloud, trajectory, or fused-surface level.

A direct application is multi-view 3D reconstruction. Individual observations from either manipulator provide only partial object geometry, while calibrated observations from both arms can be transformed into the shared world frame to obtain a more complete measured surface. The arm-specific observations also make it possible to study single-arm versus dual-arm perception and quantify the benefit of complementary viewpoints.

The dataset can further support multimodal robotic perception. RGB, metric depth, and stereo infrared measurements may be used independently or jointly for object segmentation, recognition, feature fusion, and sensor-ablation studies. Since the corresponding robot configuration and camera pose are retained, perception models can additionally incorporate viewpoint or pose information rather than treating each image as an isolated observation.

Another relevant direction is active perception and next-best-view planning. The recorded camera trajectories and partial observations provide examples of how object visibility changes as an eye-in-hand camera moves around the workspace. Future methods could use this information to predict viewpoints that reduce occlusion or provide additional geometric coverage.

The complementary partial observations may also support future learning-based shape completion. Existing methods such as PCN, PoinTr, and related point-cloud completion approaches predict geometry that is absent from the input observation [25, 24]. In contrast, the geometric pipeline in this paper reconstructs only surfaces supported by measured RGB-D observations. Shape completion should therefore be treated as a downstream learning task requiring an appropriate complete-shape target or self-supervised training objective rather than as an output already provided by the dataset.

From a robotics perspective, the shared metric representation can also support collaborative manipulation studies. For example, one manipulator may observe surfaces that are occluded from the other, while both observations remain related through a common world frame. Such data can be useful for future systems in which an embodied robot reasons jointly about viewpoint, object geometry, and manipulation actions.

Table III summarizes the position of the proposed dataset relative to representative object-centric RGB-D and robotic perception resources discussed in Section II. The comparison is intended to highlight differences in acquisition design rather than to suggest that the datasets address identical research problems.

The Washington RGB-D Object Dataset and BigBIRD provide multi-view RGB-D observations of physical objects for object recognition and 3D perception [12, 18]. The YCB Object and Model Set combines physical manipulation objects with associated visual and geometric models and has become widely used in robotics research [3]. GraspNet-1Billion connects RGB-D observations with large-scale grasp annotations [10], while ROBI focuses on multi-view observations of reflective objects in robotic bin-picking environments [23].

The proposed dataset differs primarily in the way observations are acquired and geometrically linked. Two independently moving eye-in-hand cameras observe the same object from opposite sides of a shared manipulation workspace, while synchronized robot joint states and calibration parameters are retained with the original sensor recordings. This allows the pose of each selected observation to be reconstructed through robot kinematics and enables direct analysis of complementary arm-specific and dual-arm measurements.

The purpose of this comparison is not to claim that existing datasets lack multi-view or robotic information. Instead, the distinction of the proposed dataset is the combination of two independently moving eye-in-hand sensors, continuous multimodal acquisition, synchronized manipulator states, and a calibration chain that expresses both camera streams in the same metric world frame. This combination supports direct study of complementary robotic viewpoints while preserving the raw acquisition data for alternative processing and learning methods.