Open-vocabulary object detection

Open-vocabulary RGB-D perception for metric shape estimation

A real-time pipeline that finds objects by natural-language description and measures them with classical geometry — a vision foundation model decides what is in the scene, depth-based optimization decides exactly where it is and how big it is.

Input: RealSense RGB-D stream plus free-text object prompts (“a box”, “a bottle”)
Output: Tracked oriented boxes and cylinders with metric pose and dimensions, stable across frames
Runs at: 5 Hz live on robot hardware, ROS 2

Text-prompted segmentation → Depth back-projection → Primitive fitting → Tracking

Videos

Demo

Open-vocabulary RGB-D detection and metric shape tracking

What it is

The pipeline detects, fits, and tracks parametric shape primitives — oriented boxes and cylinders — from a single RGB-D camera. Meta’s SAM3 segments object instances directly from text prompts, so new object categories cost a phrase, not a retraining run.

Each instance mask is back-projected through the depth image into a 3D point cloud, and the primitive’s pose and dimensions are then recovered by geometric optimization against both the point cloud and the original image mask. A tracker links detections across frames and smooths the estimates, optionally in a fixed world frame while the robot moves.

Why the hybrid design

The split of responsibilities is the core idea. Large vision models are excellent at open-vocabulary recognition but are not reliable rulers; depth data with classical optimization measures accurately but cannot name what it is measuring. Combining them keeps each side doing what it is best at — and keeps the metric half fully debuggable, since every fitted box can be replayed and inspected offline from recorded masks without a GPU.

Technical highlights

Single-view box fitting that respects occlusion. A camera only ever sees one or two faces of a box; the fitter derives orientation from detected face planes rather than point statistics, which would bias the estimate diagonally across the visible faces.
Fitting scored in 3D and 2D simultaneously. Candidate shapes are refined against the point cloud and the segmentation mask’s silhouette, so depth noise and mask errors check each other.
Temporal consistency as a soft prior. The previous frame’s tracked shape participates in fitting as a gently weighted candidate — stabilizing dimensions the current view barely constrains without overriding real changes.
Replayable by design. Segmentation results are recorded in a self-describing format, so the entire geometry pipeline can be re-run and improved offline from logged data.

Stack

ROS 2 · C++ & Python · SAM3 · PCL / RANSAC · Nonlinear optimization