Autonomous mobile manipulation system using the Boston Dynamics SPOT robot to detect, localize, grasp, and arrange chairs around a table using fiducial-marker-based perception.
This project integrates:
- Boston Dynamics SPOT SDK
- Intel RealSense RGB-D cameras
- ArUco/AprilTag pose estimation
- Multi-camera localization
- Coordinate-frame transformations
- Autonomous waypoint navigation
- Vision-guided robotic manipulation
The system demonstrates a complete perception-to-action pipeline for structured indoor mobile manipulation tasks.
The robot autonomously:
- Detects fiducial markers attached to chairs and walls
- Localizes itself relative to a global origin marker
- Computes chair poses in the global frame
- Identifies unarranged chairs
- Navigates to the target chair
- Grasps the chair using SPOT’s manipulator arm
- Moves the chair to a predefined goal configuration
- Repeats until all chairs are arranged
The project explores the question:
Are fiducial markers sufficient for performing autonomous mobile manipulation tasks in structured indoor environments?
- Dual Intel RealSense D435i cameras
- ArUco/AprilTag marker detection
- Pose estimation using OpenCV
- Multi-camera spatial registration
- Global origin marker
- Wall reference markers
- SPOT body-frame localization
- Homogeneous transformation matrices
- SPOT SDK locomotion commands
- Waypoint-based navigation
- Manipulator arm control
- Gripper actuation
- Real-time pose updates
- Chair state validation
- Autonomous task sequencing
- Closed-loop manipulation pipeline
- Boston Dynamics SPOT
- 6-DOF manipulator arm
- Parallel jaw gripper
- 2 × Intel RealSense D435i cameras
- SPOT onboard perception cameras
- ArUco / AprilTag markers
- 20 cm × 20 cm markers
- Mounted on:
- chairs
- walls
- reference surfaces
.
├── arrange_chairs_around_table.py
├── pose_estimation.py
├── spot_robot_commands.py
├── read_video_stream.py
├── utils.py
├── saved_objects_poses.json
├── aruco_markers.pdf
├── Camera_Calibration/
│ ├── Camera_1/
│ └── Camera_2/
├── chair_aruco.3mf
├── wall_aruco.3mf
└── README.mdMaster execution pipeline.
Responsibilities:
- Initialize SPOT
- Configure cameras
- Update object poses
- Detect unarranged chairs
- Execute navigation and manipulation
Perception and localization module.
Features:
- ArUco detection
- 6-DoF pose estimation
- Camera-to-world transformations
- SPOT localization
Key Functions:
estimate_poses_of_aruco_tags()localize_spot_wrt_origin()update_poses_of_chairs()
SPOT SDK interface layer.
Features:
- Robot initialization
- Locomotion
- Arm manipulation
- Gripper control
- Fiducial detection
Key Functions:
move_robot_to_location()move_SPOT_behind_chair()grasp_chair_using_SPOT()move_arm_to_grasp_pose()
RealSense camera interface.
Features:
- Camera streaming
- Calibration loading
- Frame acquisition
- Image visualization
Transformation and helper utilities.
Features:
- Homogeneous transformations
- Rotation conversions
- Pose utilities
- Object management
- Waypoint logic
The system uses multiple coordinate frames:
| Frame | Description |
|---|---|
| Origin Frame | Global world reference |
| Camera Frame | RealSense camera coordinates |
| SPOT Body Frame | Robot body reference |
| Gripper Frame | Manipulator end-effector |
| Chair Marker Frame | Fiducial marker on chair |
- Configure RealSense pipelines
- Load calibration parameters
- Detect wall fiducials
- Compute global coordinate system
- Detect chair markers
- Compute chair poses in world frame
- Detect visible wall markers
- Compute SPOT pose relative to origin
- Identify nearest unarranged chair
- Move SPOT through predefined waypoints
- Move behind chair
- Detect marker using gripper camera
- Execute grasp
- Transport chair
- Release chair
- Boston Dynamics SPOT
- Intel RealSense D435i cameras
- Python 3.10+
- OpenCV
- NumPy
- SciPy
- pyrealsense2
- Boston Dynamics SDK
pip install numpy scipy opencv-python pyrealsense2Install Boston Dynamics SDK:
pip install bosdyn-client bosdyn-mission bosdyn-apiRun calibration before execution:
python calibration.pyCalibration files will be stored in:
Camera_Calibration/Place chairs in desired goal configuration.
Capture goal-state images.
Move chairs to arbitrary positions.
python arrange_chairs_around_table.py| Metric | Value |
|---|---|
| Total Trials | 10 |
| Chairs Arranged | 21/28 |
| Success Rate | 75% |
| Average Position Error | 1.6 cm |
| Average Orientation Error | 5° |
Requires markers attached to objects.
Markers may become invisible during navigation.
Accurate camera calibration is critical.
Limited robustness in dynamic environments.
Potential collisions during placement.
- Markerless object detection
- Learning-based grasp planning
- SLAM integration
- Dynamic obstacle avoidance
- ROS2 migration
- Multi-robot coordination
- Force-feedback manipulation
- Real-time collision checking
This project demonstrates:
- Vision-guided mobile manipulation
- Hybrid external + onboard perception
- Fiducial-based autonomous manipulation
- Real-world deployment on SPOT
- Closed-loop chair arrangement
If you use this work, please cite:
@mastersthesis{siddharth2026spot,
title={Are Fiducial Markers All We Need to Perform Mobile Manipulation Tasks?},
author={Senthilnathan, Siddharth},
school={University of Minnesota},
year={2026}
}- Boston Dynamics
- University of Minnesota
- RPM Lab
- Intel RealSense SDK
- OpenCV ArUco Library
This project is intended for academic and research purposes.