NeRF-Control: Perception-Aware Trajectory Optimization
AuthorsA. Gaggar, T. Murphey
Published at2026 Intl. Conference on Robotics and Automation [Submitted]
TLDR:
Perception models neglect the motion costs of physical sensing agents to acquire training data, such as execution time, total distance traveled, or energy expenditure. In this work, we present a perception-aware objective function for trajectory optimization that jointly balances reducing model uncertainty, ensuring scene coverage, and minimizing total distance traveled. This objective is used to develop an iterative trajectory optimization framework for closed-loop, active data acquisition. To our knowledge, this is the first work to formulate NeRF-aware trajectory optimization for active data collection under perception objectives and motion constraints.
Furthermore, we demonstrate autonomously trained NeRFs on a 7-DoF robotic arm by executing optimized view trajectories for two tabletop objects. Training NeRFs on hardware brings its own challenges, with reconstructed models differing in position, orientation, and scale from ground truth validation images. Lighting inconsistencies, motion blur, and defocus must also be addressed. On average, our method achieves 33% better performance compared to SOTA methods. </p>
Background:
Robots that collect data for perception, mapping, or modeling incur motion costs: every new viewpoint consumes time, energy, and potentially introduces safety risks. Yet most research in neural radiance fields (NeRFs) has focused on reconstruction quality and training efficiency, assuming new training data can be acquired instantaneously and for free. The gap between perception-driven, active view selection and dynamics aware trajectory planning is critical to overcome
to deploy NeRFs on physical agents.
Website with code, videos, and interactive figures will be published after ICRA review.
Prior work has focused on which views to collect but not how to collect them efficiently. Our work formulates a perception-aware objective for continuous trajectory optimization, optimizing both NeRF-specific objectives and the physical cost of data acquisition. This enables motion-efficient, online data collection for NeRFs on real robots.
Our contributions are:
1) A differentiable objective function for trajectory optimization that balances reducing NeRF model uncertainty, ensuring scene coverage, and minimizing motion cost of an agent.
2) Developing the objective within an iterative trajectory optimization framework with a continuous dynamics model to create discrete viewpoints for NeRF training.
3) End-to-end, online evaluation in simulation and on a 7-DoF robot arm. In simulation, our method achieves 33% higher reconstruction accuracy with 39% reduced trajectory length compared against uniform and TSP-based baselines; in hardware, we achieve better reconstruction, while taking 54% less time on average for the robot to execute our trajectory.