Robotics and Biology Laboratory

Leveraging Problem Structure in Interactive Perception for Robot Manipulation of Constrained Mechanisms


PhD board

Prof. Dr. Oliver Brock (TUB)
Prof. Dieter Fox (University of Washington)
Prof. Tamim Asfour (Karlsruhe Institute of Technology)

Roberto Martín-Martín

Title: Leveraging Problem Structure in Interactive Perception for Robot Manipulation of Constrained Mechanisms


In this thesis we study robot perception to support a specific type of manipulation task in unstructured environments, the mechanical manipulation of kinematic degrees of freedom. In these tasks the goal of the robot is to create controlled motion, i.e. to change configuration of the kinematic degrees of freedom (DoF) of the objects in the environment. Often, the environment contains articulated objects. Their manipulation is specially complex because the knowledge about their properties that would facilitate the task (e.g. their motion constraints, the geometry of their parts, their dynamic and frictional properties) are first revealed when the robot interacts with the object. Therefore, the perception of these objects should exploit interactions to create information-rich sensor signals. This type of problem and the perceptual methods that incorporate actions are called interactive perception. In this thesis we propose a general approach for interactive perception and instantiations of this approach into perceptual systems to build kinematic, geometric and dynamic models of articulated objects.

Perceptual problems in the domain of robot mechanical manipulation of DoF possess special challenges. While unstructured environments are usually continuously changing, robot mechanical manipulation exacerbates this characteristic. But in fact, these changes in the environment contain crucial information for a robot that aims to change purposely the state of the world. Perception for robot manipulation has to extract information from changing sensor signals and their relationship to changes in the environment and to actions. The perceptual process has to deliver information quickly and in an online manner, based only on past and current sensor signals, so that the information can be applied to ongoing interactions. And the perceptual solutions must be versatile enough to cope with a broad range of environmental and task conditions in which the robot should be able to manipulate DoF.

To address these challenges, we propose an approach for interactive perception that leverages four structural regularities of perceptual problems in the domain of robot mechanical manipulation of DoF. First, our approach leverages the dependency between robot actions and changes in the sensor stream using ideas from interactive perception. Second, our approach exploits the temporal structure of the physical processes involved in the mechanical manipulation of DoF using temporal recursion. Third, our approach makes use of task-specific priors that encode physical regularities of the world. These physical priors relate to the manipulation of DoF in unstructured environments and the sensor signal formation: physics laws that govern the motion of objects (e.g. kinematics), mathematical models for the signal formation (e.g. projective geometry), and assumptions about the physical properties of the environment (e.g. that the environment is composed of rigid solid parts). And fourth, our approach leverages dependencies between multiple perceptual subtasks that extract different information patterns about the same articulated object.

The approach we propose leverages the aforementioned problem structure with an interconnected network of recursive estimation processes encoding physical priors and exploiting robot interactions. We instantiated this approach in several robot perceptual systems, presented in consecutive chapters, to extract information about articulated objects –kinematic, geometric and dynamic properties– using only RGB-D information, or a combination of RGB-D and proprioceptive signals (e.g. applied wrenches, configuration of robot’s joints). We study our proposed approach through these interactive perception systems. We evaluate if the systems can extract task-relevant information for the mechanical manipulation of DoF of articulated mechanisms for different objects and in varying and challenging environmental and task conditions. To truly demonstrate that the perceived information is useful for robot manipulation, we complement the perceptual systems with methods to monitor, control and steer the robot interaction based on the online perceived information. We also propose and evaluate a novel method to generate and select informative actions for interactive perception based on the information acquired so far.

February 2018