Research Overview

I like to introduce my research by discussing its practical application. Let’s take the illustration above as an example: a robot assisting a motor-impaired person with everyday chores, such as retrieving an orange juice bottle from a packed and cluttered fridge. Obstructions at the fridge’s front complicate the robot’s direct access to the object. Hence, the robot must plan to rearrange these obstacles, possibly by safely nudging them aside, to retrieve the desired object.

Challenges

Though seemingly straightforward for humans, who tackle such tasks daily and seemingly effortlessly, this task introduces a number of challenging problems for robots. I will highlight some challenges below.

Perception and Computer Vision

At the forefront is perception. A robot requires the capability to sense the environment, build a model of the world to work with, and identify objects of interest, obstacles and their position in space. Even with substantial advancements in deep learning and computer vision, determining the position and orientation of objects in 3D space, especially in cluttered and partially obscured settings, remains a challenge. In real-life scenarios, like our fridge example, nearby and overlapping items can confuse the most advanced systems, sometimes resulting in significant errors and false positives.

Planning

A robot needs a capability to plan its motion. The term motion planning refers to the problem of finding a solution to drive a robot from an initial state to a goal state. In the context of a robotic arm, with a more specific term, manipulation planning, the problem is to find the sequence of controls that will move the arm from the current configuration, to achieve a certain manipulation goal (say, the robot reaching into the fridge to grasp an object). Working with a robotic arm, it usually involves a much higher degree of freedom (DOF) system, which makes it much harder and computationally expensive to plan.

Motion planning has indeed progressed, especially for collision-avoidance motion. But in certain scenarios, collision isn’t always detrimental; for instance, while a robot must avoid damaging its surroundings, it may need to interact with objects to move them, for example by pushing them. The primary challenge is the vast dimensionality of these planning problems, demanding intelligent solutions and algorithms.

Physics-based Manipulation

While I touched on collision-free planning earlier, also known as “geometric motion planning”, there’s another dimension: physics-based planning. Here, motion planning algorithms must account for the real-world physics and dynamics of actions. Integrating these physics-aware algorithms entails using a physics simulator, which introduces further complexities. One major challenge is the potential discrepancy between simulated physics and real-world dynamics, posing the risk of real-world failures despite simulated successes. Physics simulation is also expensive to run¹ which poses challenges for fast and real-time motion planning.

Open world manipulation

Ideal environments, where robots operate with fixed objects, known geometries, and repetitive actions, seldom exist outside manufacturing settings. Real-world environments are dynamic, with varied objects and unpredictable human interactions. Thus, creating a system adaptive to these “open worlds” presents its own set of challenges. How does each fridge look like? How are objects expected to be placed in every fridge?

The general focus and interest of my research is in the above challenges: perception, motion planning, physics-based manipulation planning and motion control in an open world. I am interested in building algorithms that enable robots to solve real world problems to amplify what humans can do.