To ask for help or not to ask: A predictive approach to human-in-the-loop motion planning for robot manipulation tasks

The work focuses on the problem of non-prehensile physics-based robotic manipulation in clutter. We consider the Reaching Through Clutter (RTC) problem, where tens of robots are working in a warehouse to fulfil orders. The robots need to reach and grasp a desired object from the back of a cluttered shelf. On the shelf, multiple objects are blocking the way to the goal object, and this poses motion planning and control challenges for autonomous robots.

While some robots are able to autonomously reach and grasp a goal object from some shelves, for other, harder, instances of the problem, a human is available to provide high-level guidance to the robots. The human provides high-level guidance, the robot leverages the high-level input and integrates it in its motion planning algorithm and continues fully autonomously. Such a system is at least as good as the state-of-the-art autonomous system, with the benefit that when a robot fails, it can fall back to a human for high-level guidance.

Motivation

For a similar problem to Reaching Through Clutter, Amazon challenged the community with the Amazon Picking Challenge (APC) [1]. Although the competition focused exclusively on autonomous systems, it demonstrated that the problem is challenging even with relaxed assumptions.

The problem of Reaching Through Clutter (RTC) in a physics-based non-prehensile setting is challenging due to the following reasons:

1. The state space is of high dimensionality. The state space consists of the configuration of the robot and of all the objects in the environment.
2. Physics simulation is required to, given the current state of the environment and a robot action, simulate/predict the system dynamics. Such simulation in such a high-dimensional space is computationally expensive to run, however.
3. This is an under-actuated system, where the robot can manipulate the objects through contact, but it cannot grasp and, therefore, control them directly.
4. Finally, the problem of Reaching Through Clutter is NP-Hard [2].

Nevertheless, Reaching Through Clutter has the potential for near-term impact to warehouse and service robotics, and solutions to this problem can accelerate the development of such systems in the real-world.

To ask or not to ask for help?

In the above description, I motivated a system that leverages human input, however, in the setting with tens of robots and a single human operator, an interesting question is, when should the robots ask for human help? Can different solutions to this problem lead to better performance in a multi-robot guidance setting?

The naive approach will be to ask for human help always, before planning [3]. Although this can help all robots at all times, it can overburden a human with multiple requests. A better approach, is to ask for intervention only when the motion planning module fails [4]. The benefit is that, for trivial problems, the robots will plan autonomously and only fall back to a human for non-trivial ones.

A shortcoming of the solution in [4] is that, sometimes, it can take a while for a robot to fail. In this work, we propose a third solution that leverages machine learning to learn when to ask for human help and employ human input earlier, before productive time is wasted.

If you are interested to learn more about the work, please read the pre-print.

Citing this paper

If you want to cite this work, please use the following:

@inproceedings{papallas2022iros,
  title={To ask for help or not to ask: A predictive approach to human-in-the-loop motion planning for robot manipulation tasks},
  author={Papallas, Rafael and Dogar, Mehmet R},
  booktitle={IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  year={2022},
  organization={IEEE}
}

Blog Post’s Bibliography

• [1] Eppner, C., Höfer, S., Jonschkowski, R., Martín-Martín, R., Sieverling, A., Wall, V. and Brock, O., 2016, June. Lessons from the amazon picking challenge: Four aspects of building robotic systems. In Robotics: science and systems.
• [2] Gordon Wilfong. Motion planning in the presence of movable obstacles. Annals of Mathematics and Artificial Intelligence, 3(1):131–150, 1991.
• [3] Rafael Papallas and Mehmet R Dogar. Non-prehensile manipulation in clutter with human-in-the-loop. In IEEE International Conference on Robotics and Automation, 2020.
• [4] Rafael Papallas, Anthony G. Cohn, and Mehmet R Dogar. Online replanning with human-in-theloop for non-prehensile manipulation in clutter — a trajectory optimization based approach. IEEE Robotics and Automation Letters, 2020.