Rafael Papallas

PhD Student Software Engineer

Cashier Robot

Human-Robot Interaction For Cashier Robot


Project Information

© 2016/2017 The University of Leeds and Rafael Papallas

Date 2016/2017
Categories Robotics
Programming Language Python
Tags , ,
GitHub Repository https://github.com/papallas/baxter_cashier
Website https://papallas.github.io/baxter_cashier/

Project Description

"Human-Robot Interaction for Cashier Robot" is my final-year project submitted at the University of Leeds. My individual project has been awarded the "Buckley Prize" as the best 60-credit project in the School of Computing for the academic year 2016-2017.

The robot used was Baxter Robot by Rethink Robotics and the programming language used was Python. This project combines several robotics areas mainly manipulation and computer vision.

As can be seen from the diagram below, the project has three main functionalities among others:

  • Skeleton Tracking: The system takes care to detect people and the positioning of their body parts like head, hands, torso etc using a depth sensor. We are interested in tracking the left and right hand of the user only so we can get the position and orientation of their hand so the robot can pick banknotes from the user's hand. Using a custom filter called "Two-Way Pose Elimination" the system is able to distinguish false positive hand-over signals (that is the user did not raise his/her hand to hand over a banknote) over actual hand-over signals (that is the user actually is handing over banknotes to the robot).
  • Money Recognition: Different approaches for money recognition has been tested including colour, template matching and AR-code recognition. Because of time constraints, the AR-code recognition has been selected mainly because (1) was very accurate with no false positives and (2) was well tested from other people (third-party library).
  • Change Handling: The robot is able to work out, given an amount due, the change required to be returned to the user and to actually iterative return the change due (1 or more banknotes) to the user's hand.

Furthermore, the system and the robot is aware of obstacles (table, depth sensor and walls) and the manipulation planning consider those obstacles. When the robot does the planning will try to find a collision-free solution for the manipulation problem.

The development of the project was split into three phases each of which brought a major functionality or improvement to the system. The final solution was tested using ten participants as well as testing each functionality isolated.


Click magnifier to enlarge