Robots are becoming more and more sophisticated, but in one respect they are still inferior to us: “Humans have outstanding visual abilities,” says Prof. Dr Marion Gebhard. “For example, we can easily recognise whether a glass is full of water and transport it with the necessary care. Robots can't do that yet.”

Therefore, people are still needed at machine workplaces to keep an eye on the processes and intervene if necessary. But the best visual skills are of no use if a person has no control over their limbs. People with paraplegia, for example, have very few opportunities to participate in the primary labour market.  

This is where the MIA project comes in: Marion Gebhard's team from the Department of Sensor Technology and Actuators at the Westphalian University of Applied Sciences has devised a control system for a robotic arm that can be operated entirely by head and eye movement. The device sits on the head like a pair of glasses and works with tiny motion sensors, like those installed in mobile phones today. “Nodding, turning, rolling - these are the degrees of freedom with which these sensors work,” Gebhard explains.  

The motion sensors are supplemented by an eye tracker that tracks pupil movement and provides further degrees of freedom: pupils up or down, pupils to the left or right, eyes closed and the so-called head gaze, i.e. the direction in which the head is oriented.

To complement the work of Marion Gebhard’s team, the second MIA team from the Human Computer Interaction Department, led by Prof. Dr. Jens Gerken, has been working out a way to simplify the control of the robot arm by visual cues and feedback. For example, it is not easy for humans to see when the robot is positioned exactly in the right position to grasp an object at the first go. To help users deal with this problem, augmented reality glasses display additional information about positioning in space directly in their field of vision, which makes control easier and also more efficient.

Many potential applications for hands-free robot control

A “hands-free” control of machines can, of course, be useful for healthy people, too: for example, if they need to stop a machine in an emergency even when they do not have their hands free at the moment. In the MIA project, however, the researchers were explicitly interested in enabling people with severe physical disabilities to participate more in the labour market.

To this end, the two teams from the Westphalian University of Applied Sciences joined forces with Büngern-Technik, a workshop for people with disabilities in Bocholt, Westphalia. Among other things, the workshop manufactures so-called adjustable wedges: movable metal fittings that are used, for instance, in height-adjustable slatted frames. The adjustable wedges ensure that the head section of the slatted frame remains in the set position.

“In theory, these adjustable wedges could be mounted by a fully automated robot,” Gebhard explains. “But this requires a lot of human preparatory work and also takes way too long.” A combined human-machine workplace may therefore be the better choice in this case. However, this would require people without mobility impairments to sort the parts and prepare them for the robot to pick up.

Man and machine work together precisely

This is where the MIA control unit comes into play: If the workplace is equipped with such a device, even people paralysed from the neck down can work on it. The robotic arm can be precisely controlled with head movements and glances, so that it can also perform non-automated work steps. The human guides it to the right place with eye and head movements and gives commands such as “grab” and “let go” by nodding, winking or voice command. “You learn to do that like you learn to change gears when driving a car,” Gebhard says.

The usability of the system was supposed to be tested directly by people with limited mobility at Büngern-Technik before the end of the project. However, like almost all research projects involving groups of test persons, MIA suffered severely from the Corona pandemic: The team could present the new technology in several workshops at Büngern, but the planned extensive trial runs had to be cancelled. “The technology works, but of course we lack the usability studies with our target group,” says the project manager. However, in order to test the basic control and visualization concept, the teams were able to conduct comprehensive studies with people without physical limitations and thus gradually optimize the system.

At the end of the project in spring 2022, however, the project partners suddenly faced a challenge that surprised them even more than the pandemic: Finding partners to further develop the device into a marketable product proved to be extremely difficult. Gebhard went to manufacturers of sports wheelchairs as well as other medical aids. “Wherever we go, everyone thinks our technology is great – but no one wants to bring it to market,” she says. “At this point in the development, we really need corporate partners who can turn the device into a prototype and then into a product. And we just can't find them at the moment.”

Functional safety and high requirements impede commercialization

There are many reasons for this, such as the high safety requirements that apply when people with physical disabilities are involved – they cannot intervene with their hands in an emergency if the control system fails. In addition, the robotic arms are still very expensive and the certification is complex and time-consuming. Workshops usually cannot afford it, and for many industrial companies the profit margins are not large enough to make the purchase worthwhile.

Marion Gebhard, however, does not want to give up yet. She firmly believes in the technology and would like to develop it into a service: Users could rent the entire workstation, including the robotic arm and controller, receive training from the provider, and then use the technology whenever and wherever it is needed – for example, to attract employees with physical disabilities to their company, but also in situations where the employees have both hands occupied otherwise.

From a scientific point of view, MIA was a complete success: In addition to a demonstrator, it yielded two doctoral theses in cooperation with the universities of Bremen and Duisburg-Essen. “One of them even received summa cum laude with four times the best grade 0.7 – that is a perfect score,” says the project manager proudly. Moreover, the team learned a lot in the process: “Dealing with the people at Büngern-Technik was a real enrichment for all of us.”