| Mar 12, 2026 |
Researchers combine 3D image recognition with language models so AI can integrate into any robot, using continuous probability calculations to guide searches.
|
|
(Nanowerk News) The new robot from Prof. Angela Schoellig’s TUM Learning Systems and Robotics Lab looks like a broomstick on wheels with a camera mounted at the top. It is one of the first robots that not only integrates image understanding but also applies it to a clearly defined task (IEEE Robotics and Automation Letters, “Where Did I Leave My Glasses? Open-Vocabulary Semantic Exploration in Real-World Semi-Static Environments”).
|
|
To find a pair of glasses misplaced in the kitchen, for example, the robot has to look around and build a three-dimensional image of the room. The camera initially provides two-dimensional images, but these pixels also contain depth information. This creates a spatial map of the environment that is accurate to the centimeter and is constantly updated. A laptop also provides the robot with information about which objects are visible in the image and what significance they have for humans.
|
|
“We have taught the robot to understand its surroundings,” says Prof. Angela Schoellig. The head of the Robotics Lab at the TUM Chair of Safety, Performance and Reliability for Learning Systems aims to develop robots that can navigate any environment independently. Humanoid robots working in factories or robots in care settings in private homes require this newly developed basic understanding, which, as Schoellig explains, “is important for all robots that move in spaces that are constantly changing.”
|
 |
| The search robot has a 3D camera on board to find lost items such as glasses. (Image: Andreas Schmitz, TUM)
|
Internet knowledge translated into the robot’s language
|
|
The robot therefore understands that a table or window sill can be used to briefly set down a pair of glasses, whereas a stovetop or a sink are not suitable for this purpose.
|
|
“The language model captures the relationships between the objects and we convert this information into the robot’s language,” explains Prof. Schoellig.
|
|
Two-digit numbers appear on the three-dimensional map of the environment, constantly recalculating the likelihood that the object being searched for is located there. According to the research results, the robot then searches the probable locations almost 30 per cent more efficiently than if it searched randomly throughout the room. Artificial intelligence is used in two ways: on the one hand in image recognition and on the other hand through the use of a language model.
|
|
Another special capability of the robot: it remembers previous images and is able to compare them with new images of its surroundings. If a new object suddenly appears in the kitchen, it recognizes the change with a high degree of certainty (95 per cent) and marks these areas as “highly probable” search locations.
|
Next step: searching behind cupboard doors
|
|
In the next step, the TUM scientist and board member at the Munich Institute of Robotics and Machine Intelligence (TUM MIRMI) also wants to search for objects that are in a drawer or behind a door. To do this, however, the robot will not merely have to draw on knowledge from the internet but will also have to interact with its surroundings. Robotic arms and hands must open a cupboard and determine whether it opens upwards or sideways and how best to grasp the handle. This will enable the robot to search even in closed spaces such as cupboards or drawers.
|
