Fraunhofer Institutes, Lamarr Institute, igus and the University of Bonn develop language model for cost-effective automation solutions in order picking

A hackathon took place at the igus headquarters in Cologne in mid-September. In line with the motto “Artificial intelligence meets robotics”, 17 participants worked in three interdisciplinary teams to develop advanced automation solutions for logistics. The aim of the hackathon was to create an automated method for packing up pizzas using a large language model (LLM for short) based on AI. The event was organised in cooperation with the Fraunhofer Institute for Intelligent Analysis and Information Systems IAIS, the University of Bonn, the Lamarr Institute and the Fraunhofer Institute for Material Flow and Logistics IML.

20 tuna pizzas, 10 salami pizzas and 25 margherita pizzas: this is the order the supermarket round the corner placed with a large manufacturer of frozen pizzas. What sounds like a simple task is still currently still done by hand. A simple and unchallenging job for which it is often impossible to find staff. 17 participants from the Fraunhofer Institutes IAIS and IML, the Lamarr Institute for Machine Learning and Artificial Intelligence as well as the University of Bonn developed a solution for this problem during a five-day hackathon at the igus headquarters in Cologne. They used the cost-efficient “low-cost robotics” made in Cologne. The task of the hackathon was to develop an automation solution that automatically packs the pizzas with the help of artificial intelligence (AI). The challenge was to control a robot in such a way that it would recognise the right products, pick them up and pack them into mixed boxes using voice instructions directly from employees. By combining generative AI (GenAI) and robotics, the aim was to develop a solution that would make this repetitive work more efficient and cost-effective.

Automating industry with AI and cost-effective robotics

During the hackathon, three teams worked on a real-life use case in which a ReBeL robot arm was placed in front of a conveyor belt with various pizza products. With the help of a webcam and an AI-based segmentation system (Segment Anything Model, SAM), the systems recognised the various products on the conveyor belt and identified their position. The language model assigned these objects to the desired products based on the natural language instructions. The robot then placed the products in the boxes by following the instructions of the language model.

Alexander Zorn from Fraunhofer IAIS was delighted with the results of the hackathon: “We are very pleased to be able to use the igus robotics expertise to develop real proof of concepts for customers in industry. The combination of artificial intelligence and robotics offers enormous potential for automating work processes and making them more efficient.” Alexander Mühlens, Vice President and Head of the Low Cost Automation Business Unit at igus GmbH, also emphasised the importance of the hackathon: “The collaboration with the institutes gives us the opportunity to show our customers what is possible with the help of AI and low-cost robotics. Our dream is to be able to control robots easily via voice command, in any application.” The hackathon showed that the use of LLMs and robotics makes many other automation options conceivable, such as simultaneously packing several boxes, sorting products according to special instructions or checking ingredients for allergens using camera systems. The successful collaboration between igus, the Fraunhofer IAIS, the Lamarr Institute of the University of Bonn and the Fraunhofer IML underlines the potential of these technologies for industrial automation.

**About the partners**

Fraunhofer IAIS is one of the leading research institutes in the fields of artificial intelligence (AI), machine learning and big data, while the University of Bonn is known as a centre for artificial intelligence and robotics. The Fraunhofer IML in Dortmund contributed its expertise in logistics automation and material flow. The Lamarr Institute is a leader in international AI research and machine learning.