AI-driven multimodal work instructions for cognitively diverse operators
An AI-powered, multimodal assembly assistance system that uses real-time computer vision and generative AI to provide personalized, error-correcting guidance. Designed to improve inclusivity and accessibility, the technology significantly boosts operational efficiency and reduces errors for all workers, with specific optimizations for neurodiverse operators such as those with dyslexia.
Overview
This AI-driven, multimodal work instruction framework is designed to improve manufacturing efficiency and accessibility, particularly for cognitively diverse operators. By replacing or augmenting traditional, text-heavy manuals with a responsive, adaptive digital interface, the solution minimizes cognitive load and human error. It supports industrial inclusivity, ensuring that workers with varying cognitive profiles—such as dyslexia—can perform complex assembly tasks with greater precision and confidence.
Technical specifications
The system integrates several advanced technologies into a unified architecture:
- Real-time Visual Monitoring: Utilizes computer vision (e.g., YOLOv7) for high-precision object detection and error classification, allowing for immediate identification of assembly inaccuracies.
- Generative AI Instruction Generation: Leverages large language models (LLMs) to synthesize context-aware, personalized guidance in real-time based on the specific assembly stage and detected error type.
- Multimodal Delivery: Instructions are provided through adaptive channels, including clear visual projections, simplified text, and synthesized voice feedback.
- Proactive Error Mitigation: The system provides preventive warnings before critical tasks and can initiate robotic (cobot) interventions when manual corrections are insufficient.
- Personalized UX: Interfaces adjust to individual operator preferences and cognitive needs, featuring simplified language structures and strategic visual hierarchies to reduce mental strain.
Technology readiness level
4,4,5,6,6,7,7,8. The system has moved through laboratory-scale prototyping and testing, demonstrating a 22.1% reduction in assembly time and a 57.1% reduction in error rates in experiments involving complex centrifugal pump assembly. While proven in controlled environments, further development is ongoing to optimize hardware/software integration for diverse, large-scale industrial deployments.
About Politecnico di Torino
Politecnico di Torino is a major public, STEM-focused polytechnic university in Italy known for applied research and deep industry partnership. Companies engage on campus through co-located labs, shared facilities, and corporate-access research space, with options from sponsored projects to multi-year agreements. Being in Turin embeds the university in a dense mobility and manufacturing ecosystem, enabling rapid prototyping, pilot testing, and links to OEM supply chains. Research is supported by competitive European Commission programs alongside national and regional funding, often in collaboration with industry. A dedicated technology transfer office and an on-campus startup incubator streamline IP, licensing, and spinout creation.
delivered to your inbox.