ITU project will study how humans react when AI systems fail
New research led by Associate Professor Paolo Burelli aims to prevent disasters at sea by tackling alarm overload, loss of trust, and confusion in increasingly autonomous vessels.
Paolo BurelliResearchCollaborationsartificial intelligence
Written 1 June, 2026 07:04 by Theis Duelund Jensen
When the cruise ship Viking Sky lost power off the coast of Norway in 2019, the danger was not only mechanical. As the ship drifted towards a rocky coastline, hundreds of alarms erupted in the control room within seconds; a flood of signals pointing in every direction but offering no clear explanation.
It was a moment when information became noise. And it is precisely that moment a new research project at the IT University of Copenhagen is designed to understand.
The project, led by Associate Professor Paolo Burelli, has received 1.75 kroner in financing from The Danish Maritime Fund and Orient’s Fond. Together, the partners are backing a research effort focused on a growing and largely underexplored risk in maritime safety: what happens when human operators are confronted with failures in complex, AI-driven systems—and cannot make sense of them.
Cognitive overload
“The issue is not only technical—it’s the meeting between human and machine,” Burelli says. “When something goes wrong, the real question is whether the operator can actually make sense of it.”
This tension between too much information and too little understanding is at the core of the new project. Research in the field shows that clear explanations from automated systems can reduce errors significantly—but that effect quickly disappears when operators are bombarded with alerts. Excessive alarms create cognitive overload, erode trust, and can ultimately lead to dangerous inaction as crews begin to ignore warnings altogether.
At ITU’s brAIn lab, Paolo Burelli and his team will investigate this problem by focusing not on the technology itself, but on the human response to it. The project examines how operators perceive, interpret, and react to failures in AI-assisted systems, especially under pressure.
“We’re trying to understand what happens cognitively when people deal with a black-box autonomous system and failure occurs—and how that affects their decisions,” says Paolo Burelli.
The research will be carried out through a series of simulated scenarios inspired by real-world incidents, including cascading failures like those seen on the Viking Sky. Participants will interact with systems that behave unpredictably, provide incomplete explanations, or offer conflicting guidance. As they do, the researchers will measure how their attention, stress levels, and decision-making processes change.
Regaining trust in technology
“We look at how people reconstruct what went wrong,” Paolo Burelli says. “Which messages help? Which ones just add noise? And how does that affect trust?”
A central ambition of the project is to understand not only how trust is lost, but how it can be regained. When systems issue false or excessive alarms, operators may begin to doubt them, a dynamic often described as a “boy who cried wolf” effect. Rebuilding trust requires clear, timely communication—but what that looks like in practice remains poorly understood, especially in the context of AI systems that are inherently uncertain.
The project addresses this through a structured research programme that examines three interconnected dimensions: how transparent AI systems are in explaining their limitations, how operators respond to different recovery strategies during failures, and how communication design influences trust over time. By linking these factors to physiological markers of cognitive load and stress, the researchers aim to move beyond intuition and establish measurable thresholds for safe and effective system design.
Underlying the project is a shift in how safety is approached in the age of AI. Instead of assuming that automation will eliminate human error, the research starts from the opposite premise: that failures are inevitable and may even become more frequent as systems grow more complex.
“We assume that automated systems will fail,” Burelli says. “The key is how fast people can understand what’s happening and recover from it. As AI-assisted decision-making becomes commonplace across industries, devising new standards is of critical importance.”
Theis Duelund Jensen, Press Officer, phone +45 2555 0447, email thej@itu.dk