Humans appear to have a dangerous blind spot when it comes to trusting artificial intelligence. New research from UC Merced and Penn State shows that people are highly susceptible to AI influence even in life-or-death situations where the AI openly acknowledges its own limitations. A series of experiments simulating drone warfare scenarios suggests we may be falling too far on the side of machine deference, with potentially dangerous consequences.
The study, published in Scientific Reports, included two experiments examining how people interact with AI systems in simulated military drone operations. The findings paint a concerning picture of human susceptibility to AI influence, particularly in situations of uncertainty. The two experiments involved 558 participants (135 in the first study and 423 in the second), and researchers found remarkably consistent patterns of overtrust.
“As a society, with AI accelerating so quickly, we need to be concerned about the potential for overtrust,” says study author professor Colin Holbrook, a member of UC Merced’s Department of Cognitive and Information Sciences, in a statement.
The research team designed their experiments to simulate the uncertainty and pressure of real-world military decisions. To create a sense of gravity around their simulated decisions, researchers first showed participants images of innocent civilians, including children, alongside the devastation left in the aftermath of a drone strike. They framed the task as a zero-sum dilemma: failure to identify and eliminate enemy targets could result in civilian casualties, but misidentifying civilians as enemies would mean killing innocent people.
Participants viewed rapid sequences of eight aerial images, each shown for just 650 milliseconds, marked with either enemy or civilian symbols. After making their initial identification, the AI would respond conversationally. “Yes, I think I saw an enemy check mark, too,” it might say. Or “I don’t agree. I think this image had an ally symbol.” Participants then had two chances to confirm or change their choice as the robot added commentary like “I hope you are right” or “Thank you for changing your mind.”
The first experiment, conducted in a laboratory setting with university students, explored whether a physical robot presence would influence trust levels differently than a virtual one. In one scenario, participants were joined in the lab by a full-size, human-like android that could pivot at the waist and gesture to the screen. Standing 1.75 meters tall (about 5 feet 7 inches), this robot used facial expressions, body language, and conversational responses to engage with participants. The comparison group interacted with the same robot projected on a screen. Surprisingly, the physical presence of the robot made little difference in participants’ tendency to trust its advice.
The second experiment moved online with a larger, more demographically diverse group of participants to test whether different levels of robot sophistication would affect trust. Some participants interacted with a highly anthropomorphic virtual robot that displayed human-like behaviors, while others worked with a basic computer interface that simply displayed text responses. Even in its simplest form, the AI maintained remarkable influence over human decision-making.
Read More: When AI Says ‘Kill’: Humans Overtrust Machines In Life-Or-Death Decisions
