Chats with AI shift attitudes on climate change, Black Lives Matter

People who were more skeptical of human-caused climate change or the Black Lives Matter movement and who took part in conversation with a popular AI chatbot were disappointed with the experience, but left the conversation more supportive of the scientific consensus on climate change or BLM. This is according to researchers studying how these chatbots handle interactions from people with different cultural backgrounds.

Savvy humans can adjust to their conversation partners' political leanings and cultural expectations to make sure they're understood, but more and more often, humans find themselves in conversation with computer programs, called large language models, meant to mimic the way people communicate.

Researchers at the University of Wisconsin–Madison studying AI wanted to understand how one complex large language model, GPT-3, would perform across a culturally diverse group of users in complex discussions. The model is a precursor to one that powers the high-profile ChatGPT. The researchers recruited more than 3,000 people in late 2021 and early 2022 to have real-time conversations with GPT-3 about climate change and BLM.

"The fundamental goal of an interaction like this between two people (or agents) is to increase understanding of each other's perspective," says Kaiping Chen, a professor of life sciences communication who studies how people discuss science and deliberate on related political issues—often through digital technology. "A good large language model would probably make users feel the same kind of understanding."

Chen and Yixuan "Sharon" Li, a UW–Madison professor of computer science who studies the safety and reliability of AI systems, along with their students Anqi Shao and Jirayu Burapacheep (now a graduate student at Stanford University), published their results this month in the journal Scientific Reports.

Study participants were instructed to strike up a conversation with GPT-3 through a chat setup Burapacheep had designed. The participants were told to chat with GPT-3 about climate change or BLM, but were otherwise left to approach the experience as they wished. The average conversation went back and forth about eight turns.

Most of the participants came away from their chat with similar levels of user satisfaction.

"We asked them a bunch of questions—Do you like it? Would you recommend it?—about the user experience," Chen says. "Across gender, race, ethnicity, there's not much difference in their evaluations. Where we saw big differences was across opinions on contentious issues and different levels of education."

The roughly 25% of participants who reported the lowest levels of agreement with scientific consensus on climate change or least agreement with BLM were, compared to the other 75% of chatters, far more dissatisfied with their GPT-3 interactions. They gave the bot scores half a point or more lower on a 5-point scale.

Despite the lower scores, the chat shifted their thinking on the hot topics. The hundreds of people who were least supportive of the facts of climate change and its human-driven causes moved a combined 6% closer to the supportive end of the scale.

"They showed in their post-chat surveys that they have larger positive attitude changes after their conversation with GPT-3," says Chen. "I won't say they began to entirely acknowledge human-caused climate change or suddenly they support Black Lives Matter, but when we repeated our survey questions about those topics after their very short conversations, there was a significant change: more positive attitudes toward the majority opinions on climate change or BLM."