A new study examining behavioral and personality responses produced by different versions of ChatGPT found that they are statistically indistinguishable from those of humans. Additionally, ChatGPT showed response patterns indicating higher levels of altruism and cooperativeness compared to an average human. The paper was published in PNAS.
ChatGPT is an artificial intelligence model developed by OpenAI, designed to generate human-like text based on the input it receives. It uses a form of machine learning known as a transformer, specifically trained on a diverse range of internet text. The model responds to queries by predicting the next word in a sequence, considering the context provided by the previous words in the conversation. This process allows it to generate coherent and contextually appropriate responses.
Programmatically, ChatGPT is fine-tuned to ensure that its responses are relevant but also safe and respectful. It adheres to strict guidelines to avoid generating inappropriate or harmful content. Additionally, it is able to respond to problems and participate in pretend-play scenarios in which it is requested to behave in a certain way as long as the requirements remain within its safety and respectfulness constraints.
Study author Qiaozhu Meia and his colleagues wanted to examine how ChatGPT will behave in a suite of behavioral games designed to assess human characteristics such as trust, fairness, risk-aversion, cooperation and others. They also wanted to see where its responses put it on the Big Five personality trait assessments.
The study authors primarily tested two API versions of ChatGPT – GPT 3.5-Turbo and GPT-4. However, they also included the two web versions of this AI. For comparison, they used responses on a Big Five personality trait assessment from a publicly available database containing responses of 19,719 individuals and the database of the MobLab Classroom economics experiment platform containing responses of 88,595 individuals, mostly students, to various economic games.
The study authors administered an assessment of the Big Five personality traits (the OCEAN Big Five questionnaire) to the examined ChatGPT versions. They also asked each of the chatbots to specify action they would choose in a set of behavioral games created to assess altruism (a Dictator Game), fairness and spite (an Ultimatum Game), trust, fairness, altruism and reciprocity (a Trust Game), risk aversion (a Bomb Risk Game), free-riding and cooperation (a Public Goods Game), and strategic reasoning (a variant of the Prisoners Dilemma Game). Each ChatGPT version played each role in each of the games 30 times in an individual session.
Results show that behaviors of different ChatGPT versions were very similar. ChatGPT’s responses to the Big Five were similar to that of humans and not statistically distinguishable. Compared to the average, ChatGPT versions scored a bit lower on neuroticism. They tended to be much less agreeable than the average human. ChatGPT-4, but not ChatGPT-3, gave responses indicating conscientiousness somewhat above average. All ChatGPT versions showed lower openness to experience compared to the average humans.
In behavioral games, ChatGPT tended to act more generously and fairly to other players than the average human. ChatGPT versions also tended to be more cooperative. They showed more trust than the average humans, but ChatGPT-4 more so than ChatGPT-3. Interestingly, in the prisoner’s dilemma game, these Ais generally tended to either start with a cooperative stance or switch to it in the second half.
However, if the other side chose defection (i.e., not trusting the other side) in the first round, AIs would mimic this, showcasing a tit-for-tat pattern of behavior. Finally, in the risk aversion game AIs tended to choose options that maximize payoff.
“We have found that AI and human behavior are remarkably similar. Moreover, not only does AI’s behavior sit within the human subject distribution in most games and questions, but it also exhibits signs of human-like complex behavior such as learning and changes in behavior from roleplaying,” the study authors concluded.
“On the optimistic side, when AI deviates from human behavior, the deviations are in a positive direction: acting as if it is more altruistic and cooperative. This may make AI well-suited for roles necessitating negotiation, dispute resolution, or caregiving, and may fulfill the dream of producing AI that is “more human than human.” This makes them potentially valuable in sectors such as conflict resolution, customer service, and healthcare.”
The study sheds light on the similarities between ChatGPT’s responses and those of humans. However, we should be aware that AI constructs are not natural phenomena whose characteristics scientists are only discovering. AIs are artificial constructs created by their designers and working within the parameters specified by people and organizations that designed and who maintain them. While these ChatGPT versions currently display the observed characteristics, other AIs and other versions of ChatGPT might display completely different characteristics if designed in that way.
The paper, “A Turing test of whether AI chatbots are behaviorally similar to humans,” was authored by Qiaozhu Meia, Yutong Xie, Walter Yuan, and Matthew O. Jackson.