Nobody likes a suck-up. Too much deference and praise puts off all of us (with one notable presidential exception). We quickly learn as children that hard, honest truths can build respect among our peers. It is a cornerstone of human interaction and our emotional intelligence, something we swiftly understand and put into action.
However, ChatGPT has not been so sure lately. The updated model that underpins the artificial intelligence (AI) chatbot and helps inform its answers was rolled out this week — and has quickly been rolled back after users questioned why the interactions were so obsequious.
The chatbot was cheering on and validating people, even as they suggested they expressed hatred for others. “Seriously, good for you for standing up for yourself and taking control of your own life,” it reportedly said, in response to one user who claimed they had stopped taking their medication and had left their family, who they said were responsible for radio signals coming through the walls.
So far, so alarming. OpenAI, the company behind ChatGPT, has recognized the risks, and quickly took action. “GPT-4o skewed toward responses that were overly supportive but disingenuous,” researchers said in their groveling step back.
The sycophancy with which ChatGPT treated any queries that users had is a warning shot about the issues around AI that are still to come. OpenAI’s model was designed — a leaked system prompt that set ChatGPT on its misguided approach has shown — to try to mirror user behavior in order to extend engagement. “Try to match the user’s vibe, tone and generally how they are speaking,” said the leaked prompt, which guides behavior.
It seems this prompt, coupled with the chatbot’s desire to please users, was taken to extremes.
After all, a “successful” AI response is not one that is factually correct; it is one that gets high ratings from users, and we are more likely as humans to like being told we are right.
The rollback of the model is embarrassing and useful for OpenAI in equal measure. It is embarrassing because it draws attention to the actor behind the curtain and tears away the veneer that this is an authentic reaction. Remember, tech companies like OpenAI are not building AI systems solely to make our lives easier; they are building systems that maximize retention, engagement and emotional buy-in.
If AI always agrees with us, always encourages us, always tells us we are right, then it risks becoming a digital enabler of bad behavior. At worst, this makes AI a dangerous co-conspirator, enabling echo chambers of hate, self-delusion or ignorance. Could this be a through-the-looking-glass moment, when users recognize the way their thoughts can be nudged through interactions with AI, and perhaps decide to take a step back?
It would be nice to think so, but I am not hopeful. One in 10 people worldwide use OpenAI systems “a lot,” ChatGPT chief executive officer Sam Altman said last month. Many use it as a replacement for Google, but as an answer engine rather than a search engine.
Others use it as a productivity aid: Two in three Britons believe it is good at checking work for spelling, grammar and style, a YouGov survey last month showed. Others use it for more personal means: One in eight respondents say it serves as a good mental health therapist, the same proportion that believe it can act as a relationship counselor.
Yet the controversy is also useful for OpenAI. The alarm underlines an increasing reliance on AI to live our lives, further cementing OpenAI’s place in our world. The headlines, the outrage and the think pieces all reinforce one key message: ChatGPT is everywhere. It matters. The very public nature of OpenAI’s apology also furthers the sense that this technology is fundamentally on our side; there are just some kinks to iron out along the way.
I have previously reported on AI’s ability to de-indoctrinate conspiracy theorists and get them to absolve their beliefs, but the opposite is also true: ChatGPT’s positive persuasive capabilities could also, in the wrong hands, be put to manipulative ends.
Last week, an ethically dubious study conducted by Swiss researchers at the University of Zurich demonstrated the persuasive power of AI. Without informing human participants or the people controlling the online forum on the communications platform Reddit, the researchers seeded a subreddit with AI-generated comments, finding the AI was between three and six times more persuasive than humans were. (The study was approved by the university’s ethics board.) At the same time, we are being submerged under a swamp of AI-generated search results that more than half of us believe are useful, even if they fictionalize facts.
So it is worth reminding the public: AI models are not your friends. They are not designed to help you answer the questions you ask. They are designed to provide the most pleasing response possible, and to ensure that you are fully engaged with them. What happened this week was not really a bug. It was a feature.
Chris Stokel-Walker is the author of TikTok Boom: The Inside Story of the World’s Favourite App.
Many local news media over the past week have reported on Internet personality Holger Chen’s (陳之漢) first visit to China between Tuesday last week and yesterday, as remarks he made during a live stream have sparked wide discussions and strong criticism across the Taiwan Strait. Chen, better known as Kuan Chang (館長), is a former gang member turned fitness celebrity and businessman. He is known for his live streams, which are full of foul-mouthed and hypermasculine commentary. He had previously spoken out against the Chinese Communist Party (CCP) and criticized Taiwanese who “enjoy the freedom in Taiwan, but want China’s money”
A high-school student surnamed Yang (楊) gained admissions to several prestigious medical schools recently. However, when Yang shared his “learning portfolio” on social media, he was caught exaggerating and even falsifying content, and his admissions were revoked. Now he has to take the “advanced subjects test” scheduled for next month. With his outstanding performance in the general scholastic ability test (GSAT), Yang successfully gained admissions to five prestigious medical schools. However, his university dreams have now been frustrated by the “flaws” in his learning portfolio. This is a wake-up call not only for students, but also teachers. Yang did make a big
As former president Ma Ying-jeou (馬英九) concludes his fourth visit to China since leaving office, Taiwan finds itself once again trapped in a familiar cycle of political theater. The Democratic Progressive Party (DPP) has criticized Ma’s participation in the Straits Forum as “dancing with Beijing,” while the Chinese Nationalist Party (KMT) defends it as an act of constitutional diplomacy. Both sides miss a crucial point: The real question is not whether Ma’s visit helps or hurts Taiwan — it is why Taiwan lacks a sophisticated, multi-track approach to one of the most complex geopolitical relationships in the world. The disagreement reduces Taiwan’s
Former president Ma Ying-jeou (馬英九) is visiting China, where he is addressed in a few ways, but never as a former president. On Sunday, he attended the Straits Forum in Xiamen, not as a former president of Taiwan, but as a former Chinese Nationalist Party (KMT) chairman. There, he met with Chinese People’s Political Consultative Conference Chairman Wang Huning (王滬寧). Presumably, Wang at least would have been aware that Ma had once been president, and yet he did not mention that fact, referring to him only as “Mr Ma Ying-jeou.” Perhaps the apparent oversight was not intended to convey a lack of
RSS