Do you think engaging with an emerging tech tool can change your firmly held beliefs? Or sway you toward a decision you would not have otherwise made? Most of us humans think we are too smart for that, but mounting evidence suggests otherwise.
When it comes to a new crop of generative artificial intelligence (AI) technology, the power of “persuasion” has been identified as a potentially catastrophic risk right alongside fears that models could gain autonomy or help build a nuclear weapon. Separately, lower-stakes designs meant to influence behavior are already ubiquitous in the products many of us use everyday, nudging us to endlessly scroll on social platforms, or open Snapchat or Duolingo to continue a “streak.”
However, recent advances in that nascent technology from China are raising fresh national security concerns. New research funded by the US Department of State and released by an Australian think tank found that Chinese tech companies are on the cusp of creating and deploying technologies with “unprecedented persuasive capabilities.”
Illustration: Yusha
From a security perspective, that could be abused by Beijing or other actors to sway political opinions or sow social unrest and division. In other words, it is a weapon to subdue enemies without any fighting, the war tactic heralded by the Chinese philosopher General Sun Zi (孫子).
The Australian Strategic Policy Institute report published last week identified China’s commercial sector as “already a global leader” in the development and adoption of products designed to change attitudes or behaviors by exploiting physiological or cognitive vulnerabilities. To accomplish that, the tools rely heavily on analyzing personal data they collect and then tailor interactions with users. The paper identified a handful of Chinese firms that it says are already using such technology — spanning generative AI, virtual reality and the more emerging neurotechnology sector — to support Beijing’s propaganda and military goals.
However, that is also very much a global issue. China’s private sector might be racing ahead to develop persuasive methods, but it is following playbooks developed by US’ big tech firms to better understand their users and keep them engaged. Addressing the Beijing risk would require us to properly unpack how we let tech products influence our lives. However, fresh national security risks, combined with how AI and other new innovations can quickly scale up these tools’ effectiveness, should be a wake-up call at a time when persuasion is already so entrenched into Silicon Valley product design.
Part of what makes addressing this issue so difficult is that it can be a double-edged sword. A science study published earlier this year found that chatting with AI models could convince conspiracy theorists to reduce their beliefs, even among those who said they were important to their identity. That highlighted the positive “persuasive powers” of large language models and their ability to engage with personalized dialogue, the researchers said.
How to prevent those powers from being employed by Beijing or other bad actors for nefarious campaigns would be an increasing challenge for policymakers that goes beyond cutting off access to advanced semiconductors.
Demanding far more transparency would be one way to start, by requiring tech companies to provide clear disclosures when content is tailored in a way that could influence behaviors. Expanding data protection laws or giving users clearer ways to opt-out of having their information collected would also limit the ability of those tools to individually target users.
Prioritizing digital literacy and education is also imperative to raise awareness about persuasive technologies, how algorithms and personalized content work, how to recognize tactics and how to avoid being potentially manipulated by these systems.
Ultimately, a lot more research is needed on how to protect people from the risks of persuasive technology and it would be wise for the companies behind these tools to lead the charge, as firms such as OpenAI and Anthropic have begun doing with AI. Policymakers should also demand firms share findings with regulators and relevant stakeholders to build a global understanding of how those techniques could be exploited by adversaries. That information could then be used to set clear standards or targeted regulation.
The risk of technology so sophisticated that it allowing Beijing to pull the strings to change what you believe or who you are might still seem like a far-off, sci-fi concern. However, the stakes are too high for global policymakers to respond only after that has been unleashed. Now is the time for a global reckoning on how much personal information and influence we give tech companies over our lives.
Catherine Thorbecke is a Bloomberg Opinion columnist covering Asia tech. Previously she was a tech reporter at CNN and ABC News.
Taiwan-India relations appear to have been put on the back burner this year, including on Taiwan’s side. Geopolitical pressures have compelled both countries to recalibrate their priorities, even as their core security challenges remain unchanged. However, what is striking is the visible decline in the attention India once received from Taiwan. The absence of the annual Diwali celebrations for the Indian community and the lack of a commemoration marking the 30-year anniversary of the representative offices, the India Taipei Association and the Taipei Economic and Cultural Center, speak volumes and raise serious questions about whether Taiwan still has a coherent India
Recent media reports have again warned that traditional Chinese medicine pharmacies are disappearing and might vanish altogether within the next 15 years. Yet viewed through the broader lens of social and economic change, the rise and fall — or transformation — of industries is rarely the result of a single factor, nor is it inherently negative. Taiwan itself offers a clear parallel. Once renowned globally for manufacturing, it is now best known for its high-tech industries. Along the way, some businesses successfully transformed, while others disappeared. These shifts, painful as they might be for those directly affected, have not necessarily harmed society
Democratic Progressive Party (DPP) spokesman Justin Wu (吳崢) on Monday rebuked seven Chinese Nationalist Party (KMT) lawmakers for stalling a special defense budget and visiting China. The legislators — including Weng Hsiao-ling (翁曉玲), Yeh Yuan-chih (葉元之) and Lin Szu-ming (林思銘) — attended an event in Xiamen, China, over the weekend hosted by the Xiamen Taiwan Businessmen Association, where they met officials from Beijing’s Taiwan Affairs Office (TAO). “Weng’s decision to stall the special defense budget defies majority public opinion,” Wu said, accusing KMT legislators of acting as proxies for Beijing. KMT Legislator Wu Tsung-hsien (吳宗憲), acting head of the party’s Culture and Communications
Legislators of the opposition parties, consisting of the Chinese Nationalist Party (KMT) and the Taiwan People’s Party (TPP), on Friday moved to initiate impeachment proceedings against President William Lai (賴清德). They accused Lai of undermining the nation’s constitutional order and democracy. For anyone who has been paying attention to the actions of the KMT and the TPP in the legislature since they gained a combined majority in February last year, pushing through constitutionally dubious legislation, defunding the Control Yuan and ensuring that the Constitutional Court is unable to operate properly, such an accusation borders the absurd. That they are basing this
RSS