A new report written by a former Pentagon official who helped establish US policy on autonomous weapons argues that such weapons could be uncontrollable in real-world environments where they are subject to design failure as well as hacking, spoofing and manipulation by adversaries.
In recent years, low-cost sensors and new artificial intelligence technologies have made it increasingly practical to design weapons systems that make killing decisions without human intervention.
The specter of so-called “killer robots” has touched off an international protest movement and a debate within the UN about limiting the development and deployment of such systems.
The new report was written by Paul Scharre, who directs a program on the future of warfare at the Center for a New American Security, a policy research group in Washington.
Scharre worked in the office of the US secretary of defense from 2008 to 2013, where he helped establish US policy on unmanned and autonomous weapons. He was also one of the authors of a 2013 US Department of Defense directive that set military policy on the use of such systems.
In the report, titled Autonomous Weapons and Operational Risk, which was published yesterday, Scharre warns about a range of real-world risks associated with weapons systems that are completely autonomous.
The report contrasts these completely automated systems, which have the ability to target and kill without human intervention, to weapons that keep humans “in the loop” in the process of selecting and engaging targets.
Scharre, who served as an Army Ranger in Iraq and Afghanistan, focuses on the potential types of failures that might occur in completely automated systems, as opposed to the way such weapons are intended to work.
The report enumerates a history of the types of failures that have occurred in military and commercial systems that are highly automated.
“Anyone who has ever been frustrated with an automated telephone call support helpline, an alarm clock mistakenly set to ‘pm’ instead of ‘am,’ or any of the countless frustrations that come with interacting with computers, has experienced the problem of ‘brittleness’ that plagues automated systems,” Scharre writes.
His underlying point is that autonomous weapons systems inevitably lack the flexibility that humans have to adapt to novel circumstances and that as a result killing machines are likely to make mistakes that humans would presumably avoid.
Completely autonomous weapons are beginning to appear in military arsenals. For example, South Korea has deployed an automated sentry gun along the demilitarized zone with North Korea, and Israel operates a drone aircraft that attacks enemy radar systems when they are detected.
The center’s report focuses on a range of unexpected behavior in highly computerized systems like system failures and bugs, as well as unanticipated interactions with the environment.
“On their first deployment to the Pacific, eight F22 fighter jets experienced a Y2K-like total computer failure when crossing the international date line,” the report states. “All onboard computer systems shut down, and the result was nearly a catastrophic loss of the aircraft. While the existence of the international date line could clearly be anticipated, the interaction of the dateline with the software was not identified in testing.”
Nearly half of China’s major cities are suffering “moderate to severe” levels of subsidence, putting millions of people at risk of flooding, especially as sea levels rise, according to a study of nationwide satellite data released yesterday. The authors of the paper, published by the journal Science, found that 45 percent of China’s urban land was sinking faster than 3mm per year, with 16 percent at more than 10mm per year, driven not only by declining water tables, but also the sheer weight of the built environment. With China’s urban population already in excess of 900 million people, “even a small portion
UNSETTLING IMAGES: The scene took place in front of TV crews covering the Trump trial, with a CNN anchor calling it an ‘emotional and unbelievably disturbing moment’ A man who doused himself in an accelerant and set himself on fire outside the courthouse where former US president Donald Trump is on trial has died, police said yesterday. The New York City Police Department (NYPD) said the man was declared dead by staff at an area hospital. The man was in Collect Pond Park at about 1:30pm on Friday when he took out pamphlets espousing conspiracy theories, tossed them around, then doused himself in an accelerant and set himself on fire, officials and witnesses said. A large number of police officers were nearby when it happened. Some officers and bystanders rushed
Beijing is continuing to commit genocide and crimes against humanity against Uyghurs and other Muslim minorities in its western Xinjiang province, U.S. Secretary of State Antony Blinken said in a report published on Monday, ahead of his planned visit to China this week. The State Department’s annual human rights report, which documents abuses recorded all over the world during the previous calendar year, repeated language from previous years on the treatment of Muslims in Xinjiang, but the publication raises the issue ahead of delicate talks, including on the war in Ukraine and global trade, between the top U.S. diplomat and Chinese
HYPOCRISY? The Chinese Ministry of Foreign Affairs yesterday asked whether Biden was talking about China or the US when he used the word ‘xenophobic’ US President Joe Biden on Wednesday called for a hike in steel tariffs on China, accusing Beijing of cheating as he spoke at a campaign event in Pennsylvania. Biden accused China of xenophobia, too, in a speech to union members in Pittsburgh. “They’re not competing, they’re cheating. They’re cheating and we’ve seen the damage here in America,” Biden said. Chinese steel companies “don’t need to worry about making a profit because the Chinese government is subsidizing them so heavily,” he said. Biden said he had called for the US Trade Representative to triple the tariff rates for Chinese steel and aluminum if Beijing was