When US analysts hunting terrorists sought new ways to comb through the troves of phone records, e-mails and other data piling up as digital communications exploded over the past decade, they turned to Silicon Valley computer experts who had developed complex equations to thwart Russian mobsters intent on credit card fraud.
The partnership between the intelligence community and Palantir Technologies, a Palo Alto, California, company founded by a group of inventors from PayPal, is just one of many that the US National Security Agency (NSA) and other agencies have forged in recent years as they have rushed to unlock the secrets of “Big Data.”
Today, a revolution in software technology that allows for the highly automated and instantaneous analysis of enormous volumes of digital information has transformed the NSA, turning it into the virtual landlord of the digital assets of Americans and foreigners alike. The new technology has, for the first time, given US spies the ability to track the activities and movements of people almost anywhere in the world without actually watching them or listening to their conversations.
New disclosures that the NSA has secretly acquired the telephone records of millions of Americans, and access to e-mails, videos and other data of foreigners from nine US Internet companies have provided a rare glimpse into the growing reach of the US’ largest spy agency.
They have also alarmed the government: On Saturday night, Shawn Turner, a spokesman for the director of national intelligence, said that “a crimes report has been filed by the NSA.”
With little public debate, the NSA has been undergoing rapid expansion in order to exploit the mountains of new data being created each day. The US government has poured billions of dollars into the agency over the past decade, building a 93,000m2 fortress in the mountains of Utah, apparently to store huge volumes of personal data indefinitely. It created intercept stations across the country, according to former industry and intelligence officials, and helped build one of the world’s fastest computers to crack the codes that protect information.
While the flow of data across the Internet once appeared too overwhelming for the NSA to keep up with, the revelations of the last few days suggest that the agency’s abilities are now far greater than most outsiders believed.
“Five years ago, I would have said they don’t have the capability to monitor a significant amount of Internet traffic,” said Herbert Lin, an expert in computer science and telecommunications at the National Research Council.
Now, he said, it appears “that they are getting close to that goal.”
On Saturday, it became clear how close: Another NSA document, again cited by the Guardian, showed a “global heat map” that appeared to represent how much data the NSA sweeps up around the world.
It showed that in March there were 97 billion pieces of data collected from networks worldwide; about 14 percent of it was in Iran, much was from Pakistan and about 3 percent came from inside the US, though some of that might have been foreign data traffic routed through US-based servers.
A SHIFT IN FOCUS
The agency’s ability to mine meta-data, data about who is calling or e-mailing, has made wiretapping and eavesdropping on communications far less vital, according to data experts. That access to data from companies that Americans depend on daily raises troubling questions about privacy and civil liberties that officials in Washington, insistent on near-total secrecy, have yet to address.