US firm pushed to brink by China hack attack

A family-owned firm came under relentless assault after accusing China of pirating its software to build the Green Dam cybercensor

By Michael Riley  /  Bloomberg

Tue, Dec 04, 2012 - Page 9

During his civil lawsuit against the People’s Republic of China, Brian Milburn says he never once saw one of the country’s lawyers. He read no court documents from China’s attorneys because they filed none. The voluminous case record at the US District courthouse in Santa Ana, California, contains a single communication from China: a curt letter to the US Department of State, urging that the suit be dismissed.

That does not mean Milburn’s adversary had no contact with him.

For three years, a group of hackers from China waged a relentless campaign of cyberharassment against Solid Oak Software, Milburn’s family-owned, eight-person firm in Santa Barbara, California.

The attack began less than two weeks after Milburn publicly accused China of appropriating his company’s parental filtering software, CYBERsitter, for a national Internet censoring project. And it ended shortly after he settled a US$2.2 billion lawsuit against the Chinese government and a string of computer companies in April.

In between, the hackers assailed Solid Oak’s computer systems, shutting down Web and e-mail servers, spying on an employee with her webcam, and gaining access to sensitive files in a battle that caused company revenues to tumble and brought it within a hair’s breadth of collapse.

As the public dispute unfolded in decorous courtrooms, Milburn’s computer prowess was tested to its limits in what amounted to a digital home invasion by what he later learned was one of the most prolific hacking teams in China. He waged his own desperate one-man fight without weapons or help from authorities, swapping out servers, puzzling over middle-of-the-night malfunctions, and watching his sales all but evaporate — his every keystroke monitored by spies who had turned his technology against him.

Milburn, 61, rarely took a day off during that time as he struggled around the clock to keep his computer network running and his firm afloat. He doubts he will ever know exactly what was going on, but he has theories.

“It felt like they had a plan,” says Milburn, sitting in his office two blocks from Santa Barbara’s main drag, where he is now focused on rebuilding his business. “If they could just put the company out of business, the lawsuit goes away. They didn’t need guys with guns or someone to break my kneecaps.”

The cyberattack against Solid Oak provides a rare look at the clandestine methods in play as high-tech spies and digital combatants seek to gain a brass-knuckle advantage in the global economy, from trade disputes to big-dollar deals to lawsuits.

US officials say that China in particular uses its national security apparatus for such intrusions, targeting thousands of US and European corporations and blurring the traditional lines of espionage.

While his civil case was pending, Milburn did not discuss the cyberintrusion publicly, saying only that the company and its Los Angeles-based law firm had received e-mails containing spyware. He had no idea who was behind it until last August, when he provided malware samples to a security firm at the request of a reporter.

A forensic analysis of the malware by Joe Stewart, a threat expert at Atlanta-based Dell SecureWorks, identified the intruders who rifled Solid Oak’s networks as a team of Shanghai-based hackers involved in a string of sensitive national security-related breaches going back years.

Commercial hacker hunters — who refer to the team as the Comment group, for the hidden program code they use known as “comments” — tie it to a multitude of victims that include the the president of the EU Council, major defense contractors and even US President Barack Obama’s 2008 presidential campaign. The group has been linked to the People’s Liberation Army, China’s military, according to leaked classified cables.

The Solid Oak attack is a micro tale of what some of the US and Europe’s largest corporations have experienced, says US Representative Mike Rogers, a Republican who chairs the US House of Representatives Intelligence Committee.

The campaign to steal private files and intellectual property, even to the point of collapsing businesses, amounts to a criminal racket for commercial gain, Rogers says.

“I used to work organized crime in Chicago — I don’t know, but it sure seems like there are a lot of similarities,” says Rogers, a former FBI agent.

Headquartered in a converted Victorian house, Milburn’s small company seems an unlikely candidate to become entangled in an international feud with China, except for one thing: It was a market leader in the US for software that lets parents and schools block objectionable Web content, like pornography and violence.

China was looking for software to do the same thing on a national scale. In May 2009, Chinese officials ordered Web-filtering software called Green Dam Youth Escort installed on every computer sold in the country. They touted the software’s ability to protect young Internet users by filtering pornography. Critics in China, who identified more than 6,000 political keyword filters, branded it an extension of China’s censorship regime.

When University of Michigan researchers examined the program in June 2009 to see how it worked, they discovered that thousands of lines of code directly matched Milburn’s software, which has 1.1 million active users. Included, apparently by mistake, was a CYBERsitter upgrade announcement — the “smoking gun” that the software had been pirated, according to Milburn.

An independent analysis later found that four of the five active filters were copied almost verbatim from CYBERsitter and that Green Dam could not operate correctly when those filters were disabled. It is possible the code was stolen in an earlier hack, but Milburn believes the thieves simply bought a copy and broke the encryption protecting the code.

In interviews with reporters, he said he was considering a lawsuit and vowed to pursue an injunction.

On June 24 — 12 days after Milburn went public with his legal intentions — the hackers made their first appearance. Working from her home office 240km south of Santa Barbara in Orange County, Jenna DiPasquale, 39, who is Milburn’s daughter and Solid Oak’s one-woman marketing department, received a carefully forged e-mail containing hidden spyware.

It looked like a routine message from Milburn, so DiPasquale clicked on the attachment, realizing only later that the e-mail address was a couple of letters off. Solid Oak employees received more bogus e-mails over the next few days, setting off alarm bells.

Milburn contacted Matthew Thomlinson, a Microsoft threat expert for help. Thomlinson found the malware had downloaded software that burrowed into the company’s Microsoft operating system, automatically uploading more tools the hackers could use to control the network remotely. The malware had been created on a Chinese-language computer, he concluded.

As far as Milburn knew, though, his attackers could have been anyone from seasoned professionals to hacktivists tapping on a keyboard in a Beijing basement, he says.

The more urgent question was whether the attackers were behind the strange things that began happening in his network.

DiPasquale was at her desktop computer, helping the company’s attorneys with research sometime in August, when she noticed the light on her webcam come on. A few days later, a message flashed on her laptop indicating that the camera on that machine had been activated as well. She made an alarmed call to Milburn. After learning that Chinese hackers had eavesdropped on the Dalai Lama and his staff using their own computers, he went through the office, covering every webcam and microphone with black electrical tape.

Then the company’s e-mail servers began shutting down, sometimes two or three times a week, slowing e-mail traffic, the main way the company provides customer service. Similar problems began plaguing the Web servers — a bigger problem since Web sales of CYBERsitter supply more than half of Solid Oak’s revenue. By September and October, Web site sales were off 55 percent from mid-year and Milburn was struggling to figure out how the hackers might be behind it.

“I panicked,” says Milburn, who combines a beach comber’s countenance with the nervous energy of a workaholic. “What the hell is happening to my income, where is the money going, why aren’t we getting orders?”

“This slow realization came that, wait a second, they’re coming after us now,” says DiPasquale, who felt she could no longer trust her own computer. “It was very scary.”

Milburn had contacted the FBI after the flurry of e-mail assaults, and an agent from the Seattle field office called and took details, including samples of the malware and, later, server logs, he says.

However, the agency shed almost no light on the situation, he says, and he was never told if the material was useful.

That does not mean the bureau was in the dark about Milburn’s attackers. US law enforcement and intelligence officials had amassed a long dossier on the group, which they had been tracking since 2002, according to leaked cables and two people familiar with government investigations into the group.

Laura Eimiller, an FBI spokeswoman in Los Angeles, said the bureau could not comment on its interactions with Solid Oak or any investigation.

Milburn forged ahead in court in an attempt to win damages for the alleged theft. He and his small team of lawyers had spent six months analyzing the similarities in the two software programs. He filed suit in January 2010 against the Chinese government and two Chinese software companies that had developed Green Dam.

Milburn’s suit also named seven big computer manufacturers, including Sony and Lenovo Group, which the suit alleges had begun installing or distributing the software in the program’s early phases.

As in the digital fight, not all of Milburn’s legal adversaries were what they seemed. Zhengzhou Jinhui Computer System Engineering Co, one of the two Chinese companies that developed Green Dam, had ties to the People’s Liberation Army University, a research center for China’s military, according to a June 2009 US embassy diplomatic cable published by Wikileaks the following year.

No one from Zhengzhou Jinhui was available to address the CYBERsitter allegations, according to a person who answered the telephone at the company.

A spokesman for China’s Ministry of Foreign Affairs said he had no information on the cyberassault against Solid Oak and declined to comment further.

When Milburn’s suit was filed, Chinese officials said the government “highly values and fully respects the intellectual property rights of software.”

Six days after the suit was filed on Jan. 5, 2010, Milburn’s Los Angeles-based law firm at the time, Gipson Hoffman & Pancione, was hit with a cyberintrusion using e-mails similar to those aimed at Solid Oak but with different malware, according to the law firm. Forensics analysis shows that attack probably emanated from China as well, says Stewart, the Dell SecureWorks threat expert.

It had been clear to everyone that one motive for the attacks might be espionage related to possible legal action, Milburn says. If the hackers were able to steal documents or record conversations, they could preview strategies and negotiating positions, even identify legal weaknesses in the case.

Milburn decided not to take chances with the lawsuit. Using techniques gleaned from talking to security experts, his small team developed their own ad hoc counter-espionage measures. Solid Oak and its lawyers exchanged legal documents using rotating Web mail accounts or document-sharing sites like San Francisco-based Dropbox, deleting the accounts after a single use.

Occasionally, Milburn drove to an empty house he and his wife owned in the hills around Santa Barbara. Sitting at the kitchen table, he would make phone calls or exchange e-mails with his attorneys, alternating between four different cell phones from three different carriers.

The lawsuit seemed to trigger a more serious phase of the attack, Milburn says.

Computer failures that had occurred a couple times a week now sometimes happened two or three times a day.

Milburn constantly had to reboot servers, occasionally in the middle of the night. During work hours, it became hard for DiPasquale to get Milburn on the phone because he always seemed preoccupied fixing something. Tempers at work flared more often.

“Everybody started to wonder what they were doing wrong on a personal level,” DiPasquale says, adding that because Milburn could not trace the source of the network problems, it became hard to sort out who was to blame or why.

“Things got very tense,” DiPasquale says.

One thing was clear: the technology that ran Milburn’s company was no longer solely under his control.

In March 2010, a staccato of text message alarms woke him in the middle of the night, signaling that his servers were all shutting down. He hurriedly drove the winding road to the office to find that his commercial-grade SonicWALL firewall had failed, taking his entire network off line. He spent a good part of the next day on the phone with the manufacturer, who was stumped.

“Those things are like old carburetor engines, they never quit,” Milburn says.

After his e-mail servers crashed during an exchange with his attorneys, he crawled under the large house that serves as the company’s headquarters in search of a device that someone might have physically planted. Pawing through cobwebs with a flashlight, he spent an hour opening utility boxes and eyeing the fiber-optic cable. He found nothing.

Milburn says he was riding “that fine line between ultra-caution and paranoia.”

Born in Santa Monica, Milburn did not graduate from high school, but he has a relentlessly autodidactic drive that is common in early tech entrepreneurs. He taught himself how to write code, and eventually mastered complex Internet software protocols.

Laura Milburn, 63, his wife of 21 years, calls him “brilliant,” but also “incredibly stubborn.”

A few years earlier she watched him in a legal tussle with a neighbor who had built a deck over what they thought was their property line. Milburn ended up spending more than US$100,000 in a year-long fight just so they could split the difference, she says.

“He’s not the kind of person who would back down to someone because they threaten him,” Laura Milburn says.

Even so, “I don’t think he had a clue what he was getting into,” she adds.

Both of those traits explain why Milburn did not hire an expensive incident response team to hunt the hackers down in his network — the kind larger corporations often use.

Milburn, after all, had built Solid Oak’s network himself.

“I thought they might be able to get around some IT guy, but there’s no way they were going to get around me,” he says.

Milburn learned everything he could about computer security. He read professional papers and called up experts he knew. He began writing his own software to monitor the connections his computers were making to outside networks, looking for tell-tale signs of the hackers at work.

In April 2010, during a 6:30am check of his servers — by then part of his daily routine — Milburn stumbled on a folder buried in an obscure Microsoft directory, one that was normally unused. What he found inside startled him. The file contained the encrypted versions of all eight passwords in his system — the keys to the entire network. The hackers could use the passwords to control just about anything he could, from Web servers to e-mail.

The folder was gone two days later, he says, and in its place were several pieces of software he did not recognize. Later, he found out they were custom-designed software the hackers use to perform tasks on corporate networks. He had found their tool kit.

Rather than panic, Milburn said he felt an adrenalin rush.

“It was like, ‘okay, now I can figure out what they’re doing,’” he says.

After months of detective work, Milburn was no longer chasing ghosts.

Even at the best of times, Solid Oak’s headquarters is a warren of server rooms and cluttered offices that, Milburn says, could sometimes resemble the inside of a well-maintained garage.

In the summer of 2010, it reflected the disarray of a company in crisis, littered with the results of Solid Oak’s two on-going battles, one legal, one digital.

The firewall that blew out in March, a small box the size of an office telephone, still sat propped in a chair. Stacks of legal documents covered tables and spilled onto the floor. Two 18m data cables — which Milburn could use in a pinch to circumvent his own compromised e-mail system via a commercial Internet connection — ran from one end of the office to the other.

Milburn’s biggest concern was that the hackers seemed to be trying to hit the heart of his business. The lawsuit months earlier had brought a rush of publicity for CYBERsitter, and Milburn released a new version of the software. That combination would normally boost sales.

While bulk sales and orders over the phone were up, 60 percent of Solid Oak’s business depended on users buying the US$39.95 program directly from the Web site. As the network problems continued, so did the fall in sales. Milburn would not provide month-to-month sales figures, saying it could aid competitors, but he says the normally profitable company dipped into the red after a big drop in Web sales the month the lawsuit was filed. Net losses averaged US$58,000 a month after that, even as Milburn slashed expenses, he says.

Tracing the drop, he could see that customers were coming to the Web site to buy the software like always. They would type in credit card numbers and click submit, but most of the orders — on some days 98 percent — were not going through, Milburn says. He replaced servers and tried other fixes. Nothing worked.

As his income dried up, Milburn kept the company afloat in part with insurance proceeds from the loss of two properties in the November 2008 Tea Fire in the hills of Santa Barbara that burned 210 homes over three days.

He went without pay, and DiPasquale agreed to forgo her salary for a few months too. She and her husband, a professional chef, drew down their savings, but by the summer of 2010, the money was running out.

Some tough conversations played out at home, DiPasquale says. She argued that what was going on was wrong; quitting would mean the hackers had won.

Her husband wondered exactly what they had gotten into and where it would end.

“He was saying: ‘What are we up against? Is there going to be someone sitting outside the house?” she says.

Because she was working alone at home, he made sure the house alarm was on every day before leaving for work.

In his own battle, Milburn became more obsessed. He would get up by 5am, work until 7pm grab something to eat, then sign on from home to check his servers again. Constantly missing meals, Milburn began subsisting on pre-packaged sandwiches from a convenience store close to the office.

“It would be ten o’clock at night and I’d get an idea, ‘Huh, let me just check this,’” Milburn says. “That would lead to another hour of frustration trying to figure something out.”

Examining the script that controlled the payment processing function in November that year, he noticed that a single character was missing from the string — an apostrophe. That was enough to cause the page to time out, rather than to complete the credit card transaction. Customers were leaving in frustration.

The apostrophe was sometimes there and sometimes not, so some payments went through. There may have been other ways that the hackers were sabotaging his sales, but Milburn was sure he had found at least one.

“A hacker could certainly edit the script and break it so it would not work,” says Stewart, the Dell SecureWorks threat expert. “That would be a great way to do it without calling attention to the fact that they were in the system.”

No one ever told Milburn that he was facing not amateurs, but professionals who had ransacked secure US government networks, until the results of Stewart’s analysis last August.

The tools Milburn found in his network were unique to the Comment group, according to Stewart.

They included software designed to let the hackers send out stolen files and steal security credentials.

Without a more in-depth investigation, Stewart said it was difficult, if not impossible, to determine the hackers’ goal as they rifled Milburn’s network. Some of what Milburn experienced, including repeated and regular crashing of his servers, could have been an unintended side-effect as the hackers infested the network with backdoors and other malware.

Or it might have been deliberate. From a hacker’s point of view, everything Milburn experienced is technically “pretty elementary,” says Nicholas Percoco, who heads SpiderLabs, a Chicago-based security division of Trustwave.

Percoco and his team are paid by corporations to hack into their networks to test security — what is known as penetration testing.

“If I can do it, the Chinese certainly can do it,” he says.

At one point, Milburn was able to identify a server that the hackers appeared to be using as a staging point to attack other targets. He was never able to shut down their activities, though.

In August last year, a California district judge rejected a move by some of the defendants to shift Solid Oak’s lawsuit to China, and ruled that it could go ahead in a US court. Negotiations for settlement moved forward in earnest.

Solid Oak reached agreement with defendants for an undisclosed sum in February, and the case was dismissed two months later. Milburn says he cannot discuss the terms, including exactly which defendants participated.

His attorney, Gregory Fayer, now at Fayer Gipson, says the Chinese government, which had by then declared that the Green Dam program would be strictly voluntary, was not among them.

In US District Court in California, the presiding judge declared China in default in the lawsuit for failing to respond.

Within two months of the settlement, Milburn says, the unusual activity in the company’s computer network had nearly stopped.

The wild ride of those three years did more than wreak havoc on Solid Oak’s computers. It threw into question Milburn’s retirement plans, he says.

During the worst moments, he wondered if he would have to start over, get rid of the CYBERsitter domain name and try again under a new digital identity, just to be free of his adversaries.

Milburn now feels he can move on, even if he did not prevail. Sales have not fully recovered, but he says he now has a chance to rebuild his customer base.

“It turns out they were just better than me,” says Milburn, whose doctor recently diagnosed him with a stress-related ailment.

“But it was the right thing to do,” he says. “You don’t do anybody a favor by not taking a stand on this kind of stuff.”

With the company’s finances now more stable, DiPasquale recently went out and bought a new computer.

“I just wanted to tie the last one to an anvil and toss it in the sea,” she says.

Even so, DiPasquale says: “I don’t think I’ll ever feel completely safe on my own computer again.”