More than 1,100km from Wall Street, the New York Stock Exchange’s (NYSE) backup data center on Cermak Road in Chicago is supposed to safeguard US markets, standing by at all hours in case disaster strikes the world’s largest venue for trading shares.
When markets are closed, it participates in a well-worn routine, with NYSE staffers turning on and off systems to ensure everything works.
Heading into Tuesday, an NYSE employee failed to properly shut down Cermak’s disaster-recovery system — leading to a disaster.
Photo: Bloomberg
That error, described by people with direct knowledge of NYSE’s internal operations, is what triggered wild market swings when trading opened on Tuesday in Manhattan. The chaos affected more than 250 companies including Wells Fargo & Co, McDonald’s Corp, Walmart Inc and Morgan Stanley, in some cases sending stock prices swinging by 25 percentage points in a matter of minutes.
The episode prompted the exchange to cancel thousands of trades at a cost that is still being determined. Meanwhile, market professionals and day traders are rattled and waiting for the exchange to elaborate on what it publicly called a “manual error” involving its “disaster recovery configuration.”
“They’re going to need to come up with something better” to reassure investors and regulators, said Joseph Saluzzi, a partner and cofounder of Themis Trading LLC, whose firm avoided losses.
“There’s zero tolerance” for systems failures “when it comes to the opening and the close,” he said.
The turmoil resulted because the backup system in Cermak was left running, misleading the exchange’s computers to treat the 9:30am opening bell as a continuation of trading, and so they skipped the day’s opening auctions, which neatly set initial prices, one of the people said.
Without that routine step, orders flowed through at prices all over the place. That soon tripped circuit breakers designed to prevent excessive market swings, setting off alarms on screens across the investing world.
NYSE executives spent hours pinpointing the problem until they were confident there would not be further fallout, the people said on condition of anonymity.
Officials also began reviewing whether the dislocated trades could be labeled “clearly erroneous” under the market’s rules and then canceled.
Shortly before 3pm on Tuesday, the exchange said it would unwind the most egregious transactions.
“A core value of [Intercontinental Exchange Inc] and the NYSE is our commitment to collaboration,” a spokesperson for NYSE said in a statement on Thursday. “The issues around our market open on Tuesday are our collective responsibility, and we have moved swiftly and decisively to resolve them as a team.”
NYSE has yet to determine how much the debacle may cost. It’s already fielding claims from firms under the exchange’s rules, the people said.
The venue gathers about US$500,000 per month for a fund to compensate exchange members — mostly broker-dealers representing their investor clients — when such mishaps occur. Decisions on claims for Tuesday’s episode are to be made by the end of the month.
Behind closed doors, NYSE executives are grappling not just with the potential cost, but the impact on the exchange’s reputation. They are in contact with regulators and lawmakers to explain what happened.
They plan to examine the platform’s procedures and management, potentially reworking rules to be more flexible and provide further protections.
The exchange had not determined as of late Wednesday whether anyone would face disciplinary action.
PATENTS: MediaTek Inc said it would not comment on ongoing legal cases, but does not expect the legal action by Huawei to affect its business operations Smartphone integrated chips designer MediaTek Inc (聯發科) on Friday said that a lawsuit filed by Chinese smartphone brand Huawei Technologies Co (華為) over alleged patent infringements would have little impact on its operations. In an announcement posted on the Taiwan Stock Exchange, MediaTek said that it would not comment on an ongoing legal case. However, the company said that Huawei’s legal action would have little impact on its operations. MediaTek’s statement came after China-based PRIP Research said on Thursday that Huawei filed a lawsuit with a Chinese district court claiming that MediaTek infringed on its patents. The infringement mentioned in the lawsuit likely involved
Taipei is today suspending work, classes and its US$2.4 trillion stock market as Typhoon Gaemi approaches Taiwan with strong winds and heavy rain. The nation is not conducting securities, currency or fixed income trading, statements from its stock and currency exchanges said. Authorities had yesterday issued a warning that the storm could affect people on land and canceled some ship crossings and domestic flights. Taiwan Semiconductor Manufacturing Co (TSMC, 台積電) expects its local chipmaking fabs to maintain normal production, the company said in an e-mailed statement. The main chipmaker for Apple Inc and Nvidia Corp said it has activated routine typhoon alert
GROWTH: TSMC increased its projected revenue growth for this year to more than 25 percent, citing stronger-than-expected demand for AI devices and smartphones The Taiwan Institute of Economic Research (TIER, 台灣經濟研究院) yesterday raised its forecast for Taiwan’s GDP growth this year from 3.29 percent to 3.85 percent, as exports and private investment recovered faster than it predicted three months ago. The Taipei-based think tank also expects that Taiwan would see a 8.19 percent increase in exports this year, better than the 7.55 percent it projected in April, as US technology giants spent more money on artificial intelligence (AI) infrastructure and development. “There will be more AI servers going forward, but it remains to be seen if the momentum would extend to personal computers, smartphones and
Catastrophic computer outages caused by a software update from one company have once again exposed the dangers of global technological dependence on a handful of players, experts said on Friday. A flawed update sent out by the little-known security firm CrowdStrike Holdings Inc brought airlines, TV stations and myriad other aspects of daily life to a standstill. The outages affected companies or individuals that use CrowdStrike on the Microsoft Inc’s Windows platform. When they applied the update, the incompatible software crashed computers into a frozen state known as the “blue screen of death.” “Today CrowdStrike has become a household name, but not in