A swarm of artificial intelligence (AI) “crawlers” is running rampant on the Internet, scouring billions of Web sites for data to feed algorithms at leading tech companies — all without permission or payment, upending the online economy.
Before the rise of AI chatbots, Web sites allowed search engines to access their content in return for increased visibility, a system that rewarded them with traffic and advertising revenues.
However, the rapid development of generative AI has allowed tech giants such as Google and OpenAI to harvest information for their chatbots with Web crawlers, without humans ever needing to visit the original sites.
Photo: Bloomberg
Traditional content producers, such as media outlets, are being outpaced by AI crawlers, which have cut into their online operations and advertising revenues.
“Sites that gave bots access to their content used to get readers in exchange,” but the arrival or generative AI completely breaks that model, Dataiku AI strategy head Kurt Muehmel said.
Wikipedia last month reported human Internet traffic fell by 8 percent between last year and this year, because of a rise in AI search engine summaries.
“The fundamental tension is that the new business of the Internet that is AI-driven doesn’t generate traffic,” Cloudflare CEO Matthew Prince said.
Cloudflare, a US Internet services provider, processes more than 20 percent of all Internet traffic. It this year announced a new measure aimed at blocking AI crawlers from accessing content without payment or permission from Web site owners.
“It’s basically like putting a speed limit sign or a no trespassing sign,” Prince said. “Badly behaving bots can get by that, but we can track that... Over time, we can tighten these controls in a way that we’re confident the AI companies can’t get through.”
The measure, which applies to more than 10 million Web sites, has already “attracted the attention of artificial intelligence giants,” he added.
On a smaller scale, US start-up TollBit is providing online news publishers with tools to block, monitor and monetize AI crawler traffic.
“The Internet is a highway,” TollBit CEO and cofounder Toshit Panigrahi said, describing the company as a “tollbooth on the Internet.”
TollBit works with more than 5,600 sites, including USA Today, Time magazine and the Associated Press, allowing media outlets to set their own access fees.
The analytics are free for publishers, but AI companies are charged a “transaction fee for every piece of content they access,” he said.
For Muehmel, the online takeover by AI crawlers cannot be resolved with only “partial measures or by an individual company.”
“This is an evolution of the entire Internet economy, which will take years,” he said.
Prince said, if the bot swarm continues to roam freely online, “all of the incentives for content creation are going to go away,” adding that it would be “a loss, not just for us humans that want to consume it, but actually for the AI companies that need original content in order to train their systems.”
PARLIAMENT CHAOS: Police forcibly removed Brazilian Deputy Glauber Braga after he called the legislation part of a ‘coup offensive’ and occupied the speaker’s chair Brazil’s lower house of Congress early yesterday approved a bill that could slash former Brazilian president Jair Bolsonaro’s prison sentence for plotting a coup, after efforts by a lawmaker to disrupt the proceedings sparked chaos in parliament. Bolsonaro has been serving a 27-year term since last month after his conviction for a scheme to stop Brazilian President Luiz Inacio Lula da Silva from taking office after the 2022 election. Lawmakers had been discussing a bill that would significantly reduce sentences for several crimes, including attempting a coup d’etat — opening up the prospect that Bolsonaro, 70, could have his sentence cut to
A powerful magnitude 7.6 earthquake shook Japan’s northeast region late on Monday, prompting tsunami warnings and orders for residents to evacuate. A tsunami as high as three metres (10 feet) could hit Japan’s northeastern coast after an earthquake with an estimated magnitude of 7.6 occurred offshore at 11:15 p.m. (1415 GMT), the Japan Meteorological Agency (JMA) said. Tsunami warnings were issued for the prefectures of Hokkaido, Aomori and Iwate, and a tsunami of 40cm had been observed at Aomori’s Mutsu Ogawara and Hokkaido’s Urakawa ports before midnight, JMA said. The epicentre of the quake was 80 km (50 miles) off the coast of
China yesterday held a low-key memorial ceremony for the 1937 Nanjing Massacre, with Chinese President Xi Jinping (習近平) not attending, despite a diplomatic crisis between Beijing and Tokyo over Taiwan. Beijing has raged at Tokyo since Japanese Prime Minister Sanae Takaichi last month said that a hypothetical Chinese attack on Taiwan could trigger a military response from Japan. China and Japan have long sparred over their painful history. China consistently reminds its people of the 1937 Nanjing Massacre, in which it says Japanese troops killed 300,000 people in what was then its capital. A post-World War II Allied tribunal put the death toll
A passerby could hear the cacophony from miles away in the Argentine capital, the unmistakable sound of 2,397 dogs barking — and breaking the unofficial world record for the largest-ever gathering of golden retrievers. Excitement pulsed through Bosques de Palermo, a sprawling park in Buenos Aires, as golden retriever-owners from all over Argentina transformed the park’s grassy expanse into a sea of bright yellow fur. Dog owners of all ages, their clothes covered in dog hair and stained with slobber, plopped down on picnic blankets with their beloved goldens to take in the surreal sight of so many other, exceptionally similar-looking ones.
RSS