A swarm of artificial intelligence (AI) “crawlers” is running rampant on the Internet, scouring billions of Web sites for data to feed algorithms at leading tech companies — all without permission or payment, upending the online economy.
Before the rise of AI chatbots, Web sites allowed search engines to access their content in return for increased visibility, a system that rewarded them with traffic and advertising revenues.
However, the rapid development of generative AI has allowed tech giants such as Google and OpenAI to harvest information for their chatbots with Web crawlers, without humans ever needing to visit the original sites.
Photo: Bloomberg
Traditional content producers, such as media outlets, are being outpaced by AI crawlers, which have cut into their online operations and advertising revenues.
“Sites that gave bots access to their content used to get readers in exchange,” but the arrival or generative AI completely breaks that model, Dataiku AI strategy head Kurt Muehmel said.
Wikipedia last month reported human Internet traffic fell by 8 percent between last year and this year, because of a rise in AI search engine summaries.
“The fundamental tension is that the new business of the Internet that is AI-driven doesn’t generate traffic,” Cloudflare CEO Matthew Prince said.
Cloudflare, a US Internet services provider, processes more than 20 percent of all Internet traffic. It this year announced a new measure aimed at blocking AI crawlers from accessing content without payment or permission from Web site owners.
“It’s basically like putting a speed limit sign or a no trespassing sign,” Prince said. “Badly behaving bots can get by that, but we can track that... Over time, we can tighten these controls in a way that we’re confident the AI companies can’t get through.”
The measure, which applies to more than 10 million Web sites, has already “attracted the attention of artificial intelligence giants,” he added.
On a smaller scale, US start-up TollBit is providing online news publishers with tools to block, monitor and monetize AI crawler traffic.
“The Internet is a highway,” TollBit CEO and cofounder Toshit Panigrahi said, describing the company as a “tollbooth on the Internet.”
TollBit works with more than 5,600 sites, including USA Today, Time magazine and the Associated Press, allowing media outlets to set their own access fees.
The analytics are free for publishers, but AI companies are charged a “transaction fee for every piece of content they access,” he said.
For Muehmel, the online takeover by AI crawlers cannot be resolved with only “partial measures or by an individual company.”
“This is an evolution of the entire Internet economy, which will take years,” he said.
Prince said, if the bot swarm continues to roam freely online, “all of the incentives for content creation are going to go away,” adding that it would be “a loss, not just for us humans that want to consume it, but actually for the AI companies that need original content in order to train their systems.”
DENIAL: Pyongyang said a South Korean drone filmed unspecified areas in a North Korean border town, but Seoul said it did not operate drones on the dates it cited North Korea’s military accused South Korea of flying drones across the border between the nations this week, yesterday warning that the South would face consequences for its “unpardonable hysteria.” Seoul quickly denied the accusation, but the development is likely to further dim prospects for its efforts to restore ties with Pyongyang. North Korean forces used special electronic warfare assets on Sunday to bring down a South Korean drone flying over North Korea’s border town. The drone was equipped with two cameras that filmed unspecified areas, the General Staff of the North Korean People’s Army said in a statement. South Korea infiltrated another drone
COMMUNIST ALIGNMENT: To Lam wants to combine party chief and state presidency roles, with the decision resting on the election of 200 new party delegates next week Communist Party of Vietnam General Secretary To Lam is seeking to combine his party role with the state presidency, officials said, in a move that would align Vietnam’s political structure more closely to China’s, where President Xi Jinping (習近平) heads the party and state. Next week about 1,600 delegates are to gather in Hanoi to commence a week-long communist party congress, held every five years to select new leaders and set policy goals for the single-party state. Lam, 68, bade for both top positions at a party meeting last month, seeking initial party approval ahead of the congress, three people briefed by
Indonesia and Malaysia have become the first countries to block Grok, the artificial intelligence (AI) chatbot developed by Elon Musk’s xAI, after authorities said it was being misused to generate sexually explicit and nonconsensual images. The moves reflect growing global concern over generative AI tools that can produce realistic images, sound and text, while existing safeguards fail to prevent their abuse. The Grok chatbot, which is accessed through Musk’s social media platform X, has been criticized for generating manipulated images, including depictions of women in bikinis or sexually explicit poses, as well as images involving children. Regulators in the two Southeast Asian
ICE DISPUTE: The Trump administration has sought to paint Good as a ‘domestic terrorist,’ insisting that the agent who fatally shot her was acting in self-defense Thousands of demonstrators chanting the name of the woman killed by a US federal agent in Minneapolis, Minnesota, took to the city’s streets on Saturday, amid widespread anger at use of force in the immigration crackdown of US President Donald Trump. Organizers said more than 1,000 events were planned across the US under the slogan “ICE, Out for Good” — referring to the US Immigration and Customs Enforcement, which is drawing growing opposition over its execution of Trump’s effort at mass deportations. The slogan is also a reference to Renee Good, the 37-year-old mother shot dead on Wednesday in her
RSS