A swarm of artificial intelligence (AI) “crawlers” is running rampant on the Internet, scouring billions of Web sites for data to feed algorithms at leading tech companies — all without permission or payment, upending the online economy.
Before the rise of AI chatbots, Web sites allowed search engines to access their content in return for increased visibility, a system that rewarded them with traffic and advertising revenues.
However, the rapid development of generative AI has allowed tech giants such as Google and OpenAI to harvest information for their chatbots with Web crawlers, without humans ever needing to visit the original sites.
Photo: Bloomberg
Traditional content producers, such as media outlets, are being outpaced by AI crawlers, which have cut into their online operations and advertising revenues.
“Sites that gave bots access to their content used to get readers in exchange,” but the arrival or generative AI completely breaks that model, Dataiku AI strategy head Kurt Muehmel said.
Wikipedia last month reported human Internet traffic fell by 8 percent between last year and this year, because of a rise in AI search engine summaries.
“The fundamental tension is that the new business of the Internet that is AI-driven doesn’t generate traffic,” Cloudflare CEO Matthew Prince said.
Cloudflare, a US Internet services provider, processes more than 20 percent of all Internet traffic. It this year announced a new measure aimed at blocking AI crawlers from accessing content without payment or permission from Web site owners.
“It’s basically like putting a speed limit sign or a no trespassing sign,” Prince said. “Badly behaving bots can get by that, but we can track that... Over time, we can tighten these controls in a way that we’re confident the AI companies can’t get through.”
The measure, which applies to more than 10 million Web sites, has already “attracted the attention of artificial intelligence giants,” he added.
On a smaller scale, US start-up TollBit is providing online news publishers with tools to block, monitor and monetize AI crawler traffic.
“The Internet is a highway,” TollBit CEO and cofounder Toshit Panigrahi said, describing the company as a “tollbooth on the Internet.”
TollBit works with more than 5,600 sites, including USA Today, Time magazine and the Associated Press, allowing media outlets to set their own access fees.
The analytics are free for publishers, but AI companies are charged a “transaction fee for every piece of content they access,” he said.
For Muehmel, the online takeover by AI crawlers cannot be resolved with only “partial measures or by an individual company.”
“This is an evolution of the entire Internet economy, which will take years,” he said.
Prince said, if the bot swarm continues to roam freely online, “all of the incentives for content creation are going to go away,” adding that it would be “a loss, not just for us humans that want to consume it, but actually for the AI companies that need original content in order to train their systems.”
Crowds in Bangladesh are flocking to snap photographs with an unlikely social media star — an albino buffalo with flowing blond hair nicknamed “Donald Trump” that is due to be sacrificed within days. Owner Zia Uddin Mridha, 38, said his brother named the 700kg bull over its flowing helmet of hair resembling the signature look of the US president. “My younger brother picked this name because of the buffalo’s extraordinary hair,” he said at his farm in Narayanganj, just outside the capital, Dhaka. Mridha said that a constant stream of curious visitors — social media fans, onlookers and children — have come throughout
It began as a satirical online project. Now millions of young people in India are flocking to it as an outlet for their frustration. A parody political party called the Cockroach Janta Party, with the insect as its symbol, has exploded across India’s social media by turning absurdist humor into protest. Memes and short videos mocking corruption, joblessness and political dysfunction have flooded social media sites, where millions of users are embracing the cockroach — known for its ability to survive harsh conditions — as a tongue-in-cheek symbol of endurance. The online movement’s rise has been unusually rapid. The Cockroach Janta Party (CJP)
HOTTER: While Indians are accustomed to summer heat, climate change has caused northwestern India to warm faster than other parts of the country, an academic said Roads and markets have emptied during afternoons and some farmers have switched to nighttime work to avoid scorching temperatures as a heat wave grips large parts of India. The India Meteorological Department forecast maximum temperatures for yesterday of about 45°C in the capital, New Delhi, where authorities have opened temporary “cooling zones” to help people cope. The weather department warned that conditions would likely persist across several northern regions in the coming days, with temperatures staying well above seasonal averages. Authorities urged people to stay indoors during the hottest hours and take precautions against heat-related illnesses. India declares a heat wave whenever maximum temperatures
BIGGER ROLE: Beijing has said it maintains an impartial stance on the war in Ukraine, but by training Russian troops, China is far more involved than previously known China’s armed forces secretly trained about 200 Russian military personnel in China late last year, and some have since returned to fight in Ukraine, according to three European intelligence agencies and documents seen by Reuters. While China and Russia have held a number of joint military exercises since Moscow’s full-scale invasion of Ukraine in 2022, Beijing has repeatedly said that it is neutral in the conflict and presents itself as a peace mediator. The covert training sessions, which predominantly focused on the use of drones, were outlined in a dual-language Russian-Chinese agreement signed by senior Russian and Chinese officers in Beijing on
RSS