A swarm of artificial intelligence (AI) “crawlers” is running rampant on the Internet, scouring billions of Web sites for data to feed algorithms at leading tech companies — all without permission or payment, upending the online economy.
Before the rise of AI chatbots, Web sites allowed search engines to access their content in return for increased visibility, a system that rewarded them with traffic and advertising revenues.
However, the rapid development of generative AI has allowed tech giants such as Google and OpenAI to harvest information for their chatbots with Web crawlers, without humans ever needing to visit the original sites.
Photo: Bloomberg
Traditional content producers, such as media outlets, are being outpaced by AI crawlers, which have cut into their online operations and advertising revenues.
“Sites that gave bots access to their content used to get readers in exchange,” but the arrival or generative AI completely breaks that model, Dataiku AI strategy head Kurt Muehmel said.
Wikipedia last month reported human Internet traffic fell by 8 percent between last year and this year, because of a rise in AI search engine summaries.
“The fundamental tension is that the new business of the Internet that is AI-driven doesn’t generate traffic,” Cloudflare CEO Matthew Prince said.
Cloudflare, a US Internet services provider, processes more than 20 percent of all Internet traffic. It this year announced a new measure aimed at blocking AI crawlers from accessing content without payment or permission from Web site owners.
“It’s basically like putting a speed limit sign or a no trespassing sign,” Prince said. “Badly behaving bots can get by that, but we can track that... Over time, we can tighten these controls in a way that we’re confident the AI companies can’t get through.”
The measure, which applies to more than 10 million Web sites, has already “attracted the attention of artificial intelligence giants,” he added.
On a smaller scale, US start-up TollBit is providing online news publishers with tools to block, monitor and monetize AI crawler traffic.
“The Internet is a highway,” TollBit CEO and cofounder Toshit Panigrahi said, describing the company as a “tollbooth on the Internet.”
TollBit works with more than 5,600 sites, including USA Today, Time magazine and the Associated Press, allowing media outlets to set their own access fees.
The analytics are free for publishers, but AI companies are charged a “transaction fee for every piece of content they access,” he said.
For Muehmel, the online takeover by AI crawlers cannot be resolved with only “partial measures or by an individual company.”
“This is an evolution of the entire Internet economy, which will take years,” he said.
Prince said, if the bot swarm continues to roam freely online, “all of the incentives for content creation are going to go away,” adding that it would be “a loss, not just for us humans that want to consume it, but actually for the AI companies that need original content in order to train their systems.”
James Watson — the Nobel laureate co-credited with the pivotal discovery of DNA’s double-helix structure, but whose career was later tainted by his repeated racist remarks — has died, his former lab said on Friday. He was 97. The eminent biologist died on Thursday in hospice care on Long Island in New York, announced the Cold Spring Harbor Laboratory, where he was based for much of his career. Watson became among the 20th century’s most storied scientists for his 1953 breakthrough discovery of the double helix with researcher partner Francis Crick. Along with Crick and Maurice Wilkins, he shared the
OUTRAGE: The former strongman was accused of corruption and responsibility for the killings of hundreds of thousands of political opponents during his time in office Indonesia yesterday awarded the title of national hero to late president Suharto, provoking outrage from rights groups who said the move was an attempt to whitewash decades of human rights abuses and corruption that took place during his 32 years in power. Suharto was a US ally during the Cold War who presided over decades of authoritarian rule, during which up to 1 million political opponents were killed, until he was toppled by protests in 1998. He was one of 10 people recognized by Indonesian President Prabowo Subianto in a televised ceremony held at the presidential palace in Jakarta to mark National
US President Donald Trump handed Hungarian Prime Minister Viktor Orban a one-year exemption from sanctions for buying Russian oil and gas after the close right-wing allies held a chummy White House meeting on Friday. Trump slapped sanctions on Moscow’s two largest oil companies last month after losing patience with Russian President Vladimir Putin over his refusal to end the nearly four-year-old invasion of Ukraine. However, while Trump has pushed other European countries to stop buying oil that he says funds Moscow’s war machine, Orban used his first trip to the White House since Trump’s return to power to push for
LANDMARK: After first meeting Trump in Riyadh in May, al-Sharaa’s visit to the White House today would be the first by a Syrian leader since the country’s independence Syrian President Ahmed al-Sharaa arrived in the US on Saturday for a landmark official visit, his country’s state news agency SANA reported, a day after Washington removed him from a terrorism blacklist. Sharaa, whose rebel forces ousted long-time former Syrian president Bashar al-Assad late last year, is due to meet US President Donald Trump at the White House today. It is the first such visit by a Syrian president since the country’s independence in 1946, according to analysts. The interim leader met Trump for the first time in Riyadh during the US president’s regional tour in May. US envoy to Syria Tom Barrack earlier
RSS