INFRA Signal 203
Making AI search smarter
↗
AI answer engines that summarize content are breaking the old traffic-for-crawl bargain: a Pew Research study found users click through to original sources only 8% of the time when Google shows an AI summary. For site operators, Cloudflare's program could cut the infrastructure cost of redundant bot crawling—over 50% of good-bot traffic re-fetches unchanged pages—while creating a mechanism to capture revenue when content powers AI answers.
Written by elseif from the cluster below · every claim links back to a sourceThe three things worth knowing
Cloudflare is launching a research program that uses signals from its network (covering more than 20% of the web) about content freshness and traffic flows to help answer engines surface better results and skip re-crawling unchanged pages.
More than 50% of crawl traffic from good bots goes to re-fetching pages that haven't changed, imposing compute and serving costs on both AI companies and site owners for no new information.
Cloudflare intends to build a system where creators are paid when their work is used to answer someone's question, replacing the current model where content is scraped for free.
THE CLUSTER