AI Signal 142
Using a documentation page as a search query to attract AI agents to your business
The author treats a documentation page as a search query that lures AI agents capable of referring users to the company's service.
It shows a shift from conventional SEO to agent-focused optimization, requiring engineers to track how language models refer traffic. Engineers must decide which LLM crawlers to allow or block, affecting server load, data costs, and the accuracy of referral analytics. The approach also reveals limits of current UTM-based tracking, pushing teams to improve onboarding surveys and build custom evaluation suites.
Written by elseif from the cluster below · every claim links back to a sourceThe three things worth knowing
The documentation page functions as a long, complex search query designed to attract AI agents that can refer interested people to the company.
Initial measurements showed Claude as the top referral source, but incomplete analytics led to a revised onboarding survey and plans for a custom eval suite.
Blocking aggressive training crawlers such as Claude-SearchBot via Cloudflare reduces server overload while preserving useful search bots.
THE READ
What the cluster adds up to.
The core change is treating a documentation page as a search query that AI agents can match, moving the focus from traditional search engine optimization to agent-focused optimization. This shift requires engineers to instrument bot traffic and understand which language models are citing the site. The cost includes setting up measurement tools, revising onboarding flows, and potentially building an evaluation suite to test model responses. If agents do not expose referral data or if their internal mentions are not captured, the approach stops providing useful signal.
Engineers also face the cost of managing bot access, deciding which crawlers to block or allow. Using Cloudflare to serve 403 or 402 responses can mitigate overload from aggressive training bots like Claude-SearchBot, but over-blocking may remove useful search bots that drive traffic. The effort to differentiate training crawlers from search crawlers adds operational complexity. If new LLM providers introduce bots without clear identifiers or change their crawling behavior, the current blocking rules may become ineffective.
Attracting more good bots involves creating content that language models favor, such as lists and clear examples, and keeping the documentation up to date. This activity demands ongoing editorial work and monitoring of referral sources to ensure the traffic remains high quality. If model updates alter weighting or if agents begin to ignore the site, the attraction effort may yield diminishing returns. Additionally, low-quality bots could game the system, requiring continual filtering.
More broadly, reliance on opaque model memory means traditional UTM parameters often fail to capture agent referrals, prompting investment in new evaluation methods and possibly paying for scrape access. These adaptations increase engineering overhead and may involve legal or policy considerations around data scraping. If platforms restrict bot access or if regulations limit scraping, the AEO strategy could cease to work as intended.
Written by elseif from the cluster below · checked for specifics the sources never containedTHE CLUSTER