AI Signal 142
HTTP content negotiation can serve Markdown to AI agents, cutting tokens and improving retrieval
↗
A proposed pattern uses the HTTP Accept header with the value text/markdown to serve Markdown representations of web content to AI agents, reducing token consumption and improving retrieval signal-to-noise by stripping navigation, scripts, and layout markup.
For operators whose sites are crawled by AI agents, serving Markdown via content negotiation means agents spend context window capacity on actual prose rather than DOM noise, directly improving RAG pipeline quality. The approach relies on existing HTTP standards rather than requiring new infrastructure or separate API endpoints.
Written by elseif from the cluster below · every claim links back to a sourceThe three things worth knowing
Serving Markdown via Accept headers drops nav, styles, scripts, and layout wrappers, reducing the bytes AI agents must process.
The approach improves retrieval signal-to-noise by removing ads, related-content rails, and modal overlays that muddy RAG embedding.
Copy-paste configuration guides exist for Nginx, Caddy, Cloudflare Workers, Next.js, and numerous other server frameworks.
THE CLUSTER